A group of computers or machines connected together is often referred to as a cluster...Clusters are often used in data centers, cloud computing, and scientific research for large-scale data processing..These machines or computers are connected through a network and collaborate to execute operations that might be too demanding for a single machine.
Clusters can serve various purposes, such as:
- High-Performance Computing (HPC): Multiple machines work together to process complex calculations faster.
- Load Balancing: Distributes work evenly across multiple machines to ensure no single machine is overwhelmed.
- Fault Tolerance: If one machine fails, others in the cluster can continue to handle the workload, ensuring high availability.
- Distributed Storage: Data is spread across multiple machines to improve scalability and data redundancy.
==========================================
Here is a cleaner, more structured version of the original content that maintains the key ideas and adds some clarifications:
Hadoop Architecture and Workflow in the Pre-Cloud Era
In the pre-cloud era, when cloud technologies were not as developed, large-scale data processing was done using Hadoop clusters on on-premise infrastructure. Here’s an overview of how the system worked and the roles involved:
1. Creation and Management of Clusters
-
Hadoop Administrator: This role was responsible for setting up the Hadoop cluster. The Hadoop admin would create a cluster by configuring multiple machines (worker nodes) to work together. These nodes were interconnected to ensure they could communicate with each other effectively.
-
Role of Cluster: The Hadoop cluster would typically consist of:
- Master node: Responsible for managing and assigning tasks, scheduling, and coordinating work.
- Worker nodes: Responsible for processing and storing data.
The cluster was housed on-premise, meaning that organizations had their own data centers with physical servers.
2. Data Ingestion and Processing
- Before data could be processed, it had to be ingested into the Hadoop cluster. For this purpose, tools like Oracle GoldenGate, Kafka, Sqoop, and NiFi were commonly used. These tools were responsible for pulling data from external databases and loading it into the Hadoop system.
- Once the data was ingested, the processing happened on the cluster. The data was split into blocks for efficient processing. Hadoop distributed the data across different machines for parallel processing to avoid system crashes.
3. Working with Clusters
- Edge Node: Developers didn’t directly access the master node (which was critical for cluster management); instead, they would connect to an edge node (a gateway node) to interact with the cluster. This was done to ensure security and proper access controls.
- Processing Data: Developers would write code in frameworks like Hive, HBase, Pig, and Spark to process the data. The tasks were assigned to workers, and the master node coordinated the distribution of data across the workers.
- Hive: A SQL-like interface to process data on top of Hadoop.
- HBase: A NoSQL database built on top of Hadoop.
- Pig: A high-level platform for processing large data sets.
4. Data Storage in Blocks
- Data Blocks: When data was loaded into the Hadoop system, it was divided into blocks (default size: 256MB). These blocks were distributed across the worker nodes. The reason for this is similar to memory management in a computer. If all data were processed at once, it could overwhelm the system’s memory (RAM) and cause crashes.
- Each worker machine had a local storage that held data blocks. The data in these blocks was processed by the mapper and reducer tasks behind the scenes.
- If a block was processed, the worker would notify the master node. Once all workers had finished their tasks, the results would be collected and processed by the master.
5. Distributed Data Processing
- The Hadoop Distributed File System (HDFS) was used for storing data. Data blocks were stored in the HDFS layer, which managed how data was distributed and replicated across worker nodes to prevent data loss.
- Processing: After data was stored in blocks, the processing logic was executed by the Hadoop engines (e.g., MapReduce, Hive, Pig, Spark). These engines performed operations on the blocks to process the data in parallel, using the map-reduce model.
6. The Role of the Hadoop Developer
- The Hadoop Developer did not have the responsibility to set up clusters or configure the network between machines. Their role was to write the ETL scripts (Extract, Transform, Load) to process data on the cluster using frameworks like Hive, HBase, Pig, etc.
- Once the scripts were ready, developers would deploy them to the cluster. The code would then be executed by the Hadoop framework, and the results would be returned after processing.
7. Job Execution and Monitoring
- Resource Manager: The Resource Manager in Hadoop was responsible for managing resources and assigning tasks to the worker nodes. It decided which node would handle which data block and managed the distribution of data across the cluster.
- Monitoring: A Node Manager ran on each worker machine and monitored the health of the worker. If there was a failure (e.g., if a block became corrupted or a worker crashed), the Node Manager would send alerts to the master node.
8. Why Blocks Were Introduced
- Avoiding Crashes: Blocks were introduced to avoid overwhelming the system’s memory. If a machine had only 32GB of RAM, it couldn’t handle 256GB of data all at once. By breaking up data into smaller, manageable chunks (blocks), Hadoop could efficiently process large datasets without memory crashes.
9. The Lifecycle of Hadoop Data Processing
- Hadoop Cluster Creation: The first step was to purchase a Hadoop distribution (e.g., from Cloudera, Hortonworks, or IBM) and create the cluster using the purchased distribution.
- Cluster Setup: The Hadoop administrator would install the Hadoop software and create the cluster.
- Development Phase: Developers would then write and deploy Hive, HBase, or Pig scripts to process the data.
- Processing: Once the scripts were deployed, the system would handle the distribution of tasks across the cluster, processing the data in blocks using MapReduce and other engines.
10. Challenges and Debugging
- Debugging and troubleshooting were critical tasks for Hadoop developers. In the event of a failure, such as a corrupted block or worker crash, developers needed to understand how the data was distributed and be able to interpret the error logs.
- If a block became corrupted, the developer would need to know which worker was handling it and which part of the process failed.
Transition to Cloud
With the rise of cloud technologies, many organizations moved from on-premise Hadoop clusters to cloud-based solutions (like AWS, Azure, and Google Cloud). This shift brought several advantages:
- Scalability: Cloud platforms provided automatic scaling, so companies no longer had to worry about managing physical machines and clusters manually.
- Managed Services: Cloud services like Amazon EMR, Google Dataproc, and Azure HDInsight offered managed Hadoop clusters, reducing the need for a dedicated Hadoop admin.
- Cost Efficiency: Cloud platforms offered pay-per-use models, so companies could scale up and down based on their needs, without investing in physical infrastructure.
Despite this shift, some companies still maintain on-premise Hadoop clusters for specific reasons like security, compliance, or legacy systems.
Conclusion
In the past, Hadoop developers had to work closely with Hadoop administrators to manage the clusters, understand how data was distributed across the workers, and write code to process large-scale data. Understanding the underlying architecture, such as how data was split into blocks and processed by mappers and reducers, was crucial for debugging and troubleshooting issues. With the advent of cloud technologies, many of these tasks have been simplified through managed services, but the core concepts of distributed data processing remain important.
=========================
· Hadoop administrators were responsible for setting up and configuring Hadoop clusters, ensuring communication between worker nodes.
===============================================================
in hadoop we have a master node and worker node...now to ingest any database into hadoop clsuter we need a tool to bring in the data based on its type...so as a dataengg, depedning on the type of data we will decide the tool we willbe suing to get the data from external database into hadoop cluster..now not many people using it, in oden days likes 2008 used to use this way...so if we want to put this in cv we need to know....sqoop is one of the tool to get data from external to hadoop cluster..so somehwere it has to run to ingest data from external to cluster....we need to have a machine to run the tool, and within that machine, we instll this sqoop...as you know hadoop admin will install but we can also installl...so the configuration it looks like a command line syntax...to get the data use command line sqoop, source destination and mapper(paraelle processing based on the volme of data, so 5 or 6 processeors))..
edge node(gateway) means it is not reciding in the cluster...it is outside the cluster..it is a gateway to execute the sqoop commands...so what admin does is, they give necessary persmision to exectue the sqoop commands...so we need to login into the machine where sqoop installed using different ways we willelarn later...then execute commands based on full load or incremental load based on client requirhemnt
so sqoop is a tool used to ingest any rdbs (tables) into HDFS ..data ingestion tool from external data to hadoop cluster stored in HDFS
now we learned..how we can get rdbms data (full and incremental data) into hdfs using sqoop..full load and incremental data..add this in CV
always think like this...in big data or hadoop always think data like VVV's 3Vs....
as dataengg we need to ask these questions
1.velocity (ni how mcuh speed you wanted to load the data...some people say i want data quickly or some say not urgent can have end of day...so based on that we can decide mappers orn no of prcessors)..
2, variety or veriecity (what type of data it is, oracle etc)
3. volume (how much size are we talking about)
there will be a 3rd party tool that converts network data IOT (the tool thats understand that particular data like jhealthcare etc) into normal readable format like in json format (key valu paor)..there should be some tool which can pick up this json data and put that into hdfs hadoop cluster..one of the technology being used is kafka (to read the data and wrte the data where we wanted like hdfs )...
As a data engg we deal with eighet batch data (you dont need to run the processes every second or minut, you have a flexitiy of run this pricess eery day or every hour etc) - slow moving, infrequent data uses framewok like sqoop...
or streaming data - banks etc(nothing but coninuus data) --- kafka
-->you spoke about as a dataEng we work with 2 types of data mainly..
1. batch data - you showed us how this batch data with full vs incremental is loaded into hdfs using sqoop and
2. streaming data - contonous data using kafka
--> spoke about 3 V's - importance of 3 V's
Summary
hadoop - consists of master (consists of resource manager) and worker node (disk, storage (hdfs) and processing layer/engine (map reduce)
what is the processing layer in hadoop engine - map reduce
what is the storage layer in hadoop - HDFS
if someone asks, i have a 20gb file and i want to process it...how does hadoop handles it..how do you decide the things needed to process 20gb file.
if i have a 20gb file, i will think about cluster with one master and maybe 3 workers..so 20gb will be distributed across 3 workers and who will distribute that is master...
so 20gb is stored in blocks...so default block size is 256mb, so 1GB contains 4x256mb blocks..so there will be 80 blocks.. will be distributed across this 3 worker node...you need to think, what does a machine contains, it has memoery, RAM and diskStorage..so depends on the disk storage i got..
lets say we have a disk storage of 64gb each worker node.. if we think 20gb should fit in one worker node...but the master will not do that way...as rest of the worker nodes sits idle.. master nodes contains a resource manager that equaly distrubute across 3 worker nodes.. as we know 80 blocks, each worker node lets say gets 26, 26 and 28 blocks...this is how master resource manager distributes across worker nodes and the parallely process it using a map reduce..
lets say one of the worker machine crash while processing...and that particular machine is crashed, which means the 26gb is not available...so in realtime, backup blocks are stored in all worker nodes...so when one of the worker node is available, the data back up is avaibale in other worker nodes ..its called as replication factor.. basically copying the blocks files availbale in other worker nodes..this we can decide while setting up the hadoop architecture..whenever the cluster is created, you will config with replication factor depending on the resources available to you...
inteervioew question - how actually hadoop handles the failure of the worker node (need to explain the replciation factor)
