Hadoop

what is hadoop:

Hadoop: An open-source framework by the Apache Software Foundation.
Purpose: Designed for distributed storage and processing of large data sets.
Hardware: Uses clusters of commodity hardware.
Scalability: Highly scalable to handle vast amounts of data efficiently.
it has ensure Fault Tolerance through data replication. Data replication factor by default is 3:
- Data is split into blocks.
- Each block is replicated across multiple nodes.
- If one node fails, data is still accessible from another node with a copy.

Components of Hadoop

Hadoop consists of four main components:
Hadoop Distributed File System (HDFS):
Purpose: HDFS is designed to store large files across multiple machines. It provides high throughput access to application data and is highly fault-tolerant.
Functionality: It splits large files into smaller blocks and distributes them across the cluster. Each block is replicated to ensure data reliability and availability.
MapReduce:
Purpose: MapReduce is a programming model used for processing large data sets in parallel.
Functionality: It divides the processing into two phases: the Map phase, which processes and filters data, and the Reduce phase, which aggregates the results.
YARN (Yet Another Resource Negotiator):
Purpose: YARN is responsible for resource management and job scheduling.
Functionality: It allocates system resources to various applications and manages the execution of tasks.
Hadoop Common:
Purpose: Hadoop Common provides the necessary libraries and utilities needed by other Hadoop modules.
Functionality: It includes common utilities that support the other Hadoop components.

How Hadoop Handles Big Data

Hadoop handles big data through its distributed architecture and parallel processing capabilities:
Distributed Storage: HDFS stores data across multiple nodes in a cluster, ensuring data is replicated and available even if some nodes fail.
Parallel Processing: MapReduce processes data in parallel across the nodes in the cluster, significantly speeding up data processing tasks.
Scalability: Hadoop can easily scale out by adding more nodes to the cluster, allowing it to handle increasing amounts of data.
Fault Tolerance: Data replication in HDFS ensures that data is not lost even if some nodes fail. YARN manages resources efficiently, ensuring tasks are rescheduled in case of failures.
Hadoop's architecture makes it a powerful tool for managing and processing big data, enabling organizations to gain insights from vast amounts of information.

Hadoop is a powerful tool for big data processing, but it does come with its own set of challenges:
Small File Handling:
Hadoop is designed for large files, and managing a large number of small files can be inefficient and slow
1
.
Slow Processing Speed:
The MapReduce framework reads and writes data to and from the disk, which can result in slower processing speeds compared to in-memory processing frameworks like Apache Spark
2
.
Real-Time Data Processing:
Hadoop's MapReduce is not suitable for real-time data processing. It is more suited for batch processing
2
.
Complexity:
Setting up and managing a Hadoop cluster can be complex and requires specialized knowledge
3
.
Security:
Hadoop has several security challenges, including issues with authentication, encryption, and auditing
1
.
Iterative Processing:
Hadoop is not efficient for iterative processing tasks, which are common in machine learning algorithms
2
.
High Costs:
The tight coupling of storage and compute resources can lead to high costs, especially as the cluster scales
1
.

Hadoop Architecture & Data Processing - A Simple and Logical Explanation

1. The Era Before Cloud - On-Premise Hadoop Clusters

Before cloud computing became popular, companies relied on on-premise Hadoop clusters to store and process large amounts of data. This setup required a Hadoop Admin to manually configure and manage the cluster, which included:

Setting up machines (workers and master nodes).
Establishing network connectivity between them.
Ensuring smooth processing of distributed data.

On the other hand, Hadoop Developers were responsible for:

Writing ETL scripts (Hive, Pig, HBase) to process data.
Deploying scripts on the cluster via an Edge Node (a gateway to interact with the cluster).
Debugging failures if data processing jobs failed.

2. Hadoop Cluster Components

A Hadoop Cluster consists of two main types of nodes:

Master Node (a.k.a. NameNode & ResourceManager) → The Brain

NameNode:
- Manages metadata (information about where files are stored in HDFS).
- Keeps track of which worker node (DataNode) has which data blocks.
ResourceManager:
- Handles job scheduling and resource allocation for processing tasks.
- Assigns tasks to different worker nodes (NodeManagers).

Worker Nodes (a.k.a. DataNodes & NodeManagers) → The Workforce

DataNode:
- Stores actual data blocks in HDFS...
- Storage Layer (HDFS - Hadoop Distributed File System) → Stores data in Blocks (default size 256MB).
- Sends heartbeat signals to the NameNode to confirm it's alive.
NodeManager:
- Runs processing tasks assigned by ResourceManager.
- Uses MapReduce or other processing engines like Spark to process data.

Edge Node (Gateway Node) → The Entry Point

The place where developers log in to run commands, submit jobs, and check logs.
Not part of the cluster but acts as an interface to interact with Hadoop.

3. Data Ingestion – How Data Enters Hadoop?

Before processing data, we need to ingest it into Hadoop. Different tools are used based on the type of data:

Batch Data (Structured Data - RDBMS like Oracle, MySQL, SQL Server) → Use Sqoop to import tables into HDFS.
Streaming Data (e.g., IoT, logs, social media feeds) → Use Kafka for real-time ingestion.

Example: If a client has customer subscription data in an Oracle database and wants reports on popular offers in Europe, we use Sqoop to bring that data into Hadoop for processing.

4. Why Data is Stored in Blocks?

Imagine trying to open a 100GB file on an 8GB RAM machine—it would crash!
Hadoop solves this by:

Splitting data into smaller blocks (256MB each by default).
Distributing these blocks across multiple worker nodes (DataNodes).
Processing them in parallel using MapReduce.

Example: If we have a 20GB file, it will be divided into:

80 blocks (since 1GB = 4 x 256MB blocks, 20GB = 80 blocks).
Distributed across 3 worker nodes (e.g., 26, 26, 28 blocks each).

This ensures efficient parallel processing without overloading any single machine.

5. How Hadoop Handles Failures? (Replication Factor)

If a worker node (DataNode) crashes while processing, its data is not lost because Hadoop uses a Replication Factor (default = 3):

Every block is copied to at least 2 other DataNodes.
If a DataNode fails, Hadoop automatically switches to a backup copy.

Interview Question: How does Hadoop handle worker node failure?

Answer: Replication Factor ensures that every data block is stored on multiple DataNodes. If a node fails, the data is still available on other nodes, preventing data loss.

6. Hadoop Data Processing – Hive, Pig, and HBase

To process data in Hadoop, developers use:

Hive → SQL-like queries for structured data.
Pig → Scripting language for semi-structured/unstructured data.
HBase → NoSQL database for real-time access.

Earlier, developers wrote custom MapReduce programs in Java or Python. Now, Hive and Pig handle this automatically, making development easier.

7. Batch vs. Streaming Data Processing

As a Data Engineer, you deal with two types of data processing:

Batch Processing → Large files processed at scheduled times (e.g., end of the day).
- Uses Sqoop (RDBMS to Hadoop) or Hive/Pig.
- Example: Generating daily sales reports.
Streaming Processing → Real-time, continuous data (e.g., stock market, fraud detection).
- Uses Kafka to handle real-time data.
- Example: Detecting fraudulent credit card transactions in real time.

8. The Shift to Cloud – Why Companies Moved Away from Hadoop?

Managing on-premise Hadoop clusters required:

Dedicated Hadoop Admins to maintain infrastructure.
Manual scaling of storage & computing power.

With Cloud (AWS, Azure, GCP), we now have:

Managed Hadoop Services (AWS EMR, Azure HDInsight).
Auto-scaling, cost efficiency, and no need for on-premise hardware.

Example: Earlier, companies rented Hadoop distributions like Cloudera or IBM BigInsights, but now they prefer cloud-based solutions.

📌 Summary - What to Put in Your CV?

Hadoop Experience:

Worked with on-premise Hadoop clusters (NameNode, DataNodes, ResourceManager, NodeManagers).
Ingested batch data using Sqoop and streaming data using Kafka.
Processed data using Hive & Pig scripts.

Key Concepts:

HDFS & MapReduce → Storage & Processing layers.
Replication Factor → Ensures data availability during failures.
Edge Node → Gateway to access Hadoop Cluster.

Interview Readiness:

How does Hadoop distribute a 20GB file?
- Breaks it into 80 blocks (256MB each), distributes across DataNodes.
What happens if a worker node (DataNode) fails?
- Replication Factor ensures backup blocks are available.
Batch vs. Streaming – When to use Sqoop vs. Kafka?
- Batch (scheduled data loads) → Sqoop
- Streaming (real-time data) → Kafka

🚀 With this knowledge, you’re ready to impress in Hadoop/Data Engineering interviews! 💪

===================================================================

table with examples of creating internal and external tables, including the LOAD DATA INPATH command for both:

Internal vs External Tables Comparison

Feature	External Table	Internal Table
Data Storage	Data remains in HDFS or an external location (user-defined).	Data is stored in Hive’s warehouse (default location).
Data Management	Hive manages only the metadata (schema/structure). Data stays where it is.	Hive manages both metadata and data. Data is moved to Hive's warehouse.
Table Deletion Impact	Deleting the table does not delete the data. Data remains in the external location.	Deleting the table deletes both the schema and the data in Hive's warehouse.
`LOAD DATA INPATH` Command	Required to load data from external location (HDFS, local, etc.) into the specified table location.	Optional – If data is not in the warehouse, you can use `LOAD DATA` to move it into Hive's warehouse.
Storage Location	User-defined location (specified in the `LOCATION` clause). Data is not managed by Hive.	Data is stored in Hive's default warehouse directory, under `/user/hive/warehouse/`.
Use Case	Best for shared data, external systems, or when you want to manage data outside Hive.	Best for Hive-managed data where data is fully controlled and stored within Hive's warehouse.

Examples:

1. External Table Creation:

In an external table, you define the data location in HDFS, and Hive manages only the metadata (schema). The data remains in its original location, and Hive only references it.

CREATE EXTERNAL TABLE transactions_ext (
   transaction_id STRING, 
   user_id STRING, 
   amount DOUBLE, 
   transaction_date STRING, 
   transaction_type STRING
) 
LOCATION '/user/rama/external_data/';  -- External location where data is stored

Key Point: The data stays at /user/rama/external_data/ in HDFS, and Hive only manages the schema. If you drop the external table, the data remains in HDFS.

Loading Data into External Table:

LOAD DATA INPATH '/user/rama/trans.csv' INTO TABLE transactions_ext;

This command loads the data from /user/rama/trans.csv (HDFS location or local) into the external table.
The data remains in the original location (/user/rama/external_data/).

2. Internal Table Creation:

In an internal table, Hive manages both the schema and the data. The data is physically moved into Hive's warehouse and fully controlled by Hive.

CREATE TABLE transactions_int (
   transaction_id STRING, 
   user_id STRING, 
   amount DOUBLE, 
   transaction_date STRING, 
   transaction_type STRING
) 
STORED AS TEXTFILE;  -- Storing as TEXTFILE format (you can also use ORC, Parquet, etc.)

Key Point: The data is stored in Hive's warehouse (by default, /user/hive/warehouse/) once you load it into the table.

Loading Data into Internal Table:

LOAD DATA INPATH '/user/rama/trans.csv' INTO TABLE transactions_int;

This command moves the data from /user/rama/trans.csv (HDFS or local) into Hive's warehouse.
Hive now manages both the data and schema. When you drop the table, the data is also deleted from the warehouse.

Summary of Key Differences with Examples:

Feature	External Table	Internal Table
Data Storage	Data remains at user-defined location (e.g., `/user/rama/external_data/`).	Data is moved into Hive's warehouse (default location: `/user/hive/warehouse/`).
Creation Syntax	`CREATE EXTERNAL TABLE` with `LOCATION` to specify data location.	`CREATE TABLE` (no `LOCATION` clause). Hive moves data to its warehouse.
Data Loading	Use `LOAD DATA INPATH` to load data into an external table.	Use `LOAD DATA INPATH` to load data into an internal table.
Data Deletion on Drop	Data is not deleted when the external table is dropped. Data remains in the external location.	Data is deleted when the internal table is dropped.

Use Cases:

External Table:
- Use Case: When data is stored outside Hive, shared among different systems, or when you don’t want Hive to move the data.
- Example: Data generated by other systems like log data, machine-generated data, or when using data from another source like Spark or HDFS directly.
Internal Table:
- Use Case: When you want Hive to fully manage both the data and the schema, and you don't need to keep the data in its original location.
- Example: Data that is generated or processed inside Hive for analytical purposes, and you want Hive to manage it entirely.

==============================================

--> HDFS is responsible for storing data across the cluster in a distributed manner.

Secondary NameNode: Although its name suggests it might be a backup for the NameNode, it doesn’t serve as a failover; instead, it periodically downloads the current state of the metadata from the NameNode and merges it into a checkpoint to avoid the NameNode’s metadata from growing too large.

-->

When you use hdfs dfs -put, the file is distributed into blocks and stored across DataNodes in the Hadoop cluster.
When you use hdfs dfs -get, the command retrieves the file blocks from different DataNodes and combines them to give you the original file.

In simpler terms, you're "connecting" to HDFS via these commands, and HDFS manages where the actual data is stored, replicates it, and retrieves it as needed.

-->

hdfs dfs is a command-line utility for interacting with HDFS (Hadoop Distributed File System).

It allows you to perform file operations (like listing, moving, and deleting) on HDFS, rather than your local file system.

HDFS is a distributed system, so commands will interact with data across multiple nodes in the Hadoop cluster.

---->

The HDFS DFS command is used to interact with Hadoop Distributed File System (HDFS). You can use it to perform various file operations such as uploading, downloading, listing, and managing files within HDFS.

Common HDFS DFS Commands and Their Explanations:

Certainly! Here's the information in the requested format:

hdfs dfs -ls <path>
Purpose: Lists the files and directories at the specified HDFS path.
hdfs dfs -put <local_path> <hdfs_path>
Purpose: Uploads a file from the local file system to HDFS.
hdfs dfs -get <hdfs_path> <local_path>
Purpose: Downloads a file from HDFS to the local file system.
hdfs dfs -cat <hdfs_path>
Purpose: Displays the content of a file stored in HDFS.
hdfs dfs -mkdir <path>
Purpose: Creates a new directory in HDFS.
hdfs dfs -rm <hdfs_path>
Purpose: Removes a file from HDFS.
hdfs dfs -rmdir <hdfs_path>
Purpose: Removes an empty directory from HDFS.
hdfs dfs -du <path>
Purpose: Displays the disk usage of files and directories in HDFS.
hdfs dfs -chown <user>:<group> <path>
Purpose: Changes the owner and group of a file or directory in HDFS.
hdfs dfs -chmod <permissions> <path>
Purpose: Changes the file permissions for a file or directory in HDFS.

Understanding the `hdfs dfs -put` Command (in Detail):

Command: hdfs dfs -put /home/user/trans.csv /user/rama/
Purpose: This command uploads the file trans.csv from your local file system (/home/user/) to HDFS (/user/rama/).
What Happens Internally:
- The file trans.csv is broken into blocks (e.g., 128MB or 256MB by default).
- These blocks are then distributed across multiple DataNodes in the HDFS cluster.
- The NameNode keeps track of where each block is stored (this is the metadata).
- The blocks are replicated (usually 3 times) to ensure fault tolerance in case a DataNode fails.

Why Use HDFS DFS Commands?

File Management: You can easily manage files within HDFS using these commands (e.g., upload, download, delete files).
Efficient Data Processing: Since data is stored across many DataNodes, Hadoop can process large datasets in parallel using frameworks like MapReduce or Spark.
Scalability & Fault Tolerance: HDFS ensures that data is replicated, making sure there’s no data loss even if a DataNode crashes.

What is Metadata?

Metadata is data about data.
It doesn't contain the actual data, but provides information about the data's structure and where it's stored.

In the Context of Hadoop & HDFS:

NameNode stores the metadata.
DataNodes store the actual data (in blocks).

What Metadata Includes:

File Name (e.g., trans.csv).
File Size (e.g., 500MB).
Block Locations (where each block of data is stored on which DataNode).
Replication Information (how many copies of each block exist).
Permissions (who can access the data).
Modification Time (when the file was last modified).
Ownership Information (who owns the file).

Hadoop’s Metadata Workflow:

Data Splitting: Large files are split into smaller blocks (default 128MB or 256MB).
Data Storage: Blocks are stored on DataNodes in the cluster.
Metadata Storage: NameNode stores metadata about where each block is located and other file attributes.
File Access: When a file is needed, Hadoop checks the metadata to find out which DataNodes have the blocks.
Replication: Replication Factor ensures that each block is stored on multiple DataNodes (default is 3 replicas).

Why is Metadata Important?

Efficient Data Access: Helps quickly locate where data is stored.
Fault Tolerance: If a DataNode fails, metadata helps find other copies of the data.
Distributed Processing: Helps coordinate where the data is for parallel processing (MapReduce or Spark).

Summary:

Data: The actual content (e.g., your file).
Metadata: Information about the data (e.g., location, size, replication, etc.).
NameNode handles metadata.
DataNodes store the actual data in blocks.

============================

SUMMARY:

Put Data in HDFS First: Before creating tables, the data should be uploaded to HDFS using hdfs dfs -put or similar commands.
Internal Table:
- Hive manages both the data and the metadata.
- Data is physically moved from HDFS into the Hive warehouse.
- Must use LOAD DATA INPATH to move data into the internal table.
- Stored as TEXTFILE or other formats specified in the table definition.
External Table:
- Data remains in HDFS and is not managed by Hive.
- We only define the schema and metadata in Hive, and point to the HDFS location using the LOCATION clause.
- No need for LOAD DATA as data is directly read from the HDFS location.
Table Creation:
- Internal Table: CREATE TABLE int_trans (...) STORED AS TEXTFILE;
- External Table: CREATE EXTERNAL TABLE ext_trans (...) LOCATION '/user/rama/trans.csv';
Describe Formatted: Use DESCRIBE FORMATTED table_name; to check metadata and storage location for both internal and external tables.
Show Tables: SHOW TABLES; to list all tables in the current database.
Create and Use Database:
- Create Database: Used to logically organize tables.
- Use Database: Switches to a specific database for running queries and managing tables.
Drop Database: DROP DATABASE <db_name>; to remove the database (use CASCADE to remove tables inside).

Other Common Hive Commands:

Create Database: CREATE DATABASE db_name;
Use Database: USE db_name;
Show Tables: SHOW TABLES; (lists all tables in the current database).
Describe Table: DESCRIBE FORMATTED table_name; (shows metadata of the table, including storage location).
Drop Database: DROP DATABASE db_name; (deletes the database, but you can use CASCADE to also drop the tables).
Drop Table: DROP TABLE table_name; (removes table and its data in case of internal tables).

Create Database: We create a database to logically organize and manage tables within Hive, making it easier to group related datasets.

Use Database: The USE command switches the context to a specific database, allowing queries and operations to target tables within that database.

Sure! Here's a simple breakdown in bullet points:

Create Partitioned Table:
- You create a table in Hive and specify the column by which the data will be partitioned (e.g., transaction_type).
- Example: PARTITIONED BY (transaction_type STRING).
Create Directories in HDFS:
- You manually create directories in HDFS for each partition (e.g., /user/rama/partition/deposit, /user/rama/partition/withdrawal).
Add Partitions to Hive:
- You tell Hive about the partitions by adding them using ALTER TABLE and linking them to the HDFS directories.
- Example: ALTER TABLE partition_trans ADD PARTITION (transaction_type='deposit') LOCATION '/user/rama/partition/deposit'.
Load Data into Partitions:
- You load the data from HDFS into the appropriate partition using LOAD DATA.
- Example: LOAD DATA INPATH '/user/rama/trans.csv' INTO TABLE partition_trans PARTITION (transaction_type='deposit').
Verify Partitions:
- Check that the partitions are created successfully using SHOW PARTITIONS.
- This will show the list of partitions, like transaction_type=deposit, transaction_type=withdrawal.
Query Partitioned Data:
- You can query data by specifying the partition, which makes querying faster.
- Example: SELECT * FROM partition_trans WHERE transaction_type='deposit';.

Key Points:

Partitioning: Organizes data into folders based on a column (e.g., transaction_type).
Data Loading: Data is loaded into specific partitions (folders) in HDFS.
Efficient Queries: Queries are faster because only relevant partitions are searched.
===============================================

Data Storage Format Comparison in Hive

1️⃣ Overview

We will create multiple external Hive tables to store the same raw data but in different formats:

CSV (Uncompressed, row-based)
ORC (Optimized Row Columnar, compressed, columnar)
Avro (Schema evolution, row-based)
Parquet (Efficient columnar format)

We will then run a GROUP BY query on each table and compare execution times.

2️⃣ Step-by-Step Implementation

🔹 Step 1: Load the CSV Data into Hive

First, ensure the CSV file is available in HDFS and create an external table to read it.

CREATE EXTERNAL TABLE trans_csv (
    transaction_id STRING,
    user_id STRING,
    amount DOUBLE,
    transaction_date STRING,
    transaction_type STRING
)  
ROW FORMAT DELIMITED  
FIELDS TERMINATED BY ','  
LOCATION '/user/rama/trans.csv';

✅ Now, the CSV data is accessible via Hive.

🔹 Step 2: Create External Tables for ORC, Avro, and Parquet

Since Hive does not automatically convert file formats, we create new tables and manually insert data.

👉 ORC Table

CREATE EXTERNAL TABLE trans_orc (
    transaction_id STRING,
    user_id STRING,
    amount DOUBLE,
    transaction_date STRING,
    transaction_type STRING
)  
STORED AS ORC  
LOCATION '/user/rama/trans_orc';

INSERT INTO TABLE trans_orc  
SELECT * FROM trans_csv;

✅ Data is now stored in ORC format.

👉 Avro Table

CREATE EXTERNAL TABLE trans_avro (
    transaction_id STRING,
    user_id STRING,
    amount DOUBLE,
    transaction_date STRING,
    transaction_type STRING
)  
STORED AS AVRO  
LOCATION '/user/rama/trans_avro';

INSERT INTO TABLE trans_avro  
SELECT * FROM trans_csv;

✅ Data is now stored in Avro format.

👉 Parquet Table

CREATE EXTERNAL TABLE trans_parquet (
    transaction_id STRING,
    user_id STRING,
    amount DOUBLE,
    transaction_date STRING,
    transaction_type STRING
)  
STORED AS PARQUET  
LOCATION '/user/rama/trans_parquet';

INSERT INTO TABLE trans_parquet  
SELECT * FROM trans_csv;

✅ Data is now stored in Parquet format.

🔹 Step 3: Run Queries to Compare Execution Times

Now, execute the same query on all four formats and measure execution time.

SELECT transaction_type, SUM(amount)  
FROM trans_csv  
GROUP BY transaction_type;  

SELECT transaction_type, SUM(amount)  
FROM trans_orc  
GROUP BY transaction_type;  

SELECT transaction_type, SUM(amount)  
FROM trans_avro  
GROUP BY transaction_type;  

SELECT transaction_type, SUM(amount)  
FROM trans_parquet  
GROUP BY transaction_type;

✅ Measure query execution times to compare performance.

3️⃣ Performance Analysis and Comparison

Format	Storage Type	Compression	Read Performance	Best Use Case
CSV	Row-based	No compression	Slow	Simple raw data storage
ORC	Columnar	High compression	Fast	Best for analytical queries
Avro	Row-based	Moderate compression	Medium	Schema evolution required
Parquet	Columnar	High compression	Fastest	Optimized for large-scale queries

CSV is slow because it is not optimized for queries.
ORC and Parquet perform best because they are columnar and support efficient compression.
Avro is useful for data exchange, but its query performance is moderate.

📌 Summary

Step 1: Load CSV data into Hive as an external table.
Step 2: Create external tables for ORC, Avro, and Parquet.
Step 3: Insert CSV data into these tables to convert formats.
Step 4: Run GROUP BY queries on each table to compare execution times.
Step 5: Analyze performance differences between formats.

✅ Key Takeaway:
For large-scale data analysis, ORC and Parquet are the best formats due to fast query performance and high compression efficiency. 🚀

CSV format is inefficient because it lacks columnar storage and requires full file scanning.
ORC and Parquet are optimized for analytical queries, making them significantly faster.
Avro is suitable for data exchange, but not as optimized for analytical queries as ORC/Parquet.
Recommendation: Use Parquet or ORC for better performance in Hive-based analytics.