hadoop - distributed architecture and parallel processing.
- used for processing large datasets
- highly scalable to handle vast amount of data
- components and how hadoop handles big data - HDFS (stores large files across multiple machines making sure data is replicated even if one node fails, data is available in another node to copy (thisis called fault tolerance)), MapReduce (process large data in parallel, 2 phases - map phase process and filters the data and reduce phase aggregates the results), YARN (responsible for resource management and job scheduling)
- challenges - not suitable for small files and it is inefficient and slow for small files...mapredcue framework reads and writes data to and from the disk which slows the processing speed compared to in-memory processing frameworks like apache spark......not suitable for real-time processing, suitable for batch processing......high cost- it can lead to high cost due to tight coupling of storage and compute resource esp as the cluster scales..
- On premise hadoop clusters - hadoop admin manages the infrastructure which includes, setting up worker and master nodes, establish network connections between them etc...hadoop developers writes ETL scripts to process data, deploy scripts on cluster via edge node (gateway to interact with cluster)
- hadoop cluster components..Master node - namenode - which stores metadata of the files, resource manager - schedule jobs and manage resouces... worker nodes - datanode - this contins the actual data in blocks stored as hdfs, node manager - uses mapreduce to pricess the data....edge node - it is a gateway to access the cluster where the developers runs the scripts, check logs etc, an interface to interact with hadoop
- data ingestion into hadoop - before processing in hadoop, depending on the type of data i.e batch or streaming data...we use sqoop to bring the data into hadoop for processing and kafka for realtime ingestion.
- hadoop data processing - Hive - sql like queries for structured data and Pig for semi/unstrutured data.. HBase - nosql database..hive and pig - behind the scene MapReduce framework works.
- batch (data coming at scheduled times - sqoop, hive, pig)..streaming (data coming continuously, kafka sued to process)
- GCS connector (Google Cloud Storage connector) is a library/plugin that allows Hadoop, Spark, and other big data tools to read from and write to Google Cloud Storage (GCS) as if it were a native Hadoop filesystem.
When to Use
DistCp:-
Copying TBs of data between clusters or from on-prem HDFS to cloud storage.
-
High-performance, fault-tolerant transfers using multiple mappers.
What is
DistCp?DistCp(Distributed Copy) is a Hadoop utility used to copy large amounts of data in parallel between:-
Two HDFS clusters
-
HDFS and cloud storage (like GCS or Azure Blob/ADLS)
-
HDFS and local/remote file systems
It’s built on MapReduce, so it's ideal for large-scale data transfers.
-
🔒 Authentication:
Uses a service account key to authenticate Hadoop with GCS securely.
Authenticate On-Prem with GCP
To access GCS, you must authenticate your Hadoop cluster.
✅ Use a Service Account Key File
-
In GCP Console, create a Service Account with Storage access (
Storage AdminorStorage Object Admin) - Generate a JSON key file
-
Store the key securely on Hadoop nodes
We used Cloud Interconnect (managed by our DevOps team) to set up a high-bandwidth, low-latency private connection between our on-prem Hadoop cluster and Google Cloud.
-
For initial setup, we created a GCP service account, generated a key, and updated
core-site.xmlto allow Hadoop to authenticate and access GCS via the GCS connector. -
We used Hadoop DistCp to migrate ~200 TB of data from HDFS to Google Cloud Storage (GCS).
-
We scheduled nightly syncs of ~2 TB over Interconnect to keep GCS in sync with our on-prem data.
- if someone ask how did you optimise your query processing time? then we can say, we use columnar storage and within the columnar storage whenever i store the data, i partition the data aswell...
When to Use
hdfs fsck?✅ If you suspect data corruption in HDFS
✅ To check for missing or under-replicated blocks
✅ Before running HDFS recovery or repairs
✅ To diagnose DataNode failuresHDFS can distribute files into blocks and cannot understand the tabular format... that’s why we use Hive..."
✔️ That’s correct — just clarify that Hive adds schema + SQL on top of HDFS dat
========================================
Partitioning Data (in Storage) - A storage technique to divide files based on a column (e.g. date, region)..Query performance by scanning only relevant data...
hadoop hdfs only supports files and dsitrbuted them across the nodes as blocks...it doesnt understand the file tabular strutures, thats why we use hive to bult schmea on top of hdfs using int/managed or ext table to perform queries..
hdfs dfs -put localpathfile to hdfspath...so this command will distibute the filw into blocks
hdfs dfs -get ..so we can convert hdfs distibuted files into one single file
- suppose we have 10GB text file, there are other formats that can read the data fast with different formats..so inrder to process the data there are different tech used by file formats like ORC parquet pr AVRO to speed the process...so before that we need to ask if we want the data to be wrtten or queried fast? so writing is fast in AVRO i.e write trhe data into hdfs storage as it zips it (i.e convert text file into AVRO and then write data into stirage)..and for processing we use ORC or parquet....parquet is more preferbale for new engines like spark...but ORC better for hive
- it is imp for interviews..ROW storage (rdbms like oracle, mysql) vs Columnar storage (datawarehouse)data is tored in row wise and in coumnar each column is stored in each block...so wwhen we execute a query, it will check the where clause where country = US, it will pick up that block and no need to scan all other data, so the index will be picked from country = USso data processing is fast in columar data
- in row we do sequential scan
- latency - how much time it taking to process the data...or reduce the processing time is our duty as dataenggin cv we can write - optimised data pipeline latencies
- f someone ask how did you optimise your query processing time? then we can say, we use columnar storage and within the columnar storage whenever i store the data, i partition the data aswell...