Theory
HDFS (Hadoop Distributed File System)
📋 Overview
HDFS is a distributed, scalable, and fault-tolerant file system designed to run on commodity hardware. It is the primary storage layer for the Apache Hadoop ecosystem, optimized for high-throughput access to large datasets (Petabytes). Unlike traditional file systems, HDFS follows a "Write Once, Read Many" philosophy, making it ideal for batch processing.
🏗️ Core Principles & Characteristics
- Master-Slave Architecture:
- NameNode (Master): Manages the namespace, file metadata (names, hierarchy), and maps blocks to DataNodes. Kept entirely in RAM for speed.
- DataNode (Slave): Stores the actual data blocks and performs read/write operations.
- Block-Based Storage: Files are split into large blocks (default 128MB) to reduce NameNode metadata overhead.
- Replication Strategy: Each block is replicated (default 3x) across different racks to ensure data durability and rack awareness.
- Data Locality: HDFS encourages moving computation to the data (e.g., MapReduce) rather than moving data to the computation.
⚖️ Trade-offs: Pros & Cons
- Pros:
- Fault Tolerance: Automatic recovery from node or rack failure.
- Huge Scale: Handles millions of files and petabytes of data.
- Streaming Data Access: Optimized for batch throughput rather than low-latency random access.
- Cons:
- The "Small File Problem": Millions of small files overwhelm the NameNode's RAM.
- High Latency: Not suitable for real-time applications or transactional databases.
- Single Master Bottleneck: While NameNode HA exists, it remains a central point of complexity.
🌍 Real-World Implementation
- Big Data Warehousing: Storing raw logs for processing by Spark or Hive.
- Machine Learning: Feeding massive training sets into distributed ML pipelines.
- Archival Storage: Cost-effective long-term storage of cold data using commodity hardware.
- Cloud Alternatives: AWS S3, Google Cloud Storage (GCS), and Azure Blob Storage are often used as HDFS replacements in cloud-native architectures.
💡 Interview "Gotchas" & Tips
- Secondary NameNode is NOT a Standby: It does NOT provide high availability. It performs "checkpointing" (merging Edit Logs and FsImage) to speed up NameNode restarts. For HA, you need a Standby NameNode using Shared Edits (Quorum Journal Manager).
- Data Consistency: HDFS is eventually consistent for metadata but provides strong consistency for block writes once closed.
- Rack Awareness: Explain how HDFS places replicas: 1st on local node, 2nd on a different rack, 3rd on a different node in that second rack.
📐 Suggested Architecture Primitives
- NameNode: Metadata manager.
- DataNode: Block storage nodes.
- Quorum Journal Manager (QJM): For NameNode High Availability.
- HDFS Federation: To scale the namespace beyond a single NameNode.
Canvas