Theory
Fault Tolerance & System Resilience
📋 Overview
Fault Tolerance is the property that enables a system to continue operating properly in the event of the failure of one or more of its components. It is not about preventing failures (which is impossible at scale), but about surviving them without significant service degradation or data loss.
🏗️ Core Principles & Characteristics
- Redundancy: Having "spare" components (servers, databases, power supplies) ready to take over.
- Replication: Maintaining multiple copies of data across different disks or geographic regions.
- Isolation (Bulkheads): Segmenting the system so a failure in one area doesn't cascade to others.
- Graceful Degradation: The ability to maintain core functionality while non-essential features fail (e.g., Netflix playing video without recommendations).
- Self-Healing: Automated detection and recovery (e.g., Kubernetes restarting a crashed container).
⚖️ Trade-offs: Pros & Cons
- Pros:
- High Availability: Keeps the business running 24/7.
- Customer Trust: Minimizes user-facing errors and downtime.
- Operational Peace of Mind: Automation handles routine failures without waking up engineers.
- Cons:
- Cost: Redundancy typically doubles or triples infrastructure spend.
- Complexity: Managing distributed state and failover logic is difficult.
- Performance: Synchronous replication for fault tolerance can add latency.
🌍 Real-World Implementation
- Netflix Chaos Monkey: A tool that intentionally kills production instances to test fault tolerance.
- Hystrix/Resilience4j: Libraries that implement Circuit Breakers to stop cascading failures.
- RAID: Redundant Array of Independent Disks for hardware-level fault tolerance.
- Availability Zones (AZs): Datacenters within a region connected by low-latency fiber to survive facility-level disasters.
💡 Interview "Gotchas" & Tips
- FT vs. HA: Fault Tolerance (zero downtime) is harder and more expensive than High Availability (minimal downtime).
- Single Point of Failure (SPOF): Always look for the one component (e.g., a single Load Balancer or DB) that can take down the whole system.
- Byzantine Fault Tolerance: Handling nodes that don't just fail but act maliciously or send "garbage" data (Blockchain context).
- MTBF & MTTR: Mean Time Between Failures and Mean Time To Repair.
📐 Suggested Architecture Primitives
- Load Balancers: To route traffic away from failed nodes.
- Circuit Breakers: To prevent cascading failures.
- Active-Passive / Active-Active Clusters: For redundant processing.
Canvas