Theory
System Resiliency & Fault Tolerance
📋 Overview
Resiliency is a system's ability to remain functional despite the failure of individual components. In a distributed environment, failures (network partitions, hardware crashes, dependency outages) are not just possible—they are inevitable. A resilient architecture focuses on Fault Isolation and Graceful Degradation, ensuring that a failure in a non-critical service (like "Recommended Products") doesn't crash the core business path (like "Add to Cart").
🏗️ Core Principles & Characteristics
- Reliability vs. Resiliency: Reliability is the probability that a system won't fail. Resiliency is how well it recovers when it inevitably does.
- Redundancy: Eliminating Single Points of Failure (SPOF) by having multiple instances of every component across different Availability Zones (AZs).
- Self-Healing: The system's ability to detect a failure (via health checks) and automatically take corrective action (restarting a pod, failing over to a backup).
- Observability: You cannot be resilient if you are blind. Real-time metrics and alerts are prerequisites for any resilient design.
⚖️ Trade-offs: Pros & Cons
- Pros: Drastically higher uptime; improved user trust; reduced "on-call" fatigue for engineers; protects business revenue.
- Cons: Significantly higher infrastructure costs (due to redundancy); increased complexity (circuit breakers and failovers add logic that must be tested); "Resiliency testing" (Chaos Engineering) can be risky.
🌍 Real-World Implementation
- Netflix (Chaos Monkey): Famously shuts down production instances randomly to ensure their systems can handle real-world turbulence.
- Resilience4j / Hystrix: Java libraries used to implement Circuit Breaker, Bulkhead, and Rate Limiter patterns in microservices.
- Kubernetes (Liveness/Readiness Probes): Automatically kills and restarts containers that stop responding, providing foundational self-healing.
- Multi-Region Failover: Routing 100% of traffic from
us-east-1tous-west-2when a major cloud provider outage occurs.
💡 Interview "Gotchas" & Tips
- Cascading Failures: Explain how a single slow service can consume all available threads in a thread pool, causing every other service to hang. Mention Bulkheads as the solution.
- Circuit Breakers: Explain the states: Closed (Normal), Open (Failing - stop calling), and Half-Open (Testing for recovery).
- Graceful Degradation: In an interview, suggest "returning default results" or "cached data" instead of a 500 error when a dependency fails.
📐 Suggested Architecture Primitives
- Circuit Breaker Pattern: To prevent a failing service from "poisoning" the rest of the system.
- Bulkhead Pattern: Isolating resources (thread pools, CPUs) for different services so one doesn't starve the others.
- Exponential Backoff + Jitter: When retrying failed requests to avoid the "Thundering Herd" problem.
- Load Shedding: Intentionally dropping requests when the system is near its breaking point.
Canvas