Theory
Data Recovery & Durability
📋 Overview
Data Recovery is the process of restoring lost, corrupted, or inaccessible data to its original state. In modern system design, recovery isn't just about backups; it's about building "Durability"—the guarantee that once data is committed, it survives system failures, disk crashes, and even entire regional outages. This involves a combination of database internals (WAL), distributed replication, and multi-tier backup strategies.
🏗️ Core Principles & Characteristics
- Write-Ahead Logging (WAL): Every change is written to a sequential log on disk before it is applied to the data files. This allows the DB to "replay" the log after a crash.
- RPO (Recovery Point Objective): The maximum amount of data loss you can tolerate (measured in time, e.g., "We can lose 15 minutes of data").
- RTO (Recovery Time Objective): The target time to restore service after a failure (e.g., "The site must be back up in 2 hours").
- Checkpoints: Periodically flushing in-memory data to disk to reduce the amount of WAL that needs to be replayed during recovery.
- Replication (Sync vs Async): Copying data to other nodes/regions to ensure a single disk failure isn't fatal.
⚖️ Trade-offs: Pros & Cons
- Synchronous Replication:
- Pros: Zero data loss (RPO = 0).
- Cons: High latency (write must wait for all replicas); reduced availability (if a replica is down, writes fail).
- Asynchronous Replication:
- Pros: Low latency; high availability.
- Cons: Risk of data loss during a primary failure (the "Replication Gap").
- Full vs. Incremental Backups: Full backups are easy to restore but slow to take; incremental backups are fast to take but require replaying multiple files to restore.
🌍 Real-World Implementation
- PostgreSQL/MySQL: Use WAL and automated snapshots (like AWS RDS Multi-AZ) to handle recovery.
- Redis: Uses AOF (Append-Only File) for durability and RDB (Snapshots) for fast restarts.
- S3 Versioning: Protects against accidental deletion or ransomware by keeping old versions of files.
- Point-in-Time Recovery (PITR): Allows you to restore a database to the exact second before a "Bad Update" occurred by combining a full backup with WAL logs.
💡 Interview "Gotchas" & Tips
- "Committed" vs. "Saved": Just because an app says "Success" doesn't mean the data is durable. It must be
fsynced to non-volatile storage. - The "Corruption" Risk: Replication is NOT a backup. If you run
DELETE FROM users, that deletion is instantly replicated to all nodes. You need a Backup to recover from human/logic errors. - Testing Recovery: A backup is useless unless you have tested the Restore process. Mention "Game Days" or automated restore tests.
- Air-Gapped Backups: Storing a copy of data in a completely disconnected account/system to prevent ransomware from deleting your backups.
📐 Suggested Architecture Primitives
- WAL (Write-Ahead Log): For crash recovery.
- Snapshot Service: (AWS EBS Snapshots) for periodic full backups.
- Cross-Region Replication: For disaster recovery (DR).
- Checksums: To detect silent data corruption (bit rot) at rest.
- Object Versioning: To prevent data loss from accidental overwrites.
Canvas