Theory
Distributed Locking with ZooKeeper, etcd & Consul
📋 Overview
Distributed locking is a mechanism used to coordinate access to shared resources across multiple nodes in a distributed system. Unlike local locks (mutexes), distributed locks must be resilient to network partitions and node failures. Coordination services like ZooKeeper, etcd, and Consul are purpose-built to provide the strong consistency (CP in CAP) required for safe locking.
🏗️ Core Principles & Characteristics
- Strong Consistency: These systems use consensus protocols (Zab for ZK, Raft for etcd/Consul) to ensure a single, consistent view of the lock state.
- Ephemeral Nodes (ZooKeeper): A client creates a temporary node. If the client's session ends (e.g., crash), the node is deleted, automatically releasing the lock.
- Leases & Sessions (etcd/Consul): Locks are tied to a TTL-bound lease. The client must "heartbeat" to keep the lease alive.
- Watch Mechanism: Instead of polling, clients can "watch" a lock key and be notified immediately when it is released.
- Linearizability: Guarantees that operations appear to happen instantaneously in a specific order.
⚖️ Trade-offs: Pros & Cons
- Pros:
- High Reliability: Strong guarantees against race conditions.
- Liveness: Automatic cleanup via ephemeral nodes/leases prevents deadlocks.
- Fairness: Can implement FIFO locks using sequential node ordering.
- Cons:
- Latency: Every lock acquisition requires a consensus round-trip (slower than Redis).
- Operational Complexity: Managing a ZK or etcd cluster is non-trivial.
- Throughput Limits: Not suitable for millions of locks per second due to the overhead of consensus.
🌍 Real-World Implementation
- Kubernetes: Uses etcd for leader election among controllers and cluster state locking.
- Apache Kafka: Historically used ZooKeeper for controller election and partition ownership (now moving to KRaft).
- Service Discovery: Consul uses locks to manage service leadership and configuration updates.
💡 Interview "Gotchas" & Tips
- Session Expiry vs. GC Pauses: If a Java app has a long Stop-The-World GC pause, the ZK session might expire, and the lock could be released while the app thinks it still holds it.
- Fencing Tokens: To solve the above, use an incrementing "fencing token" (like ZK's
zxid) that the resource (e.g., a DB) checks to ensure the writer is still the valid lock holder. - Thundering Herd: Explain how using "Watch" on a single node can cause a spike in traffic when the lock is released; sequential nodes (queuing) solve this.
- Wait-Free vs. Blocking: Understand the difference between trying to acquire and waiting in a queue.
📐 Suggested Architecture Primitives
- ZooKeeper Quorum: A minimum of 3 or 5 nodes to ensure availability.
- Ephemeral Sequential Nodes: For fair, distributed queuing.
- Raft Consensus Group: The underlying mechanism for etcd and Consul consistency.
Canvas