Theory
Leader Election in Distributed Systems
📋 Overview
Leader election is the process of designating a single node as the organizer or "coordinator" of some task distributed among several nodes. It is a critical mechanism for ensuring data consistency and avoiding "Split-Brain" scenarios in distributed databases, job schedulers, and coordination services.
🏗️ Core Principles & Characteristics
- The Leader: Handles writes, coordinates state, and assigns tasks.
- The Followers/Standbys: Replicate the leader's state and stand ready to take over.
- Consensus Algorithms: The "voting" mechanisms used to elect a leader:
- Raft: Modern, easier to understand, used by etcd (Kubernetes) and Consul.
- Paxos: The original complex consensus algorithm used by Google (Chubby).
- Heartbeats: The leader constantly sends "I'm alive" signals. If they stop, a new election is triggered.
⚖️ Trade-offs: Pros & Cons
- Pros:
- Strict Consistency: Only one node makes decisions, eliminating conflicts.
- High Availability: Automatic failover ensures the system survives node crashes.
- Cons:
- Complexity: Implementing a bug-free consensus algorithm is notoriously difficult.
- Performance Hit: Elections take time (usually 100ms - 1s), during which the system may be "read-only" or unavailable.
🌍 Real-World Implementation
- Kubernetes (etcd): Uses Raft to elect a leader that manages the cluster state.
- Kafka: Uses Zookeeper (or KRaft) to elect a "Controller" broker to manage partition assignments.
- MongoDB: Replica sets use a Raft-like protocol to elect the "Primary" node.
💡 Interview "Gotchas" & Tips
- The "Split-Brain" Problem: What if the network splits and two nodes think they are the leader? Solution: Quorum (Majority). A leader must have more than 50% of the votes to act.
- Zookeeper vs. Raft: Zookeeper uses Ephemeral Nodes (if node crashes, node disappears) for election. Raft uses Term Numbers and Votes.
- Fencing Token: When a new leader is elected, it gets a "fencing token" (an incrementing number). The DB will ignore requests from the old leader because its token is too low.
📐 Suggested Architecture Primitives
- etcd / Zookeeper: Dedicated coordination services.
- Consensus Protocol (Raft/Paxos): The logic for voting.
- Quorum (n/2 + 1): The minimum number of nodes for a valid election.
- Heartbeat Mechanism: For health monitoring.
Canvas