Theory
Hot Keys in Distributed Systems
📋 Overview
A "Hot Key" occurs when a specific data item (a key in a cache or a row in a database) receives a disproportionately high volume of traffic compared to other keys. In a distributed system, this leads to load imbalance where one shard or node is overwhelmed (the "hotspot") while others remain idle, often resulting in latency spikes, node crashes, or cascading failures.
🏗️ Core Principles & Characteristics
- Disproportionate Access: A "Pareto Principle" (80/20 rule) often applies, but in extreme cases (e.g., a celebrity profile), a single key can represent 90%+ of a node's traffic.
- Sharding Bottleneck: Because sharding is often based on
hash(key) % nodes, all requests for that one key land on exactly one node, bypassing the benefits of horizontal scaling. - Resource Contention: High CPU for request processing, memory pressure for caching the same item, and network bandwidth saturation.
⚖️ Trade-offs: Pros & Cons
- The Problem:
- Latency Spikes: Queuing at the hot node increases response times for all keys on that node.
- Cascading Failure: If the hot node crashes, the traffic may shift to a replica or the database, potentially taking them down too.
- The Solutions:
- Pros of Replication: Spreading the key across multiple nodes reduces per-node load.
- Pros of Client-side Caching: Eliminates the request before it even hits the distributed cache.
🌍 Real-World Implementation
- Social Media: A viral post or celebrity profile (Twitter/X, Instagram).
- E-commerce: A flash sale item or a "Deal of the Day" (Amazon/Flipkart).
- Gaming: A global leaderboard key or a popular game lobby ID.
- Solution: Hashing with Suffixes: Append a random number to the hot key (e.g.,
user_123_1,user_123_2) and replicate the data across those keys to spread the load.
💡 Interview "Gotchas" & Tips
- Consistent Hashing doesn't fix hot keys: It only fixes the reshuffling of keys when nodes change. It doesn't help if millions of people want the same key.
- Adaptive Caching: Mention systems like Facebook's Scuba or YouTube's caching layers that automatically identify hot keys and move them to L1 (local) caches.
- Write-heavy Hot Keys: If the hot key is being updated (e.g., a like counter), mention In-Memory Aggregation (buffer the likes for 1 second, then write the sum of +100 to the DB).
📐 Suggested Architecture Primitives
- Local L1 Cache: Cache the hot key on the application server itself.
- Read Replicas: Scale reads by duplicating the data.
- Rate Limiter: Protect the hot node from being overwhelmed.
- Message Queue: Buffer writes to a hot key to prevent DB lock contention.
Canvas