Theory
Low-Latency Design Patterns
📋 Overview
Low-latency design is the art of minimizing the "Time-to-Response" across every layer of the technology stack. In industries like High-Frequency Trading (HFT), real-time bidding (RTB), and competitive gaming, latency is often measured in microseconds, and even a 10ms delay can result in significant business loss.
🏗️ Core Principles & Characteristics
- Mechanical Sympathy: Understanding the underlying hardware (L1/L2 caches, NUMA) to write code that aligns with CPU architecture.
- Zero-Copy Networking: Passing data pointers through the stack instead of copying byte arrays between memory buffers (e.g., using
sendfileormmap). - Data Locality: Keeping frequently accessed data in CPU caches or near-memory (RAM) to avoid the "Memory Wall" (the speed gap between CPU and Main Memory).
- Non-Blocking I/O: Using event loops (e.g., Epoll, Kqueue) to handle thousands of connections without the overhead of context-switching between threads.
⚖️ Trade-offs: Pros & Cons
- In-Memory Processing:
- Pros: Sub-microsecond access times.
- Cons: High volatility; requires complex snapshots/Write-Ahead Logs (WAL) for durability.
- Asynchronous Processing:
- Pros: High throughput; prevents "Head-of-Line" blocking.
- Cons: Increases system complexity and makes debugging distributed traces harder.
- Protocol Choice (Binary vs. Text):
- Pros (Binary): Faster serialization/deserialization (Protobuf/Avro).
- Cons (Binary): Not human-readable; requires schema management.
🌍 Real-World Implementation
- Financial Trading: Uses LMAX Disruptor (a lock-free inter-thread communication library) and kernel bypass techniques (like Solarflare OpenOnload) to achieve sub-10 microsecond latencies.
- CDNs (Content Delivery Networks): Use Anycast IP routing and Edge Computing (e.g., Cloudflare Workers) to process requests within 50 miles of the user.
- Databases: Redis and Aerospike utilize shared-nothing architectures and in-memory storage to serve millions of operations per second with sub-millisecond p99s.
💡 Interview "Gotchas" & Tips
- The P99 Delusion: Never talk about "Average Latency." Always focus on the Tail Latency (P99/P99.9). A system with a 10ms average but a 2s P99 is "jittery" and broken for many users.
- Garbage Collection (GC) Pauses: In languages like Java/Go, "Stop-the-World" GC is the enemy of low latency. Mention "Zero-allocation" coding or using off-heap memory.
- Context Switching: Explain how having too many threads causes the CPU to spend more time "switching" than "working." Suggest thread-per-core architectures.
📐 Suggested Architecture Primitives
- Ring Buffers: For lock-free communication between producers and consumers.
- Sidecar Proxies (Envoy): For low-latency service-to-service communication with built-in circuit breaking.
- Read Replicas & Local Caching: To eliminate cross-region network hops for read-heavy workloads.
- Batching with Timeout: Collecting small requests into a single network packet to optimize throughput while capping latency.
Canvas