Latency vs. Throughput: System Performance

📋 Overview

Latency and Throughput are the two primary pillars of system performance. While they are related, they represent different perspectives on "speed." Optimizing for one often involves trading off the other. A high-performance system must balance low-latency responses for individual users with high-throughput capacity for the entire user base.

---

🏗️ Core Principles & Characteristics

• Latency: The time delay for a single request to travel from source to destination and back (Round Trip Time). Measured in ms.
• Throughput: The total volume of work completed per unit of time (e.g., 10,000 Requests Per Second). Measured in RPS, TPS, or Gbps.
• The Pipeline Analogy:
  • Latency is the time it takes for a single drop of water to travel through the pipe.
  • Throughput is the width of the pipe (how much water flows out per second).

---

⚖️ Trade-offs: Pros & Cons

• Batching:
  • Pros: Increases Throughput (processing 1,000 items in one go is more efficient).
  • Cons: Increases Latency (the 1st item must wait for the 1,000th item before processing starts).
• Asynchronous Processing:
  • Pros: Improves perceived latency for the user.
  • Cons: The actual throughput remains limited by the background worker capacity.

---

🌍 Real-World Implementation

• Gaming: Latency (Lag) is the most critical metric. 100ms vs 200ms determines if a game is playable.
• Video Streaming: Throughput is critical. You need to download 5MB every second to keep the video playing, but it doesn't matter if the first byte took 500ms to arrive.
• Databases: Analytical (OLAP) DBs focus on throughput (scanning billions of rows); Transactional (OLTP) DBs focus on latency (updating one row in 1ms).

---

💡 Interview "Gotchas" & Tips

• The "Slow Pipe" Fallacy: You can have a high-bandwidth (throughput) 1Gbps connection but if it's via satellite, the latency will be 600ms. It's like a massive truck moving at 5mph—lots of data, but very slow.
• Little's Law: L = λ * W. Number of requests in a system (L) equals the arrival rate (λ) times the average time spent in the system (W).
• Tail Latency (P99): Always mention that "average" latency is misleading. Users care about the 99th percentile—the slowest 1% of requests.

---

📐 Suggested Architecture Primitives

• CDN: To reduce latency by moving data closer to the user.
• Message Queues: To handle high-throughput spikes by buffering.
• Batch Processors: To maximize throughput for non-time-sensitive data.
• Load Balancers: To scale throughput horizontally.