You ship a feature. The dashboard looks green. Throughput is up, CPU is efficient, and the system handles more requests per second than last quarter. Then support pings you: “Users say checkout feels slow.”
That is the latency vs throughput trap.
Latency is how long one unit of work takes. Throughput is how much work the system completes over time. You can improve one and hurt the other. Batch more work, and throughput often rises while individual requests wait longer. Process everything immediately, and latency drops until your workers spend half their time context switching.
Google’s SRE teams have long argued that you need to watch latency, traffic, errors, and saturation together, not as isolated metrics. Latency tells you user pain, traffic tells you demand, errors tell you correctness, and saturation tells you when the system is running out of room. AWS performance guidance echoes the same principle: architecture decisions should reflect workload-specific latency, throughput, jitter, and bandwidth requirements. Netflix engineers have also shared cases where removing a bottleneck improved throughput dramatically while simultaneously reducing average and tail latency.
Stop Treating Latency and Throughput Like Opposites
Latency and throughput are not enemies. They are coupled variables.
The simplest mental model is Little’s Law: concurrency = throughput × latency. If your API serves 500 requests per second and the average latency is 200 ms, you have about 100 requests in flight. Push throughput to 1,000 requests per second without reducing latency, and now you need about 200 concurrent requests. That means more memory, more sockets, more queue depth, and more failure surface.
Here is the practical version: throughput gains are only “free” until they create queues. After that, every extra request waits behind earlier work. Latency does not rise politely. It bends upward, then spikes.
Use the Right Metric for the Job
Average latency is the performance dashboard equivalent of “the water is fine.” Maybe it is, unless you are the user stuck in the p99.
| Metric | What it tells you | When it matters most |
|---|---|---|
| p50 latency | Typical user experience | Everyday responsiveness |
| p95 latency | Slow-but-common requests | SLOs and product UX |
| p99 latency | Tail pain | Payments, search, login |
| Throughput | Completed work per second | Scaling and cost efficiency |
| Saturation | Resource pressure | Capacity planning |
For user-facing systems, optimize around percentiles. A 100 ms average with a 2-second p99 still feels broken to a meaningful slice of users. For offline jobs, throughput might matter more than tail latency, especially when the user never waits for the result.
Pick the Constraint Before You Tune
Most teams tune too early. They add Redis, increase worker counts, widen connection pools, or shard a database before naming the actual constraint.
Ask one question first: What are we optimizing for, and at what limit?
For a checkout API, you might set a target like: “p95 under 300 ms at 800 RPS with error rate below 0.1%.” For a video transcoding pipeline, the target might be: “process 10,000 files per hour at the lowest stable cost.” Those are very different systems. The first protects interaction quality. The second protects batch completion time and unit economics.
One AWS engineering example is especially instructive. Engineers optimized Amazon SQS internals and managed to reduce both average and tail latency while improving scalability for downstream systems. The lesson was simple: architectural bottlenecks often punish both latency and throughput at the same time.
Balance the Tradeoff in Five Moves
First, define separate SLOs for latency and throughput. Do not say “make it fast.” Say “p95 below 250 ms at 2,000 RPS.” That forces the conversation out of vibes and into engineering.
Second, load test until the knee of the curve. The knee is where throughput keeps rising, but latency starts climbing sharply. Operate below that point, not on top of it.
Third, use queues intentionally. Queues smooth bursts and improve throughput, but they also hide latency. Track queue depth and time-in-queue, not just worker utilization.
Fourth, batch only where waiting is acceptable. Batching database writes, logs, analytics events, and model inference can improve throughput. Batching login, checkout, or autocomplete usually punishes the user.
Fifth, add backpressure before collapse. Rate limits, circuit breakers, bounded queues, and load shedding keep the system honest when demand exceeds capacity.
A Worked Example: The Queue That Looked Fine
Imagine a service handling 1,000 requests per second with 100 ms average latency.
Using Little’s Law:
Concurrency = Throughput × Latency
Concurrency = 1,000 RPS × 0.1 seconds
Concurrency = 100 in-flight requests
Now traffic doubles to 2,000 RPS. If latency stays at 100 ms, you need 200 in-flight requests. Fine, maybe.
But if database contention pushes latency to 400 ms, concurrency becomes:
2,000 × 0.4 = 800 in-flight requests
That is not just “four times slower.” It is eight times the in-flight load from the original system. This is how healthy services become incident channels.
Know Which Lever You Are Pulling
Caching usually reduces latency and increases throughput because fewer requests hit the bottleneck. Replication can improve read throughput, but it may add consistency complexity. Batching raises throughput, but it adds waiting time. Compression reduces bandwidth pressure, but it adds CPU cost. Async processing improves perceived latency, but only if users do not need the result immediately.
This is the real performance craft: every lever has a bill.
The article examples you provided lean into the same useful pattern: define the concept plainly, show the mechanism, then give operators a way to act without pretending there is a magic fix. That structure fits performance engineering well because latency and throughput problems rarely have one-click answers.
FAQ
Is latency more important than throughput?
For user-facing paths, usually yes. For batch processing, analytics, streaming ingestion, and backups, throughput often matters more.
Can you improve both at once?
Yes, when you remove waste: bad queries, lock contention, chatty network calls, cold starts, inefficient serialization, or poor memory behavior.
What is the biggest mistake teams make?
Optimizing average latency while ignoring p95 and p99. Users feel the tail.
When should you scale horizontally?
After you know the bottleneck. Scaling workers against a saturated database often just creates a larger traffic jam.
Honest Takeaway
Latency is the user’s stopwatch. Throughput is the business’s capacity meter. You need both, but you cannot balance them until you define the workload, measure percentiles, and find the knee of the curve.
The best systems do not chase “fast.” They choose where waiting is acceptable, where it is fatal, and where extra throughput is worth the cost.
