Module 01 // FoundationsLesson 03 / 3414 min read
Latency vs throughput
Latency is how long one request takes; throughput is how many requests you can finish per second. You need both, and improving one doesn't always help the other.
Think of it like this
Analogy
Think of a coffee shop. Latency is how long you wait from ordering to getting your drink. Throughput is how many drinks the shop hands out per hour.
Hiring a second barista roughly doubles throughput, but your own latte still takes three minutes to make. A faster espresso machine cuts your wait and also lets the shop serve more people. And when the line is out the door, your wait grows even though nobody is working any slower.
The problem it solves
“Make it faster” is ambiguous. Users feel latency: the time between a click and a result. The business needs throughput: enough capacity for every user at peak. Mixing them up leads to the wrong fix. Adding servers raises throughput but won't speed up a slow database query. Adding a cache speeds up reads but won't help if you're out of CPU.
Averages make it worse. A 100 ms average can hide one request in a hundred taking 3 seconds, and those are the ones users remember.
A real example
Example
The morning rush at a coffee shop
One barista takes about 3 minutes per drink. That's the latency: the time from ordering to holding your cup. Working flat out, they make 20 drinks an hour. That's the throughput.
At 8:30 the line reaches the door. Hiring a second barista doubles throughput to 40 drinks an hour, but your latte still takes 3 minutes to make. What it does fix is the wait in line, which was most of your total time.
Then someone orders 12 drinks for their office. Everyone behind them waits 30 extra minutes. The average wait barely moves, but the unlucky customers at the slow end, the p99, have a terrible morning. That's why you watch percentiles and not just averages.
How it works
Measuring latency: use percentiles
Sort all response times from fastest to slowest. The p50 (median) is the middle one. The p99 is the one that's slower than 99% of the others. Percentiles show the experience of real users; averages don't.
| Percentile | Meaning | Typical target for a web API |
|---|---|---|
| p50 | Half of requests are faster | 50–100 ms |
| p95 | 1 in 20 is slower | 200–300 ms |
| p99 | 1 in 100 is slower | under 500 ms–1 s |
The tail matters more than it looks. If one page calls 30 services and each has a 1% chance of being slow, about a quarter of page loads hit at least one slow call.
Numbers every engineer should know
These are rough orders of magnitude, not exact timings. They change with hardware, but the ratios between them hold:
| Operation | Rough time |
|---|---|
| Read from CPU cache | about 1 nanosecond |
| Read from main memory | about 100 nanoseconds |
| Read 1 MB sequentially from memory | a few microseconds to tens of microseconds |
| Read from a fast SSD | tens to hundreds of microseconds |
| Round trip inside one data center | about 0.5 ms |
| Read from an in-memory cache over the network | about 0.5–1 ms |
| Simple indexed database query | about 1–10 ms |
| Round trip across a continent | about 40–80 ms |
| Round trip between continents | about 100–200 ms |
The lesson: memory is thousands of times faster than the network, and the network inside a data center is hundreds of times faster than across the world. Good designs keep hot data in memory and keep users close to servers.
Throughput, concurrency and Little's law
If each request takes 100 ms and a server can work on 50 at once, it can finish about 50 ÷ 0.1 = 500 requests per second. That's Little's law: requests in progress = arrival rate × time each one takes.
It also explains why systems fall over. When requests arrive faster than they finish, they queue up. Each request now waits behind others, so latency grows, which means more are in progress, which uses more memory and threads, which slows everything further.
Bandwidth is not latency
Bandwidth is how much data a link carries per second. A 1 Gbps link between continents still has 150 ms round trips. For many small requests, latency dominates; for big transfers like video, bandwidth does.
How to improve each
| To lower latency | To raise throughput |
|---|---|
| Cache hot data in memory | Add servers (horizontal scaling) |
| Move servers or a CDN closer to users | Handle requests concurrently |
| Add the right database index | Batch small operations together |
| Do slow work in the background (queues) | Remove the bottleneck (often the database) |
| Cut round trips: fewer, bigger requests | Use a queue to smooth out spikes |
Some fixes trade one for the other. Batching raises throughput but makes each item wait for its batch. Queues protect throughput but add delay before work is done.
Trade-offs
What you gain
- Tracking p50, p95 and p99 tells you what users actually experience.
- Knowing the rough cost of memory, disk and network lets you spot bad designs before you build them.
- Separating latency and throughput problems points you at the right fix.
What it costs you
- Optimizing for peak throughput (batching, queuing) can raise latency for each request.
- Chasing p99 is expensive; the last few percent often need redundancy like hedged requests.
- Running servers near users everywhere costs more and makes data harder to keep consistent.
When not to use it
- Don't optimize latency below what users can notice. Under about 100 ms, interactions already feel instant.
- Don't buy throughput for traffic you don't have. Measure first.
In the cloud
Monitoring, logging and tracingEvery major cloud sells this as a managed service, so you configure it rather than build it.
AWS2
- Amazon CloudWatch (opens vendor docs)
Metrics, logs, alarms and dashboards for AWS resources and your apps.
- AWS X-Ray (opens vendor docs)
Distributed tracing. AWS now recommends OpenTelemetry instrumentation; the X-Ray SDKs are in maintenance mode.
Azure2
- Azure Monitor (opens vendor docs)
Collects metrics and logs from Azure resources, with alerts and dashboards.
- Application Insights (opens vendor docs)
Application performance monitoring and distributed tracing, built on OpenTelemetry.
Google Cloud3
- Cloud Monitoring (opens vendor docs)
Metrics, uptime checks, dashboards and alerting.
- Cloud Logging (opens vendor docs)
Stores, searches and analyzes logs.
- Cloud Trace (opens vendor docs)
Distributed tracing for latency analysis.
Apps emit metrics, logs and traces (ideally with OpenTelemetry); dashboards and alerts sit on top.
- App with OpenTelemetry
- CloudWatch metrics + logs
- X-Ray traces
- CloudWatch alarms
Interview questions
Try answering out loud first, then open the model answer.
What's the difference between latency and throughput? Can you improve one and hurt the other?
Model answer
Latency is the time for one operation; throughput is operations per unit of time. They often move together, but not always. Batching writes raises throughput because there's less overhead per item, but each item waits for its batch, so latency rises. Adding servers raises throughput without changing the latency of a single request. A cache usually improves both because it removes slow work.
Why do we track p99 latency instead of the average?
Model answer
Averages hide the slow tail. A service can average 50 ms while 1% of requests take 3 seconds. Those slow requests often hit your most valuable users, such as the ones with the most data, and when one page fans out to many services, the chance of hitting at least one slow call grows quickly. p99 (and p99.9 for large systems) shows that pain directly, and SLOs are usually written against it.
Your service's p99 jumped from 200 ms to 2 s, but traffic is normal. Where do you look?
Model answer
I'd check what changed: deploys, configuration and dependencies. Then I'd look at traces of slow requests to see which step got slower: the database (lock contention, a missing index, a bad query plan), a downstream service, garbage-collection pauses, or a saturated resource such as a connection pool. I'd compare slow and fast requests by endpoint, customer and server to see whether it's everywhere or one hot partition or one bad host.
Quiz
Pick an answer to see whether it’s right and why.
0 of 7 answered
Key terms
- Latency
- How long one request takes from start to finish, usually measured in milliseconds.
- Throughput
- How many requests a system can handle per unit of time, such as requests per second.
- Percentile (p50, p99)
- The p99 latency is the time that 99% of requests beat. It shows how slow your slowest users' requests are.
- Bandwidth
- How much data a link can carry per second, such as 1 Gbps.
- Round-trip time (RTT)
- How long a message takes to reach the other side and for the reply to come back.
Progress is saved in this browser only.