Understanding Little's law, response time percentiles, and tail latency amplification.
This trade-off is often called latency versus throughput, after the primer's name for it. This lesson uses the words as Designing Data-Intensive Applications does: latency is only the waiting, and response time is the whole time the user waits.
What you will learn
Define response time, service time, latency and throughput, and the units each is measured in.
Use W=1−ρ1/μwhat its symbols mean to see how waiting grows with utilization.
Size a pool with Little's law, and explain why p99 matters more than the average.
Precise definitions
Response time: The elapsed time from the moment a client sends a request until it receives the answer, measured in milliseconds (ms). It is what the user experiences.
Service time: The part of the response time during which a server is actively working on the request.
Latency: The part during which the request waits and nothing is working on it, such as time in a queue or crossing the network. Many people say latency for the whole response time, and common phrases such as tail latency keep that looser sense.
Throughput: The rate at which the system successfully processes operations or data, measured in requests per second (RPS) or bytes per second (GB/s).
You should aim for maximal throughput with an acceptable response time. As utilization approaches the capacity of a server, requests spend longer waiting in its queue, and the wait grows without bound. In the simplest queueing model (M/M/1: random arrivals, one server), the average time in the system is the bare service time divided by (1−ρ)what its symbols mean, where ρ is utilization:
W=1−ρ1/μwhat its symbols mean
At 50% utilization a request takes on average 2× its service time, at 90% it takes 10×, and at 99% it takes 100×. The growth is not exponential, but it is steep enough that systems are run well below full utilization.
Little's law of queueing
In any stable queuing system, the long-run average number of requests in the system (Lwhat its symbols mean) equals the average arrival rate (λwhat its symbols mean) multiplied by the average time each request spends in the system (Wwhat its symbols mean):
L=λ×Wwhat its symbols mean
For example, if your application receives 10,000 RPS (λ) and each request spends 50 ms in the system (W=0.05 s), then on average 500 requests are in flight at once. Size thread pools and connection pools for at least that, plus headroom for bursts.
The formula above is for one server that works on one request at a time. Most servers work on many at once. By Little's law, an app server that serves 1,000 requests a second, each taking 16 ms, has 16 in hand at once. A request waits only when all 16 places are taken. At 80% busy, about 3 requests in 10 wait, and the wait averaged over all requests is 1.5 ms; for one server working on one request at a time it would be 64 ms. Such a server barely waits until it is nearly full, which is why real servers run hotter than the formula for one server suggests. The lab counts waiting this way.
Why averages lie: tail latency (p99)
Averages obscure critical degradation. If 99% of requests take 10 ms, but 1% take 2,000 ms, the average looks healthy (≈30 ms), but 1 in every 100 customers experiences an unacceptable delay.
When a single user request fans out to many servers in parallel, it is only as fast as the slowest one. If one page request fans out to 50backendservices and each independently has a 1% chance of being slow:
P(at least one slow response)=1−(1−0.01)50≈39.5%what its symbols mean
Nearly 40% of page requests are slow, even though each service is slow only 1% of the time. Dean and Barroso's "The Tail at Scale" gives the same effect at Google: with 100 servers per request, 63% of user requests take over one second.
Follow a request from the clients: the load balancer hands it to one of six workers. The response time the client sees is the time the request waits in a worker’s queue plus the time the worker spends on it. Picture the arrival rate rising: each worker spends more of its time working, its queue gets longer, and the wait grows steeply as the worker nears full use.
Clients: 1 × Users
Load balancer: 2 × Load balancer
Workers: 6 × App server
Clients sends requests to Load balancer
Load balancer sends requests to Workers
Key takeaways
Response time is service time plus waiting (latency). Aim for the most throughput at an acceptable response time: time in the system is the service time divided by (1−ρ)what its symbols mean, so 2× at 50% busy and 10× at 90%, for one server.
Little's law, L=λ×Wwhat its symbols mean, gives the number of requests in flight; size thread and connection pools for at least that, plus headroom.
Averages hide slow requests. A request that fans out is as slow as its slowest call: 50 calls, each slow 1% of the time, make about 40% of requests slow.