Lesson 1.2• 5 min read

Response time versus throughput

Understanding Little's law, response time percentiles, and tail latency amplification.

This trade-off is often called latency versus throughput, after the primer's name for it. This lesson uses the words as Designing Data-Intensive Applications does: latency is only the waiting, and response time is the whole time the user waits.

What you will learn

  • Define response time, service time, latency and throughput, and the units each is measured in.
  • Use W=1/μ1−ρW = \frac{1/\mu}{1 - \rho}what its symbols mean to see how waiting grows with utilization.
  • Size a pool with Little's law, and explain why p99 matters more than the average.

Precise definitions

  • Response time: The elapsed time from the moment a client sends a request until it receives the answer, measured in milliseconds (ms). It is what the user experiences.
  • Service time: The part of the response time during which a server is actively working on the request.
  • Latency: The part during which the request waits and nothing is working on it, such as time in a queue or crossing the network. Many people say latency for the whole response time, and common phrases such as tail latency keep that looser sense.
  • Throughput: The rate at which the system successfully processes operations or data, measured in requests per second (RPS) or bytes per second (GB/s).

You should aim for maximal throughput with an acceptable response time. As utilization approaches the capacity of a server, requests spend longer waiting in its queue, and the wait grows without bound. In the simplest queueing model (M/M/1: random arrivals, one server), the average time in the system is the bare service time divided by (1−ρ)(1 - \rho)what its symbols mean, where ρ\rho is utilization:

W=1/μ1−ρW = \frac{1/\mu}{1 - \rho}what its symbols mean

At 50%50\% utilization a request takes on average 2×2\times its service time, at 90%90\% it takes 10×10\times, and at 99%99\% it takes 100×100\times. The growth is not exponential, but it is steep enough that systems are run well below full utilization.

Little's law of queueing

In any stable queuing system, the long-run average number of requests in the system (LLwhat its symbols mean) equals the average arrival rate (λ\lambdawhat its symbols mean) multiplied by the average time each request spends in the system (WWwhat its symbols mean):

L=λ×WL = \lambda \times Wwhat its symbols mean

For example, if your application receives 10,000 RPS10,000\text{ RPS} (λ\lambda) and each request spends 50 ms50\text{ ms} in the system (W=0.05 sW = 0.05\text{ s}), then on average 500500 requests are in flight at once. Size thread pools and connection pools for at least that, plus headroom for bursts.

The formula above is for one server that works on one request at a time. Most servers work on many at once. By Little's law, an app server that serves 1,0001,000 requests a second, each taking 16 ms16\text{ ms}, has 1616 in hand at once. A request waits only when all 1616 places are taken. At 80%80\% busy, about 33 requests in 1010 wait, and the wait averaged over all requests is 1.5 ms1.5\text{ ms}; for one server working on one request at a time it would be 64 ms64\text{ ms}. Such a server barely waits until it is nearly full, which is why real servers run hotter than the formula for one server suggests. The lab counts waiting this way.

Why averages lie: tail latency (p99)

Averages obscure critical degradation. If 99%99\% of requests take 10 ms10\text{ ms}, but 1%1\% take 2,000 ms2,000\text{ ms}, the average looks healthy (≈30 ms\approx 30\text{ ms}), but 11 in every 100100 customers experiences an unacceptable delay.

When a single user request fans out to many servers in parallel, it is only as fast as the slowest one. If one page request fans out to 5050 backend services and each independently has a 1%1\% chance of being slow:

P(at least one slow response)=1−(1−0.01)50≈39.5%P(\text{at least one slow response}) = 1 - (1 - 0.01)^{50} \approx 39.5\%what its symbols mean

Nearly 40%40\% of page requests are slow, even though each service is slow only 1%1\% of the time. Dean and Barroso's "The Tail at Scale" gives the same effect at Google: with 100100 servers per request, 63%63\% of user requests take over one second.

Clients1× instanceLoad balancer2× instancesWorkers6× instancesClients1× instanceLoad balancer2× instancesWorkers6× instancesClients1× instanceLoad balancer2× instancesWorkers6× instances
Figure 2

Requests queuing at a pool of workers.

Follow a request from the clients: the load balancer hands it to one of six workers. The response time the client sees is the time the request waits in a worker’s queue plus the time the worker spends on it. Picture the arrival rate rising: each worker spends more of its time working, its queue gets longer, and the wait grows steeply as the worker nears full use.

  • Clients: 1 × Users
  • Load balancer: 2 × Load balancer
  • Workers: 6 × App server
  • Clients sends requests to Load balancer
  • Load balancer sends requests to Workers

Key takeaways

  • Response time is service time plus waiting (latency). Aim for the most throughput at an acceptable response time: time in the system is the service time divided by (1−ρ)(1 - \rho)what its symbols mean, so 2× at 50% busy and 10× at 90%, for one server.
  • Little's law, L=λ×WL = \lambda \times Wwhat its symbols mean, gives the number of requests in flight; size thread and connection pools for at least that, plus headroom.
  • Averages hide slow requests. A request that fans out is as slow as its slowest call: 50 calls, each slow 1% of the time, make about 40% of requests slow.
Sources6