Lesson 17.1• 6 min read

How a model answers a request

Tokens, reading the prompt and writing the answer, and the two times a response has.

What you will learn

  • Count a model's work in tokens, and say why it writes an answer one token at a time.
  • Tell reading the prompt from writing the answer, and say which limit of the hardware each one meets.
  • Split a model's response time into the time to the first token and the time per token, and work out a whole answer from the two.

Tokens in, tokens out

A language model completes text: given a text, the prompt, it predicts what comes next, and so writes an answer. It does not work in letters or whole words. A tokenizer first cuts the text into tokens, each a character, a word or part of a word, from a vocabulary its developers fixed for that model. Each model has its own tokenizer, so the same text can be a different number of tokens for two models.

The model writes its answer one token at a time. Each new token is chosen from the prompt and every token written so far; it is then added to the end, and the next token is chosen from the longer text. This repeats until the model writes a stop token or reaches a length limit. Writing one token at a time, each from everything before it, is called autoregressive generation.

The model itself is a large set of numbers, its weights (also called parameters). Llama 3.1 8B, the model in this chapter's measurements, has about 8 billion of them. Each weight is stored in a number of bits, its precision: at 16 bits a weight takes two bytes, at 8 bits one. The copy of the model the run below served keeps the weights of the model's main calculations at 8 bits and the rest, about 1 billion, at 16 (Red Hat's published model files): its weights take 9.08 GB.

A model server runs the model on a GPU, a chip that does a great many arithmetic operations at once and has its own memory, where the weights are kept while it serves. The H100, the GPU in the run this lesson measures, has 80 GB of memory that it reads at up to 3.35 TB/s (NVIDIA's specification).

Reading the prompt: prefill

The first phase, prefill, reads the whole prompt at once: all of its tokens are processed together, in parallel. It ends with the first token of the answer, and with the results the next phase reuses (lesson 17.2 sizes them).

Because every prompt token is processed in the same pass, each weight the GPU reads from memory is used for many tokens. The work is limited by how fast the GPU can do arithmetic: prefill is compute-bound. A longer prompt is more work, so prefill takes longer.

Writing the answer: decode

The second phase, decode, writes the rest of the answer, one token per step. To make one token for one request, a step reads nearly all the model's weights from memory and does only a little arithmetic with each byte. The work is limited by how fast the GPU can read its memory: decode is memory-bandwidth-bound, and the GPU's arithmetic units sit mostly idle.

The lesson's own illustration of that limit, from the figures above. Of the run's 9.08 GB, a step reads about 8 GB: everything but most of one table, which turns each token into numbers and of which a step needs only its own tokens' rows. Counting only those 8 GB, each byte read once, and nothing else the step does, at 3.35 TB/s a step takes at least

8 GB3,350 GB/s≈2.4 ms\frac{8 \text{ GB}}{3{,}350 \text{ GB/s}} \approx 2.4 \text{ ms}

so no step of one request alone can be faster than that on this GPU, however little arithmetic it does; what the step reads besides only adds to it. Lesson 17.2 shows how a server puts that idle arithmetic to use.

Two times, not one

A model's answer has two times that matter, and they are measured apart:

  • Time to first token (TTFT): from the request reaching the model to its first token. It covers the request's wait for its turn and its prefill.
  • Time per output token (TPOT): the time between one token of the answer and the next, which is decode's pace.

Taking the time per token as steady, an answer of NNwhat its symbols mean tokens takes

Twhole=TTFT+TPOT×(N−1)T_{\text{whole}} = \text{TTFT} + \text{TPOT} \times (N - 1)what its symbols mean

since the first token arrives with the first time and each of the other N−1N - 1 takes one time per token. The time to cut text into tokens and back is left out: it is small beside the model's own time.

Which time a design must meet depends on who waits. In a chat, the answer is shown as it is written (streamed), so the user sees the first token first: the time to first token is what feels slow. A step of a pipeline that needs the whole answer before it can go on, such as a summary stored for later, waits for the last token.

A measured case. MLPerf Inference is a benchmark whose results are audited: its runs report how fast a system served a set task. In round 5.1, entry 5.1-0083, Red Hat ran Llama 3.1 8B on one H100 with the vLLM serving software. Each request asked for a summary of a news article in at most 128 tokens. Requests arrived at random times, and it completed 39.56 a second. It counted 129 tokens an answer (its 5,103.99 tokens a second ÷ 39.56), one more than an answer may hold: Red Hat's published test program adds the end token to an answer that stops at the limit before it counts the tokens. With the log's median times:

223.6 ms+23.64 ms×(129−1)≈3,250 ms223.6 \text{ ms} + 23.64 \text{ ms} \times (129 - 1) \approx 3{,}250 \text{ ms}

The run's measured median whole answer was 3,241 ms. They are close because the benchmark computes each answer's time per token from the same count: its whole time, less its first token, divided by its tokens less one. They are not equal because the median of a sum is not the sum of the medians.

Percentiles of each

Each time has its own percentiles, and they do not behave alike. The same run:

Median (p50) p99
Time to first token 224 ms 1,911 ms
Time per output token 23.6 ms 25.1 ms
Whole answer 3,241 ms 5,050 ms

The first token's p99 is more than eight times its median; the time per token's p99 is 6% above its median. A request's first token waits for its turn, and requests arrive at random, so some arrive when the server is busiest. The vLLM documentation for the release the run used (0.10.0) says that its default engine gathers the pending decode steps of the requests already running before it schedules any new prefill. Requests already writing keep a steady pace, and a new request's first token waits. That is the likely reason for this run's long first-token tail; the run does not report its cause.

The benchmark set a limit on each time's p99: 2,000 ms for the first token and 100 ms a token. The run's 1,911 ms was near its first-token limit, while its 25.1 ms a token was a quarter of its limit: of the two, the first token's was the nearer. Set a target for each time, by who waits for it: a p99 on the first token for a chat, a p99 on the whole answer for a step that waits for all of it.

A request’s two times

The first token comes after the request’s wait and the reading of its prompt; each token after it takes one time per token.

129
Time to first token224 ms
Time per output token23.6 ms
Whole answer3,250 ms
Measured whole answer3,241 ms

Medians of MLPerf Inference 5.1, entry 5.1-0083 (Red Hat): Llama 3.1 8B on 1 × NVIDIA H100 SXM 80 GB, vLLM 0.10.0. Answers as measured, up to 129 counted tokens.

Key takeaways

  • A model's work is counted in tokens, its tokenizer's pieces of text, and it writes its answer one token at a time, each from everything before it.
  • Prefill reads the whole prompt at once and is compute-bound; decode writes one token a step, reads the weights each step, and is memory-bandwidth-bound.
  • A response has two times: the time to first token and the time per output token. The whole answer is TTFT+TPOT×(N−1)\text{TTFT} + \text{TPOT} \times (N - 1)what its symbols mean: 223.6 ms + 23.64 ms × 128 ≈ 3,250 ms in the H100 run, against a measured median of 3,241 ms.
  • Their tails differ: the first token waits for its turn and has a long tail; the pace of writing is steady. Set a target for each, by who waits for it.
Sources9