Serving language models
Every question from this chapter’s lessons, grouped by the lesson that teaches it. Answer each one right to finish; you can try again whenever you like.
0 of 28 answered right
17.1 How a model answers a request
0 of 7 answered rightQuestion 1
In MLPerf entry 5.1-0082, Llama 3.1 8B on one L40S had a median time to first token of 104 ms and a median time per output token of 45.8 ms, and counted 129 tokens an answer. Taking the time per token as steady, how long does a whole answer take?
Answers within 1% count, in any unit you choose.
Question 2
A GPU serving one request at a time writes the answer slowly, and its arithmetic units sit mostly idle. Which phase is it in, and what limits it?
Question 3
A decode step of the H100 run reads about 8 GB of its weights, and an H100 reads its memory at up to 3.35 TB/s. Taking those 8 GB as read once and counting nothing else, what is the least time one decode step for one request can take on it?
Answers within 5% count, in any unit you choose.
Question 4
A service writes a summary of each support call and stores it for the next agent who opens the case. Which time should its p99 target be on?
Question 5
In the H100 run, the p99 time to first token was more than eight times its median, while the p99 time per output token was 6% above its median. What is the likely reason the first token’s tail is so long?
Question 6
Match each term to its meaning.
Question 7
Put what happens to one request at a model server in order.
17.2 Batching and the KV cache
0 of 8 answered rightQuestion 8
Llama 3.1 8B has 32 layers and 8 key/value heads of 128 numbers each. How many bytes of KV cache does one token take with an 8-bit cache, one byte a value?
Answers within 1% count, in any unit you choose.
Question 9
You size Llama 3.1 8B’s KV cache with its 32 attention heads in place of its 8 key/value heads. What happens to your estimate?
Question 10
The L40S run (entry 5.1-0082) completed 9.36 requests a second, with a mean whole answer of 6.311 s. How many requests were in flight at once, on average?
requests
Answers within 2% count.
Question 11
Why does running 32 requests in one decode step make far more tokens a second than running them one at a time?
Question 12
A server batches requests with a time limit of 50 ms. Short answers keep coming back late, held until the longest answer in their batch is done. What change fixes that?
Question 13
In vLLM 0.10.0, what happens when the KV cache has no room for every running request?
Question 14
The GH200 with 96 GB served Llama 2 70B at 10.96 requests a second, far inside both time limits, and did no better with no limits; the 144 GB part did 1.49 times as much with no limits, on 1.23 times the bandwidth and 1.5 times the memory. What do these figures support?
Question 15
Match each way of using several GPUs to what it does.
17.3 Throughput, cost and hosted models
0 of 7 answered rightQuestion 16
The help desk sends its 1,684,800 summaries a day, 19.5 a second on average, to Claude Sonnet 5.5 (2 USD a million input tokens, 10 output, 18 November 2026), at the estimate of 1,000 input and 130 output tokens a summary. What do its tokens cost in a 30-day month?
USD
Answers within 1% count.
Question 17
The help desk grows to 2,592,000 tickets a day, 30 a second on average, over the same day’s shape. One H100 system still takes at most 39.56 a second and Claude Haiku 5.5 the rest, at 0.000165 USD a summary. The levels above 39.56 a second are now 5 hours at 52.56 and 3 hours at 65.22. What do the hosted model’s tokens cost in a 30-day month?
USD
Answers within 1% count.
Question 18
The H100 run made 5,777.08 tokens a second in the benchmark’s Offline scenario and 5,103.99 in its Server scenario. Which should a design that answers users plan with, and why?
Question 19
A design keeps one GPU system for the steady load and sends the rest to a hosted model. Its token bill doubles from one month to the next, with the same calls a day. What most likely changed?
Question 20
Over the same day’s calls, on what kind of day does a hosted model alone cost least, against own GPUs sized for the peak and against own GPUs for the steady load with the hosted model for the rest?
Question 21
A team serves forty LoRA adapters of one base model on shared GPUs. One adapter’s traffic has grown until it needs several copies of the model by itself. What does the lesson advise for it?
Question 22
Match each term to its meaning.
17.4 Reusing work and starting fast
0 of 6 answered rightQuestion 24
A support assistant’s prompt holds the customer’s question, 3,000 tokens of fixed instructions and today’s date. Which order lets the most of each prompt be reused from the prefix cache?
Question 25
A model-serving pool adds servers when its load passes a threshold. Each morning its time to first token is far over target for a while as traffic rises, then recovers. What is the likely cause, and the fix?
Question 26
When does running prefill and decode on separate GPUs pay for itself?
Question 27
In what sense are speculative decoding’s answers the same, in Leviathan et al.’s paper?
Question 28
A server running large batches turns on speculative decoding, and its tokens a second fall. Why?