Reusing work and starting fast
Prompts that share a beginning, servers that must load before they answer, and two ways to spend a GPU’s idle time.
What you will learn
- Say what prefix caching reuses, why a prompt's fixed parts go first, and work out when a hosted model's prompt cache pays.
- Describe what a new model server does before its first answer, and what that means for adding servers as load rises.
- Say what running prefill and decode on separate GPUs gains, and when it is worth its cost.
- Say what speculative decoding trades, and in what sense its answers stay the same.
The same beginning, computed once
Many prompts begin the same way: instructions sent with every request, a document many questions are asked about, a conversation's earlier turns sent again with each new message. Prefill computes every prompt token's keys and values (lesson 17.2), and for the same beginning it computes the same ones each time. Prefix caching keeps the KV cache of a prompt's beginning after its request ends, and a later prompt that begins with exactly the same tokens reuses it and computes only the rest. It saves prefill's work, so the time to first token falls, most for a long shared beginning.
Only an exact beginning is reused: the match ends at the first token that differs, so a prompt that changes in its first line reuses nothing, even one character apart. Put the parts that stay the same first, in the same form every time, and what changes, such as the user's question, last. The kept KV caches take GPU memory beside the running requests' own, and the servers that keep them usually evict the least recently used first (lesson 6.4). A server reuses only the caches it holds itself, so with several servers a router sends prompts with the same beginning to the same server, much as consistent hashing keeps each key on one cache node (lesson 14.3).
A hosted model offers the same reuse at a price. Anthropic's prices on 18 November 2026: writing a prompt's beginning to its prompt cache costs 1.25 times the input price, and reading it back 0.1 times (0.05 times on Claude Sonnet 5.5 and Claude Opus 5.5); an entry lasts 5 minutes from the last request that used it, and nothing under 512 tokens is cached on these models. For a beginning of what its symbols mean tokens sent with what its symbols mean requests while it lasts, in units of the input price:
One request alone pays a quarter more for the write. Two pay against : the cache pays after one read. The lab prices it the same way: an app server set to cache the prompt pays the write and read prices for the part the challenge says is cached.
Starting a model server
A new model server answers nothing until its weights are in GPU memory: the model's files are read from storage into the machine's own memory and then copied to the GPU's, and only then does the serving software take requests. For lesson 17.1's model that is 9.08 GB to move before the first answer. How long it takes depends on where the files are and how fast each step moves them; none of this chapter's sources measures a whole start, so the lesson gives no time for it. A request held until its model is loaded waits through a cold start.
Adding servers as load rises (lesson 16.2) reacts to a measure crossing its threshold, so it always follows the load, and a new model server then adds its start before it helps. Meanwhile the servers already running take the extra requests, and their times grow. A pool that must meet its time targets through a rise runs the capacity before the rise, or sends what is over it somewhere already running, as the hosted model of lesson 17.3 does.
Prefill and decode apart
Lesson 17.2 showed the clash: with continuous batching, a new request's prefill and the running requests' decode steps share the GPU, so a long prefill slows the decode steps beside it, and a server that must meet both time targets favors one of them or is given more GPUs than its work needs. Chunked prefill eases this on one GPU. Disaggregation separates them: some GPUs only prefill, others only decode, and each request's KV cache is sent from the first to the second. Each pool is then sized for its own target: the prefill pool for the time to first token, the decode pool for the time per token. The DistServe paper (2024), which built this, reported on its own tests 7.4 times the requests, or targets 12.6 times tighter, than the systems it was compared with, with over 90% of requests inside their limits.
The cost is the move: each request's KV cache crosses from one GPU to another, which needs fast links between them and careful placement. Hands-On LLM Serving and Optimization advises it for large models under heavy load with demanding targets for both times, and the simpler setup, both phases on the same GPUs, for smaller models.
Guessing ahead
Decode leaves most of a GPU's arithmetic idle (lesson 17.1). Speculative decoding puts it to use: a small, fast draft model guesses the next few tokens, and the large model checks all of the guesses at once, in one pass, in parallel. It keeps the guesses up to the first one it rejects, and writes its own token in that place. When most guesses are kept, the answer takes fewer passes of the large model.
Leviathan et al. (ICML 2023) prove that their method changes nothing in what the large model would write: its answers are drawn from exactly the same probabilities as the large model's alone, and where the model always takes its most likely token, they are the same tokens. On their tests with T5-XXL, an 11-billion-weight model run one request at a time, the answers came 2 to 3 times faster than the standard implementation.
The trade, as Hands-On LLM Serving and Optimization describes it: the draft model and every rejected guess are extra work. It helps the time per token where decode is limited by memory, with small batches. With large batches the arithmetic is already busy, and the extra work can lower throughput; it does nothing for prefill.
A conversation’s input bill
Each turn sends the whole conversation again. With the cache, the turn before’s prompt is read back and only the rest is written.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
Claude Haiku 5.5: Anthropic’s prices of 18 November 2026, 0.1 USD a million input tokens; a cache entry lasts 5 minutes from its last use, so every turn is taken to come within 5 minutes of the one before. Nothing under 512 tokens is cached. Output tokens, the same either way, are left out.
Key takeaways
- Prefix caching keeps the KV cache of a prompt's beginning for the next prompt that begins the same way; put the fixed parts first and send prompts that share a beginning to the same server.
- A hosted model's prompt cache costs 1.25 times the input price to write and 0.1 times to read (Anthropic, 18 November 2026): it pays from the first read.
- A new model server must load its weights before it answers, so adding servers follows the load; keep capacity running before the rise or send the excess somewhere already running.
- Separate prefill and decode GPUs size each phase for its own time target, at the cost of moving each request's KV cache; speculative decoding spends idle arithmetic on guesses to cut the time per token, and its answers are the large model's own.
Sources6
- Wang and Hu, Hands-On LLM Serving and Optimization (O’Reilly, 2026), chapter 6: Prefix Caching; chapter 5: The Model Loading Process; chapters 1 and 3: cold starts and autoscaling; chapter 7: Speculative Decoding, Prefill-Decode Disaggregation
- Huyen, AI Engineering (O’Reilly, 2024), chapter 9: Prompt caching; Decoupling prefill and decode
- Anthropic: Claude pricing (prompt caching), and the prompt caching guide, read 2026-11-18
- Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI 2024
- Leviathan, Kalman and Matias, “Fast Inference from Transformers via Speculative Decoding”, ICML 2023
- Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.