Throughput, cost and hosted models
What a token costs on your own GPUs and from a provider, and which costs less over a whole day.
What you will learn
- Turn a measured rate and an hourly price into a cost per million tokens and a cost per request.
- Say why a benchmark's rate within its time limits, not its fastest rate, is the one to plan with.
- Price a hosted model's tokens, and compare a month of rented GPUs with a month of tokens over a day's shape.
- Work out when your own GPUs, a hosted model, or the two together cost least, and say what decides it.
Tokens a second belong to a whole system
A figure of tokens a second is measured on a whole stack: a GPU, a model stored at a precision, serving software with its settings, and a workload of prompts and answers. Change any one of them and the figure changes, so a figure carries over only to the same stack. The two Red Hat runs of lesson 17.1 change only the GPU: the same model, software and benchmark round, on one H100 and on one L40S, each priced as the AWS instance that holds one such GPU (the L40S was measured on IBM Cloud).
| 1 × H100 (entry 5.1-0083) | 1 × L40S (entry 5.1-0082) | |
|---|---|---|
| Requests a second | 39.56 | 9.36 |
| Tokens a second | 5,103.99 | 1,207.14 |
| AWS instance, on demand | p5.4xlarge, 6.88 USD an hour | g6e.xlarge, 1.861 USD an hour |
The H100 makes 4.2 times the tokens for 3.7 times the price: the dearer GPU costs less a token.
Throughput within the time limits
The benchmark runs each system in two ways. In its Offline scenario every request is sent at once, and the result is how many it finishes a second, with no limit on how long any one takes: the H100 run made 5,777.08 tokens a second, and since every request was sent at the start, each waited 888 s on average for its answer. In the Server scenario requests arrive at random times, and the run counts only if the p99 of each of its two times stays inside the limits lesson 17.1 gave: 5,103.99 tokens a second.
A design plans with the Server figure, because its users wait for their answers. The DistServe paper (2024) calls the most requests a second a system serves with a set share of them inside their time targets its goodput (the paper's examples set the share at 90%; the benchmark's Server scenario holds 99% inside its limits). A higher throughput bought by letting requests wait longer is not goodput.
Cost per million tokens
A GPU is rented by the hour, so its cost a token is its price an hour over the tokens it makes in an hour:
For the H100 run, USD a million tokens; for the L40S, USD. These count only the tokens written: each request's prompt was read in the same hours at no extra price.
Since each model counts tokens with its own tokenizer, AI Engineering recommends comparing serving costs per request instead. The H100 run's is USD a summary. Both figures hold only while the GPU serves its measured rate in every hour it is paid for.
Paying per token
A hosted model is one a provider runs: you send each request to its interface and pay for the tokens, input and output priced apart. Their prices on 18 November 2026, in USD a million tokens:
| Model | Input | Output |
|---|---|---|
| Claude Haiku 5.5 (Anthropic) | 0.10 | 0.50 |
| Claude Sonnet 5.5 (Anthropic) | 2 | 10 |
| gpt-5.4-mini (OpenAI) | 0.75 | 4.50 |
Both providers also take requests sent in batches, and answer them later, at half these prices: for work that can wait for its answers. Each model counts its own tokens. Take a summary of 1,000 input and 130 output tokens in Claude Haiku 5.5's count (an estimate this chapter uses): it costs USD, about 3.4 times the H100 run's cost a summary. The hosted model costs more a request, and is paid only for the requests it answers.
GPUs paid at the peak, tokens paid as used
Traffic is not steady through a day. Azure's public trace of 2024 (published with the DynamoLLM paper, HPCA 2025) records every request of a coding service for a week: on each weekday its busiest minute had 2.53 to 2.65 times the day's average rate, and its quietest hour 0.20 to 0.25 times. This lesson takes a help desk's day to have the same shape as that coding service's, which is an assumption, not a measurement.
Take a help desk that summarizes 1,684,800 support tickets a day, each summary the size above: 19.5 a second on average, and 50.7 a second in the busiest minute, 2.6 times the average. Two designs:
- Your own GPUs, sized for the peak, and for the peak with one GPU lost (lesson 2.2). The cited system here is the H100 run's: one GPU that served 39.56 requests a second within the benchmark's limits, rented as AWS's p5.4xlarge at 6.88 USD an hour. A system of one GPU is lost whole with its GPU, so its spare is a whole system: two systems carry the peak (79.13 a second), and a third stands by. (A system of several GPUs, each running its own copy of the model, loses only that copy; lesson 17.2.) All three are paid every hour of the month: USD.
- The hosted model alone: USD a month, whatever the peak.
Here the hosted model costs less. The hosted bill depends only on the day's total, while the GPUs are sized by the peak and its spare, so they grow in steps as the peak grows: a fourth system is needed from a peak of about 4.06 times the average (). With more tickets the order turns: at five times as many, 97.5 a second, eight systems (seven for the peak of 253.5 a second, and a spare) cost 39,629 USD a month, the hosted model 41,699.
Your own GPUs for the steady load, a hosted model for the rest
A third design keeps one system, set to take at most its measured 39.56 requests a second, and sends what is above that rate to the hosted model. Only the day's busiest level, 3 hours at 2.174 times the average (42.39 a second), is above it, by 2.83 a second:
With the system's 4,954 USD, the design costs 5,105 USD a month: less than either design alone. The hosted model also stands in for the spare: when the GPU is lost, and the system with it, every request goes to the hosted model, so no spare system is paid for. Whichever model answers, each summary holds a thread of the app server that called it for the whole of its time, so the app servers are sized by the model's time as well (lesson 7.1). The lab also counts the load balancers and app servers, so a challenge's month is a little higher than these, and its bounds come a little later.
What sets this bill is how many hours the load stays above what the GPUs take, not the peak alone. The same hours with a busiest minute of 4 or 6 times the average cost the same 151 USD of tokens. If instead the peak of 50.7 a second held for 24 ÷ 2.6 = 9.23 hours a day, with no tickets the rest, the day that makes a spill dearest, the same design pays USD of tokens. On such a day the system saves tokens only while the peak lasts, so the hosted model alone becomes the cheapest of the three from a peak of about 3.42 times the average: 39.56 a second for 24 ÷ 3.42 = 7.02 hours a day is then worth less in tokens than the system's 4,954 USD.
So none of the three is cheapest as a rule. Work out each design's month over the day's shape: GPUs paid at the peak's size every hour; tokens paid for what they answer, hour by hour. AI Engineering adds what such a comparison leaves out: running your own model also takes engineering time, which hosted models spare you, and which the lab does not price.
Many variants of one base model
A team may adapt a model to several tasks by fine-tuning it. A common way, LoRA (low-rank adaptation), leaves the model's weights unchanged and trains small added layers, an adapter, for each task. Serving many adapters on one copy of the base model lets requests for different adapters run in the same batches, so a few GPUs serve variants that would each need their own as separate models. S-LoRA (MLSys 2024) reported serving thousands of adapters on one GPU or several, with up to 4 times the throughput of the libraries it was compared with, on its own tests. Hands-On LLM Serving and Optimization gives the limit: an adapter whose traffic needs several copies of its own is better merged into the model and served apart, since sharing pays where each adapter's traffic is too light to keep a GPU busy.
A day’s cost, three ways
Own GPUs are paid every hour at the peak’s size; a hosted model for what it answers; both, GPUs for the steady load and tokens for the rest.
19.5 a second on average, 50.7 at the busiest minute. The hosted model costs more than the own GPUs for 3.0 hours a day; the design with both sends it summaries for 3.0 hours a day.
1 × H100: MLPerf Inference, entry 5.1-0083, Server run, 39.56 requests a second; AWS p5.4xlarge, 6.88 USD an hour. Claude Haiku 5.5: its provider’s prices of 18 November 2026, for 1,000 input and 130 output tokens a summary, the lesson’s estimate. The weekday: Azure’s LLM inference trace of 2024, a coding service. Load balancers and app servers are left out.
Key takeaways
- A tokens-a-second figure is a whole stack's: the GPU, the model and its precision, the software and the workload. Plan with the rate served within the time limits, the goodput.
- Cost per million tokens is the price an hour ÷ the millions of tokens made in an hour; compare designs per request, since each model counts tokens its own way.
- Hosted models are paid for the tokens they answer, input and output priced apart; your own GPUs are paid every hour at the size the peak and a lost GPU need.
- Which design costs least depends on how many hours the load stays above what the GPUs take: for the help desk, one H100 system for the steady load and a hosted model for the rest costs 5,105 USD a month, against 14,861 for own GPUs alone and 8,340 for the hosted model alone.
Sources13
- Huyen, AI Engineering (O’Reilly, 2024), chapter 9: Throughput and goodput; Online and batch inference APIs; chapter 4: API cost versus engineering cost
- Wang and Hu, Hands-On LLM Serving and Optimization (O’Reilly, 2026), chapter 4: Throughput Metrics; chapter 10: Multi-LoRA Serving
- Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI 2024
- MLPerf Inference 5.1, entries 5.1-0083 (1 × H100) and 5.1-0082 (1 × L40S), Red Hat: Server and Offline run logs
- MLPerf Inference 6.0, entry 6.0-0051 (HPE, 2 × RTX PRO 6000): Interactive and Server run logs
- MLPerf LoadGen: the Offline and Server scenarios (test_settings.h, 5.1) and the Interactive limits (mlperf.conf)
- AWS: Amazon EC2 on-demand prices, US East (N. Virginia), price list of 2026-11-18
- Anthropic: Claude pricing, read 2026-11-18
- OpenAI: API pricing, read 2026-11-18
- Microsoft: Azure LLM inference trace of 2024 (Azure Public Dataset), coding service, a week of requests
- Stojkovic et al., “DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency”, HPCA 2025
- Sheng et al., “SLoRA: Scalable Serving of Thousands of LoRA Adapters”, MLSys 2024
- Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.