Traffic
A day brings 3.456 million* calls summarized. The peak runs at 2.6 times the day’s average rate. Over the day: 8 hours at 0.298 times the average, 6 at 0.674, 2 at 1.145, 5 at 1.752 and 3 at 2.174. The peak and the day are a coding service’s weekdays in the Azure LLM inference trace of 2024 (published with DynamoLLM, HPCA 2025), read 18 November 2026: the peak is its busiest minute and the levels are its hours, grouped. That a day of that shape fits a contact center is this scenario’s assumption*. Each request sends 871 tokens on average, and its answer is 128 tokens, as the benchmark’s prompts and answers are.
Model server: Llama 3.1 8B on 2 × NVIDIA RTX PRO 6000 Blackwell Server Edition (TensorRT 10.11, FP4 weights), as AWS’s g7e.12xlarge at 8.28608 USD an hour on 18 November 2026. Measured in MLPerf Inference 6.0, entry 6.0-0051: 81.08 requests a second: whole answer 3,298 ms (p99 4,146 ms), first token 80 ms (p99 475 ms) and 92.24 requests a second: whole answer 6,504 ms (p99 7,634 ms), first token 164 ms (p99 805 ms).
Model server: Llama 3.1 8B on 1 × NVIDIA H100 SXM 80 GB (vLLM 0.10.0, FP8 weights), as AWS’s p5.4xlarge at 6.88 USD an hour on 18 November 2026. Measured in MLPerf Inference 5.1, entry 5.1-0083: 39.56 requests a second: whole answer 3,241 ms (p99 5,050 ms), first token 224 ms (p99 1,911 ms).
Model server: Llama 3.1 8B on 1 × NVIDIA L40S (vLLM 0.10.0, FP8 weights), as AWS’s g6e.xlarge at 1.861 USD an hour on 18 November 2026. Measured in MLPerf Inference 5.1, entry 5.1-0082: 9.36 requests a second: whole answer 5,979 ms (p99 12,560 ms), first token 104 ms (p99 407 ms). A run measured at one rate keeps its times below it, where real ones are lower. Two runs of one system were measured separately: times between their rates are estimates, and below the lower rate real ones are lower still. Above a run’s highest rate requests wait without end, so the lab counts them as not served.
Claude Haiku 5.5, at Anthropic’s prices of 18 November 2026; its times* are this scenario’s, not Anthropic’s. It charges 0.10 USD a million input tokens, 0.125 to write the prompt cache and 0.01 to read it, and 0.50 a million output tokens. Its first token comes in 500* ms (p99 1,500*), then 8* ms a token (p99 16*). A request is 1,000* tokens in and 130* out, as it counts them. It accepts at most 1,000* calls a second.
* An estimate for this scenario, not a measured value.
| Operation | Steps |
|---|
| Read | Read data |
|---|