Traffic
3* requests a second. Each request sends 9,429 tokens on average, and its answer is 654 tokens, as the benchmark’s prompts and answers are, shown as it is written.
Model server: Llama 3.1 405B on 8 × NVIDIA H100 SXM 80 GB (TensorRT 10.8, FP8 weights), as AWS’s p5.48xlarge at 55.04 USD an hour on 19 November 2026. Measured in MLPerf Inference 5.0, entry 5.0-0020: 0.428 requests a second: whole answer 41,342 ms (p99 84,861 ms), first token 1,556 ms (p99 5,731 ms).
Model server: Llama 3.1 405B on 8 × NVIDIA H200 SXM 141 GB (TensorRT 10.11, FP8 weights), as AWS’s p5en.48xlarge at 63.296 USD an hour on 19 November 2026. Measured in MLPerf Inference 5.1, entry 5.1-0075: 0.31 requests a second: whole answer 30,854 ms (p99 56,963 ms), first token 1,431 ms (p99 4,321 ms) and 0.452 requests a second: whole answer 38,119 ms (p99 76,263 ms), first token 1,537 ms (p99 5,643 ms).
Model server: Llama 3.1 405B on 8 × NVIDIA B200 SXM 180 GB (TensorRT 10.11, FP4 weights), as AWS’s p6-b200.48xlarge at 113.9328 USD an hour on 19 November 2026. Measured in MLPerf Inference 5.1, entry 5.1-0069: 1.15 requests a second: whole answer 44,436 ms (p99 72,595 ms), first token 988 ms (p99 2,511 ms) and 1.9 requests a second: whole answer 71,611 ms (p99 119,701 ms), first token 1,357 ms (p99 5,134 ms).
Model server: Llama 3.1 405B on 1 × NVIDIA H100 SXM 80 GB (TensorRT 10.8, FP8 weights), as AWS’s p5.4xlarge at 6.88 USD an hour on 19 November 2026. Its memory cannot hold the model’s weights, so it cannot run the model. A run measured at one rate keeps its times below it, where real ones are lower. Two runs of one system were measured separately: times between their rates are estimates, and below the lower rate real ones are lower still. Above a run’s highest rate requests wait without end, so the lab counts them as not served.
* An estimate for this scenario, not a measured value.
| Operation | Share | Steps |
|---|
| Read | 100% | Read data |
|---|