Hands-on design challenge

Challenge 17: Call summaries for a contact center

The situation

A contact center summarizes every call as it ends and stores the summary with the case, so the next agent who opens it reads it there. You can run an open model on GPUs you rent by the hour, call a hosted model and pay for its tokens, or do both.

Traffic

A day brings 3.456 million* calls summarized. The peak runs at 2.6 times the day’s average rate. Over the day: 8 hours at 0.298 times the average, 6 at 0.674, 2 at 1.145, 5 at 1.752 and 3 at 2.174. The peak and the day are a coding service’s weekdays in the Azure LLM inference trace of 2024 (published with DynamoLLM, HPCA 2025), read 18 November 2026: the peak is its busiest minute and the levels are its hours, grouped. That a day of that shape fits a contact center is this scenario’s assumption*. Each request sends 871 tokens on average, and its answer is 128 tokens, as the benchmark’s prompts and answers are.

Model server: Llama 3.1 8B on 2 × NVIDIA RTX PRO 6000 Blackwell Server Edition (TensorRT 10.11, FP4 weights), as AWS’s g7e.12xlarge at 8.28608 USD an hour on 18 November 2026. Measured in MLPerf Inference 6.0, entry 6.0-0051: 81.08 requests a second: whole answer 3,298 ms (p99 4,146 ms), first token 80 ms (p99 475 ms) and 92.24 requests a second: whole answer 6,504 ms (p99 7,634 ms), first token 164 ms (p99 805 ms).

Model server: Llama 3.1 8B on 1 × NVIDIA H100 SXM 80 GB (vLLM 0.10.0, FP8 weights), as AWS’s p5.4xlarge at 6.88 USD an hour on 18 November 2026. Measured in MLPerf Inference 5.1, entry 5.1-0083: 39.56 requests a second: whole answer 3,241 ms (p99 5,050 ms), first token 224 ms (p99 1,911 ms).

Model server: Llama 3.1 8B on 1 × NVIDIA L40S (vLLM 0.10.0, FP8 weights), as AWS’s g6e.xlarge at 1.861 USD an hour on 18 November 2026. Measured in MLPerf Inference 5.1, entry 5.1-0082: 9.36 requests a second: whole answer 5,979 ms (p99 12,560 ms), first token 104 ms (p99 407 ms). A run measured at one rate keeps its times below it, where real ones are lower. Two runs of one system were measured separately: times between their rates are estimates, and below the lower rate real ones are lower still. Above a run’s highest rate requests wait without end, so the lab counts them as not served.

Claude Haiku 5.5, at Anthropic’s prices of 18 November 2026; its times* are this scenario’s, not Anthropic’s. It charges 0.10 USD a million input tokens, 0.125 to write the prompt cache and 0.01 to read it, and 0.50 a million output tokens. Its first token comes in 500* ms (p99 1,500*), then 8* ms a token (p99 16*). A request is 1,000* tokens in and 130* out, as it counts them. It accepts at most 1,000* calls a second.

* An estimate for this scenario, not a measured value.

OperationSteps
ReadRead data
  • ReadRead data

Data

A CDN can answer none of the reads. A cache answers none of the requests: each is new.

What you are proving

You can price serving a model three ways (GPUs rented by the hour, tokens bought as they are used, and the two together) and choose between them by the shape of the day, not by its peak alone.

Rules

  • A model server runs one of the measured runs, on its measured system and software.Its rate and times are what the benchmark measured on prompts of this shape; another setup would need its own measurement.
  • The time that counts is the whole summary.Nobody watches a summary being written: it is stored, and the next agent reads it once it is done.

Builds on

Targets

  • Serve every request at the peak
  • Whole answer: p99 at most 6,000 ms
  • At least 99.9% available, with no single point of failure
  • At most 8,500 USD a month
  • Any tier loses one instance: nothing dropped, p99 holds, no tier over 80% busy
  • Every request answered by a model

Hints