Architecture lab

A model too large for one GPUTarget 3 RPS · first token p99 ≤ 6,000 ms

0% busy
Analysts
1x inst3 RPS
0% busy
Load balancer
1x inst3 RPS
0% busy
Question API
1x inst3 RPS
100% busy
Model server
1x inst3 RPS

Design summary

Components

  • Analysts: 1 × Users
  • Load balancer: 1 × Load balancer, 3 requests a second, 0% busy
  • Question API: 1 × App server, 3 requests a second, 0% busy
  • Model server: 1 × Model server, 3 requests a second, 100% busy

Connections

  • Analysts sends requests to Load balancer
  • Load balancer sends requests to Question API
  • Question API sends requests to Model server

0 of 3 requests a second served, average 0 ms, p99 0 ms, 5,049 USD a month. Challenge not yet passed: throughput, availability, losing one instance, answered by a model.