Architecture lab

Challenge 11: Storing a web crawlTarget 400 RPS

0% busy
Fetchers
1x inst400 RPS
40% busy
Intake
4x inst400 RPS
16% busy
Page database
2x inst400 RPS

Design summary

Components

  • Fetchers: 1 × Users
  • Intake: 4 × App server, 400 requests a second, 40% busy
  • Page database: 2 × Relational database, 400 requests a second, 16% busy

Connections

  • Fetchers sends requests to Intake
  • Intake sends requests to Page database

400 of 400 requests a second served, average 34 ms, p99 71 ms, 164,826 USD a month. Challenge not yet passed: availability, monthly cost, losing one instance.