Hands-on design challenge

Challenge 9: Pastebin at scale

The situation

A paste service much busier than the primer's keeps every paste it has stored, and almost every request reads one. Pastes are text only. Most reads are for a small share of popular pastes. Today the text of every paste and the row describing it sit together in one relational database.

Traffic

5,000 requests a second.

OperationShareSteps
Read99%Read data
Write1%Write data
  • Read99%Read data
  • Write1%Write data

Data

It keeps 40 TB of content, and 400 GB of metadata about it. A cache answers 85% of the reads that reach it.

What you are proving

Storage chosen for what it holds: the bulky paste text goes where a gigabyte is cheap, and the database keeps only the small rows that a lookup by short link needs.

Rules

  • A CDN cannot answer these reads.Every read checks whether its paste has expired, and a copy kept at an edge would go on serving a paste after it expired.
  • No paste is deleted to save space.By default a paste never expires, so the stored text only grows.

Builds on

Further reading: System Design Interview, volume 1, chapter 8.

Targets

  • Serve all 5,000 requests a second
  • At least 99.95% available, with no single point of failure
  • At most 3,900 USD a month
  • Every write kept in a relational database
  • Any tier loses one instance: nothing dropped, no tier over 80% busy

Hints