Hands-on design challenge

Challenge 11: Storing a web crawl

The situation

A crawler's fetchers hand every page they download to your intake, which must keep it. Many of those pages repeat one already stored: mirrors, syndicated articles, pages that differ only in an advert. Keep each unique page and the metadata about it, and pay to store nothing twice.

Traffic

400 requests a second. 30% of writes repeat an earlier one.

OperationShareSteps
Write100%Write data
  • Write100%Write data

Data

It keeps 500 TB of content, and 2 TB of metadata about it.

What you are proving

At this volume, storage is nearly the whole bill: checking each page against what is already kept, before it is written, means paying only for unique pages, and the pages go to storage priced for bulk content, with only their metadata in a database.

Rules

  • A CDN does not help here.Every request stores a page and none reads one back, so an edge would have nothing to answer.
  • Every unique page is kept, with its metadata.The inverted index and the snippets are built from the stored pages, and a page that changes cannot be fetched again as it was.

Builds on

Further reading: System Design Interview, volume 1, chapter 9.

Targets

  • Serve all 400 requests a second
  • At least 99.9% available, with no single point of failure
  • At most 17,500 USD a month
  • Every write kept in a relational database
  • Any tier loses one instance: nothing dropped, no tier over 80% busy

Hints