Challenge 11: Storing a web crawl
The situation
A crawler's fetchers hand every page they download to your intake, which must keep it. Many of those pages repeat one already stored: mirrors, syndicated articles, pages that differ only in an advert. Keep each unique page and the metadata about it, and pay to store nothing twice.
Traffic
400 requests a second. 30% of writes repeat an earlier one.
| Operation | Share | Steps |
|---|---|---|
| Write | 100% | Write data |
- Write100%Write data
Data
It keeps 500 TB of content, and 2 TB of metadata about it.
What you are proving
At this volume, storage is nearly the whole bill: checking each page against what is already kept, before it is written, means paying only for unique pages, and the pages go to storage priced for bulk content, with only their metadata in a database.
Rules
- A CDN does not help here.Every request stores a page and none reads one back, so an edge would have nothing to answer.
- Every unique page is kept, with its metadata.The inverted index and the snippets are built from the stored pages, and a page that changes cannot be fetched again as it was.
Builds on
- 1.2 Response time versus throughput
- 2.2 Availability patterns and failover
- 2.3 Availability in numbers
- 7.1 Queues, task queues and log-based message brokers
- 9.3 Storage: SQL or NoSQL, object storage and expiry
- 11.1 Requirements, estimates and a crawler's parts
- 11.3 Duplicate elimination and SimHash
- 11.4 Fetching, DNS caching and fault tolerance
Further reading: System Design Interview, volume 1, chapter 9.
Targets
- Serve all 400 requests a second
- At least 99.9% available, with no single point of failure
- At most 17,500 USD a month
- Every write kept in a relational database
- Any tier loses one instance: nothing dropped, no tier over 80% busy