Hands-on design challenge

Challenge 5: IoT telemetry ingestion

The situation

Devices in the field send their readings to an ingest service, which stores them in one relational database with a standby. Almost all of the traffic is writes, far more than one leader can take.

Traffic

30,000 requests a second. 1% of requests are for one key at a time.

OperationShareSteps
Read5%Read data
Write95%Write data
  • Read5%Read data
  • Write95%Write data

Data

A cache answers 85% of the reads that reach it.

What you are proving

A store that grows with its writes: every reading is kept, spread so that no part of the store, not even the one holding the busiest device, gets more load than it can bear.

Rules

  • Readings may be kept in any kind of store.Each reading is written once and never changed, so none needs a transaction.
  • A CDN cannot answer these requests.Readings are asked for one device at a time, so no two requests get the same response.

Builds on

Targets

  • Serve all 30,000 requests a second
  • At least 99.99% available, with no single point of failure
  • At most 6,000 USD a month
  • Any tier loses one instance: nothing dropped, no tier over 80% busy

Hints