Hands-on design challenge

Challenge 2: An order service that survives failures

The situation

Customers check the status of their orders here. The service runs on one load balancer, a few app servers and one relational database. It is already short of capacity, and losing any one of those machines would stop it or overload the ones left.

Traffic

5,000 requests a second.

OperationShareSteps
Read80%Read data
Write20%Write data
  • Read80%Read data
  • Write20%Write data

Data

A cache answers 85% of the reads that reach it.

What you are proving

Redundancy that holds under load: whichever single machine fails, the rest carry every request at the same speed, and the availability you claim follows from the arithmetic.

Rules

  • Orders stay in the relational database, and every write is stored in it.The orders are already there.
  • A CDN cannot answer these requests.Each request is about one customer's orders, so no two customers get the same response.
  • Machines fail independently, one at a time.The availability arithmetic assumes it, as the formulas in lesson 2.3 do.

Builds on

Targets

  • Serve all 5,000 requests a second
  • At least 99.99% available, with no single point of failure
  • At most 1,800 USD a month
  • Every write kept in a relational database
  • Any tier loses one instance: nothing dropped, no tier over 80% busy

Hints