Lesson 9.1• 8 min read

Requirements, estimates and the API

The primer’s use cases and estimates: 10 million writes and 100 million reads a month, 12.7 GB of new content a month, and the REST API.

What you will learn

  • State Pastebin's use cases and assumptions as the primer does.
  • Estimate its request rates, storage and cache size, and scale them with traffic.
  • Write its REST API.

The problem

The primer's worked example is Design Pastebin.com (or Bit.ly). Bit.ly, a URL shortener, is "a similar question, except pastebin requires storing the paste contents instead of the original unshortened url". Both map a short random key, which the primer calls the shortlink, to something larger. Both are read far more often than written, and both expose the same decisions: how to generate unique keys, where to store content, how to expire it and how to count visits.

Use cases (from the primer)

  • A user enters a block of text and gets a randomly generated link. By default it never expires; the user can optionally set a timed expiration.
  • A user enters a paste's URL and views its contents.
  • Users are anonymous.
  • The service tracks monthly visit statistics, deletes expired pastes and has high availability.

The primer leaves out accounts, editing, visibility and custom shortlinks. This chapter adds custom aliases and redirect analytics as extensions.

Assumptions (from the primer)

  • Traffic is not evenly distributed, and following a short link should be fast.
  • Pastes are text only, and analytics do not need to be real time.
  • 10 million paste writes and 100 million paste reads per month: a 10:1 read-to-write ratio.

Back-of-the-envelope estimates

The primer uses a handy conversion: a month has about 2.52.5 million seconds. (Thirty days is 2.5922.592 million, so the results below round slightly up.)

Writes=10×1062.5×106 s=4 per second on average\text{Writes} = \frac{10 \times 10^6}{2.5 \times 10^6\text{ s}} = 4 \text{ per second on average}
Reads=100×1062.5×106 s=40 per second on average\text{Reads} = \frac{100 \times 10^6}{2.5 \times 10^6\text{ s}} = 40 \text{ per second on average}

Peaks are higher, because traffic is uneven. The primer's rule of thumb: 11 request per second is about 2.52.5 million a month, and 400400 per second about 11 billion.

Size per paste. The primer assumes 1 KB of content plus a database row: shortlink 7 bytes, expiration_length_in_minutes 4 bytes, created_at 5 bytes and paste_path 255 bytes. That is about 1.271.27 KB in total.

New content per month=1.27 KB×10×106=12.7 GB\text{New content per month} = 1.27\text{ KB} \times 10 \times 10^6 = 12.7\text{ GB}
Over 3 years=12.7 GB×36≈450 GB,10×106×36=360 million shortlinks\text{Over 3 years} = 12.7\text{ GB} \times 36 \approx 450\text{ GB},\quad 10 \times 10^6 \times 36 = 360\text{ million shortlinks}

How the numbers scale. Every figure here is linear in traffic. At 10 times the primer's volume (100 million writes a month), multiply each by 10: 40 writes and 400 reads per second on average, and 127 GB of new content per month. The challenge at the end of this chapter uses a much higher rate, to exercise the lab.

Cache sizing. Size the cache by the number of distinct shortlinks you want to keep hot, not by the number of requests. If the most popular 20% of the 360 million shortlinks are worth caching, at about 500 bytes per cached entry:

0.2×360×106×500 B=36 GB0.2 \times 360 \times 10^6 \times 500\text{ B} = 36\text{ GB}

The 20% is an assumption to check against real access logs, not a law (chapter 6).


API

The primer uses a public REST API:

POST /api/v1/paste
Content-Type: application/json

{ "expiration_length_in_minutes": "60", "paste_contents": "Hello World!" }

→ { "shortlink": "foobar" }
GET /api/v1/paste?shortlink=foobar

→ { "paste_contents": "Hello World", "created_at": "YYYY-MM-DD HH:MM:SS",
    "expiration_length_in_minutes": "60" }

For a new resource, HTTP's 201 Created status with a Location header naming it is the standard response (RFC 9110). For internal calls between services, the primer suggests RPC.

Clients1× instanceDNS1× instanceLoad balancer2× instancesWrite API2× instancesRead API2× instancesMemory cache2× instancesMetadata database2× instancesObject store1× instanceClients1× instanceDNS1× instanceLoad balancer2× instancesWrite API2× instancesRead API2× instancesMemory cache2× instancesMetadata database2× instancesObject store1× instance
Figure 1

The primer’s Pastebin, scaled: one path for writes, one for reads, small rows in a database and the text in an object store.

Follow each use case. Creating a paste goes from the client, which finds the load balancer through DNS, to the Write API, which saves a small row in the metadata database and the text in the object store. Viewing a paste goes to the Read API, which looks the short link up through the memory cache, asking the database only when the cache misses, and fetches the text from the object store. Reads outnumber writes 10 to 1, so the two APIs are separate boxes that can grow apart; at 4 writes and 40 reads a second on average, two servers of each are enough, the second so that one can fail.

  • Clients: 1 × Users
  • DNS: 1 × DNS
  • Load balancer: 2 × Load balancer
  • Write API: 2 × App server
  • Read API: 2 × App server
  • Memory cache: 2 × Cache
  • Metadata database: 2 × Relational database
  • Object store: 1 × Object store
  • Clients sends requests to DNS
  • DNS sends requests to Load balancer
  • Load balancer sends requests to Write API
  • Load balancer sends requests to Read API
  • Write API sends requests to Metadata database
  • Write API sends requests to Object store
  • Read API sends requests to Memory cache
  • Memory cache sends requests to Metadata database
  • Read API sends requests to Object store

Key takeaways

  • 10 million writes and 100 million reads a month are 4 writes and 40 reads a second on average; peaks are higher.
  • About 1.27 KB a paste gives 12.7 GB a month and about 450 GB, 360 million links, over three years; every figure grows linearly with traffic.
  • Size a cache by the distinct links you want hot, not by requests: 20% of 360 million links at 500 bytes is 36 GB, an assumption to check against real access logs.
  • Creating a paste answers 201 Created with a Location header; internal calls can use RPC.
Sources4