The primer’s use cases and estimates: 10 million writes and 100 million reads a month, 12.7 GB of new content a month, and the REST API.
What you will learn
State Pastebin's use cases and assumptions as the primer does.
Estimate its request rates, storage and cache size, and scale them with traffic.
Write its RESTAPI.
The problem
The primer's worked example is Design Pastebin.com (or Bit.ly). Bit.ly, a URL shortener, is "a similar question, except pastebin requires storing the paste contents instead of the original unshortened url". Both map a short random key, which the primer calls the shortlink, to something larger. Both are read far more often than written, and both expose the same decisions: how to generate unique keys, where to store content, how to expire it and how to count visits.
Use cases (from the primer)
A user enters a block of text and gets a randomly generated link. By default it never expires; the user can optionally set a timed expiration.
A user enters a paste's URL and views its contents.
Users are anonymous.
The service tracks monthly visit statistics, deletes expired pastes and has high availability.
The primer leaves out accounts, editing, visibility and custom shortlinks. This chapter adds custom aliases and redirectanalytics as extensions.
Assumptions (from the primer)
Traffic is not evenly distributed, and following a short link should be fast.
Pastes are text only, and analytics do not need to be real time.
10 million paste writes and 100 million paste reads per month: a 10:1 read-to-write ratio.
Back-of-the-envelope estimates
The primer uses a handy conversion: a month has about 2.5 million seconds. (Thirty days is 2.592 million, so the results below round slightly up.)
Writes=2.5×106 s10×106=4 per second on average
Reads=2.5×106 s100×106=40 per second on average
Peaks are higher, because traffic is uneven. The primer's rule of thumb: 1request per second is about 2.5 million a month, and 400 per second about 1 billion.
Size per paste. The primer assumes 1 KB of content plus a database row: shortlink 7 bytes, expiration_length_in_minutes 4 bytes, created_at 5 bytes and paste_path 255 bytes. That is about 1.27 KB in total.
New content per month=1.27 KB×10×106=12.7 GB
Over 3 years=12.7 GB×36≈450 GB,10×106×36=360 million shortlinks
How the numbers scale. Every figure here is linear in traffic. At 10 times the primer's volume (100 million writes a month), multiply each by 10: 40 writes and 400 reads per second on average, and 127 GB of new content per month. The challenge at the end of this chapter uses a much higher rate, to exercise the lab.
Cache sizing. Size the cache by the number of distinct shortlinks you want to keep hot, not by the number of requests. If the most popular 20% of the 360 million shortlinks are worth caching, at about 500 bytes per cached entry:
0.2×360×106×500 B=36 GB
The 20% is an assumption to check against real access logs, not a law (chapter 6).
For a new resource, HTTP's 201 Created status with a Locationheader naming it is the standard response (RFC 9110). For internal calls between services, the primer suggests RPC.
The primer’s Pastebin, scaled: one path for writes, one for reads, small rows in a database and the text in an object store.
Follow each use case. Creating a paste goes from the client, which finds the load balancer through DNS, to the Write API, which saves a small row in the metadata database and the text in the object store. Viewing a paste goes to the Read API, which looks the short link up through the memory cache, asking the database only when the cache misses, and fetches the text from the object store. Reads outnumber writes 10 to 1, so the two APIs are separate boxes that can grow apart; at 4 writes and 40 reads a second on average, two servers of each are enough, the second so that one can fail.
Clients: 1 × Users
DNS: 1 × DNS
Load balancer: 2 × Load balancer
Write API: 2 × App server
Read API: 2 × App server
Memory cache: 2 × Cache
Metadata database: 2 × Relational database
Object store: 1 × Object store
Clients sends requests to DNS
DNS sends requests to Load balancer
Load balancer sends requests to Write API
Load balancer sends requests to Read API
Write API sends requests to Metadata database
Write API sends requests to Object store
Read API sends requests to Memory cache
Memory cache sends requests to Metadata database
Read API sends requests to Object store
Key takeaways
10 million writes and 100 million reads a month are 4 writes and 40 reads a second on average; peaks are higher.
About 1.27 KB a paste gives 12.7 GB a month and about 450 GB, 360 million links, over three years; every figure grows linearly with traffic.
Size a cache by the distinct links you want hot, not by requests: 20% of 360 million links at 500 bytes is 36 GB, an assumption to check against real access logs.
Creating a paste answers 201 Created with a Locationheader; internal calls can use RPC.