Lesson Method 3• 5 min read

Step 3: Core components

Take the use cases one at a time and go deep where the problem is hardest: data model, key generation, API.

What you will learn

  • Choose which components to take deep, and say which stay shallow.
  • Work one component through in detail: generating short links for Pastebin.
  • Write the data model and API from what step 1 sized.

Step 3: go deep where it matters

The third step is to dive into the details of each core component. The primer's own example is the short-link service, and for it, it lists what to discuss:

  • Generating and storing a hash of the full URL: MD5 and Base62, hash collisions, SQL or NoSQL, and the database schema.
  • Translating a hashed URL to the full URL: the database lookup.
  • The API and object-oriented design.

Two habits make this step work. Take the use cases one at a time, as the worked solutions do (each has a heading "Use case: …"). And ask how much code you are expected to write: the primer repeats that advice in several solutions.

The data model and the API

The table needs only what step 1 sized: the short link, the expiry, the creation time and the path to the text in the object store. The text itself is in the object store.

The public API is REST: a POST with the paste's contents and expiry returns the short link; a GET with the short link returns the contents. Internal calls between services could use RPC. The read path checks the database for the link, fetches the text from the object store if it is there, and returns an error if not.

Depth, not breadth

You will not have time to detail every box. Choose the components where the use cases are hardest or the numbers are largest, and say which ones you are leaving shallow and why. In Pastebin that is link generation and the read path; analytics can stay a one-line batch job because the assumptions allow it.

Client1× instanceWrite API (MD5,Base62)1× instancePastes table (key:short link)1× instancePaste text1× instanceClient1× instanceWrite API (MD5,Base62)1× instancePastes table (key:short link)1× instancePaste text1× instanceClient1× instanceWrite API (MD5,Base62)1× instancePastes table (key:short link)1× instancePaste text1× instance
Figure 2

The write path in detail: the Write API makes a 7-character Base62 link, stores a row keyed by it, and puts the text in the object store.

Follow a new paste. The client sends its text and expiry to the Write API. The Write API takes the MD5 hash of the user’s IP address and the time, encodes it in Base62 and keeps the first 7 characters as the short link. It puts the text in the object store and saves a row keyed by the link in the pastes table; because the link is the primary key, the database rejects a duplicate, and the Write API then generates another link. The response is the short link. The web server in front of the Write API is left out.

  • Client: 1 × Users
  • Write API (MD5, Base62): 1 × App server
  • Pastes table (key: short link): 1 × Relational database
  • Paste text: 1 × Object store
  • Client sends requests to Write API (MD5, Base62)
  • Write API (MD5, Base62) sends requests to Pastes table (key: short link)
  • Write API (MD5, Base62) sends requests to Paste text

Key takeaways

  • Take the use cases one at a time, and ask how much code you are expected to write.
  • Seven Base62 characters give 627≈3.5×101262^7 \approx 3.5 \times 10^{12} links, far more than needed; making the link the primary key lets the database reject duplicates.
  • The table holds only the small fields step 1 sized; the text lives in the object store.
  • Go deep where the use cases are hardest or the numbers largest.
Sources4