Lesson 10.1• 9 min read

Requirements, estimates and the timeline API

The primer’s use cases and estimates: 100 million active users, 6,000 tweets and 100,000 reads a second, 150 TB of new content a month, and the REST API.

What you will learn

  • State the Twitter timeline and search use cases and assumptions as the primer does.
  • Estimate tweet storage, fan-out deliveries and read rates.
  • Size the home timeline cache, including its replicas.

The problem

The primer's Design the Twitter timeline and search notes that designing the Facebook feed and Facebook search are similar questions.

Use cases (from the primer)

  • A user posts a tweet; the service pushes it to followers, sending push notifications and emails.
  • A user views the user timeline (their own activity) and the home timeline (activity from people they follow).
  • A user searches keywords.
  • The service has high availability.

Out of scope: pushing tweets to Twitter's Firehose and other streams, visibility rules (such as hiding @replies) and analytics.

Assumptions (from the primer)

  • Traffic is not evenly distributed. Posting should be fast, and so should fanning out to followers, "unless you have millions of followers".
  • Viewing the timeline and searching should be fast. Twitter is read-heavy, so optimize for fast reads; ingesting tweets is write-heavy.
  • 100 million active users.
  • 500 million tweets per day (15 billion a month), each delivered to 10 followers on average: 5 billion deliveries a day, 150 billion a month.
  • 250 billion read requests and 10 billion searches per month.

Back-of-the-envelope estimates

Using the primer's conversion, 400 requests per second ≈ 1 billion requests per month:

Quantity Per month Per second
Read requests 250 billion 250×400=100,000250 \times 400 = 100{,}000
Tweets 15 billion 15×400=6,00015 \times 400 = 6{,}000
Fan-out deliveries 150 billion 150×400=60,000150 \times 400 = 60{,}000
Searches 10 billion 10×400=4,00010 \times 400 = 4{,}000

Size per tweet (primer): tweet_id 8 bytes, user_id 32 bytes, text 140 bytes, media 10 KB on average. That is about 10 KB per tweet.

10 KB×500×106×30=150 TB of new content per month10\text{ KB} \times 500 \times 10^6 \times 30 = 150\text{ TB of new content per month}
150 TB×36=5.4 PB in 3 years150\text{ TB} \times 36 = 5.4\text{ PB in 3 years}

For comparison, Twitter reported 150 million active users and about 5,000 tweets per second on average (7,000 at the daily peak) in 2013, and 300,000 queries per second to generate timelines. So the primer's figures are the right order of magnitude.


Sizing the timeline cache

The primer keeps home timelines in a memory cache such as Redis. Each entry is a tweet ID, a user ID and a metadata byte: 8+8+1=178 + 8 + 1 = 17 bytes. It keeps "only several hundred tweets for each home timeline", and only for active users. Twitter capped home timelines at 800 entries. With the primer's 100 million active users:

100×106×800×17 B≈1.36 TB100 \times 10^6 \times 800 \times 17\text{ B} \approx 1.36\text{ TB}

Twitter reported "a couple of terabytes of RAM" for this cluster, and said each tweet is replicated on 3 machines. Replication multiplies the RAM you need, so size the cluster for the copies too.


API

The primer's REST API:

POST /api/v1/tweet
{ "user_id": "123", "status": "hello world!", "media_ids": "ABC987" }

→ { "created_at": "Wed Sep 05 00:37:15 +0000 2012", "status": "hello world!",
    "tweet_id": "987", "user_id": "123", ... }
GET /api/v1/home_timeline?user_id=123

→ [ { "user_id": "456", "tweet_id": "123", "status": "foo" }, ... ]

The user timeline API is similar, and search is GET /api/v1/search?query=hello+world. The primer's request also carries an auth_token, omitted here. For internal calls it suggests RPC.

Clients1× instanceLoad balancer23× instancesWrite API7× instancesRead API101× instancesSearch API5× instancesFan-out service7× instancesTweets database(user timelines)8× instancesObject store (media)1× instanceTimeline cache15× instancesSearch cluster20× instancesClients1× instanceLoad balancer23× instancesWrite API7× instancesRead API101× instancesSearch API5× instancesFan-out service7× instancesTweets database(user timelines)8× instancesObject store (media)1× instanceTimeline cache15× instancesSearch cluster20× instancesClients1× instanceLoad balancer23× instancesWrite API7× instancesRead API101× instancesSearch API5× instancesFan-out service7× instancesTweets database(user timelines)8× instancesObject store (media)1× instanceTimeline cache15× instancesSearch cluster20× instances
Figure 1

The primer’s design at its estimated rates: separate paths for posting, reading home timelines and searching, with home timelines in a memory cache.

Follow each use case. Posting goes through the load balancer to the Write API, which stores the tweet in the author’s user timeline in the database and its media in the object store, then hands it to the fan-out service. The fan-out service puts the tweet’s ID into the home timeline of each of the author’s followers in the timeline cache, and passes the tweet to the search cluster to index. Viewing a home timeline goes to the Read API, which reads it from the cache; searching goes to the Search API, which asks the search cluster. The three APIs and the fan-out service are counted at about 1,000 requests a second per server, the lab’s figure, plus one spare. The load balancer is counted at 5,000 a second each and the timeline cache at 12,000 a second per node, for its reads and inserts together, each plus one spare; the database is four shards of a leader and a standby, and the search cluster four shards of five (figure 4).

  • Clients: 1 × Users
  • Load balancer: 23 × Load balancer
  • Write API: 7 × App server
  • Read API: 101 × App server
  • Search API: 5 × App server
  • Fan-out service: 7 × App server
  • Tweets database (user timelines): 8 × Relational database
  • Object store (media): 1 × Object store
  • Timeline cache: 15 × Cache
  • Search cluster: 20 × App server
  • Clients sends requests to Load balancer
  • Load balancer sends requests to Write API
  • Load balancer sends requests to Read API
  • Load balancer sends requests to Search API
  • Write API sends requests to Tweets database (user timelines)
  • Write API sends requests to Object store (media)
  • Write API sends requests to Fan-out service
  • Fan-out service sends requests to Timeline cache
  • Fan-out service sends requests to Search cluster
  • Read API sends requests to Timeline cache
  • Search API sends requests to Search cluster

Key takeaways

  • Twitter is read-heavy, so optimize for fast timeline reads; ingesting tweets and fanning them out is write-heavy.
  • 500 million tweets a day at about 10 KB each is 150 TB of new content a month, 5.4 PB in three years.
  • 800 entries of 17 bytes for 100 million active users is about 1.36 TB of cache, before replication multiplies it.
Sources4