The primer’s use cases and estimates (40 bytes per transaction, 40 GB a month, 400 writes and 40,000 reads per second) and its data and API.
What you will learn
State the sales rank feature's use cases and assumptions as the primer does.
Estimate its transaction and read rates and its storage.
See why the result itself is small, and where the hard part lies.
The problem
The primer's Design Amazon's sales rank by category feature covers only the ranking. The rest of the e-commerce site is out of scope.
Use cases (from the primer)
The service calculates the past week's most popular products by category.
A user views the past week's most popular products by category.
The service has high availability.
Assumptions (from the primer)
Traffic is not evenly distributed.
Items can be in several categories, cannot change categories, and there are no subcategories.
Results must be updated hourly; more popular products might need more frequent updates.
10 million products in 1,000 categories.
1 billion transactions and 100 billion read requests per month: a 100:1 read-to-write ratio.
Amazon describes its own Best Sellers Rank in its Kindle Direct Publishing help. Ranks are "relative", they "reflect recent and historical activity, with recent activity weighted more heavily", and they are "updated at least once per day". Amazon does not publish the formula.
40 B×109=40 GB of new transactions per month,1.44 TB in 3 years
With 400 requests per second ≈ 1 billion a month:
1×400=400 transactions per second,100×400=40,000 reads per second
on average, and higher at peak. The exact 30-day figures are about 386 and 38,580 per second.
The result is small. Keeping the top 1,000 products for each of 1,000 categories is 106 rows. Even at a few hundred bytes per row with indexes, that fits in memory. The hard parts are computing it from a billion transactions a month and serving 40,000 reads a second.
The primer stores these raw logs in an object store such as Amazon S3. It writes the computed ranking to a sales_rank table with id, category_id, total_sold and product_id, indexed on id, category_id and product_id.
The Read API serves this from the sales_rank table, through a cache (lesson 4).
Shoppers1× instanceWeb server (reverseproxy)10× instancesSales API2× instancesRead API41× instancesObject store (log files)1× instanceSales Rank Service2× instancesSQL database(sales_rank)2× instancesShoppers1× instanceWeb server (reverseproxy)10× instancesSales API2× instancesRead API41× instancesObject store (log files)1× instanceSales Rank Service2× instancesSQL database(sales_rank)2× instances
Figure 1
The primer’s design: the Sales API logs each purchase to an object store, an hourly MapReduce writes the rankings to a SQL database, and the Read API serves them.
Follow a purchase, then a read. The web server passes the purchase to the Sales API, which writes it as a line in its log files, kept in an object store. Every hour the Sales Rank Service runs a MapReduce over the past week of those logs and writes each category’s ranking to the sales_rank table in the SQL database. A shopper’s request for a category goes through the web server to the Read API, which reads that table. The APIs are counted at the lab’s 1,000 requests a second a server for the primer’s 40,000 reads and 400 purchases a second, plus one spare, and the web server at 5,000 a second, plus one spare. The primer gives no count for the Sales Rank Service or the database, so each is drawn with two; lesson 4 scales the reads.
Shoppers sends requests to Web server (reverse proxy)
Web server (reverse proxy) sends requests to Sales API
Web server (reverse proxy) sends requests to Read API
Sales API sends requests to Object store (log files)
Sales Rank Service sends requests to Object store (log files)
Sales Rank Service sends requests to SQL database (sales_rank)
Read API sends requests to SQL database (sales_rank)
Key takeaways
1 billion transactions and 100 billion reads a month are about 400 writes and 40,000 reads a second on average, a 100:1 ratio.
At about 40 bytes a transaction, the raw log grows by 40 GB a month.
The top 1,000 products for each of 1,000 categories is only 106 rows, which fits in memory; the hard parts are computing it from the log and serving 40,000 reads a second.