Lesson 16.1• 9 min read

From one user to millions on AWS

The primer’s steps: one box, separate storage and database, horizontal web tier and CDN, cache and read replicas, autoscaling, then warehouse, NoSQL and queues.

What you will learn

  • Grow the primer's AWS design step by step, from one box to tens of millions of users.
  • Name the bottleneck behind each step and the AWS service that answers it.
  • Say what each step costs in complexity, security and money.

Scale in steps, driven by measurement

The primer's AWS exercise starts with one user and grows to tens of millions. It does not build the final design on day one. Each step follows the same loop:

Benchmark / load test⟶Profile⟶Fix the bottleneck, weighing trade-offs⟶Repeat\text{Benchmark / load test} \longrightarrow \text{Profile} \longrightarrow \text{Fix the bottleneck, weighing trade-offs} \longrightarrow \text{Repeat}

The primer marks growth with symbols (Users+, Users++, …) instead of user counts, because the point at which each step is needed depends on the workload. It also says that at each step "you might only need to implement one or two of these techniques".


The constraints

The primer's assumptions:

  • Traffic is not evenly distributed, and the data is relational.
  • 10 million users; 1 billion writes and 100 billion reads a month (100:1).
  • 1 KB of content per write.

Using its rule of thumb of 2.5 million seconds in a month:

Quantity Calculation Result
Writes per second 109/(2.5×106)10^9 / (2.5 \times 10^6) 400 on average
Reads per second 1011/(2.5×106)10^{11} / (2.5 \times 10^6) 40,000 on average
New content per month 1 KB×1091\text{ KB} \times 10^9 1 TB
New content in 3 years 36×1 TB36 \times 1\text{ TB} 36 TB

With an exact 30-day month (2,592,000 s) the rates are 386 writes and 38,580 reads per second. Peaks are higher, since traffic is uneven.


The steps

Start with one box. A web server on EC2 with MySQL on the same instance. Give it an Elastic IP, a public address that does not change on reboot, and point a Route 53 DNS record at it. Open only the ports needed: 80 and 443 to everyone, 22 only to allowed IPs. Scale vertically (a bigger instance) while monitoring CPU, memory, I/O and network. This setup has no redundancy and no failover, and vertical scaling gets expensive.

Users+: separate the parts. Profiling shows MySQL taking more memory and CPU, and user content filling the disk.

  • Move static content (user files, JS, CSS, images, videos) to an object store such as S3.
  • Move MySQL to its own managed instance, such as RDS, which offers multiple Availability Zones and encryption at rest.
  • Put everything in a VPC: a public subnet for the web server, a private subnet for everything else.

Now the web server and the database can scale independently. In exchange there is more to secure, and the web server must change to use the new components.

Users++: scale the web tier horizontally. The single web server is the bottleneck at peak.

  • Add a load balancer such as ELB, and terminate TLS on it.
  • Run several web servers across several Availability Zones.
  • Run MySQL as primary and standby across zones, for failover.
  • Separate web servers (acting as reverse proxies) from application servers, so each layer scales on its own.
  • Serve static, and some dynamic, content from a CDN such as CloudFront.

Users+++: relieve the database. The workload is read-heavy (100:1) and reads are overloading MySQL.

  • First tune MySQL's own cache. If that is not enough, add a memory cache such as ElastiCache for frequently read content.
  • Move session data into the cache too. The web servers become stateless, which makes autoscaling possible.
  • Add MySQL read replicas, with application logic that sends writes to the primary and reads to replicas.

Users++++: follow the daily curve. Traffic peaks in U.S. business hours and drops afterward. Paying for peak capacity all night is waste.

  • Add Auto Scaling groups, one per server type, each across several Availability Zones, with a minimum and maximum size.
  • Scale on a schedule for predictable load, or on CloudWatch metrics (CPU, latency, network, or a custom metric).
  • The primer lists the costs: more complexity, and a delay before new capacity arrives.
  • Automate operations, and keep monitoring at host level, load-balancer level, in logs, and from outside.

Users+++++: keep finding bottlenecks.

  • Keep only recent data in MySQL and move history to a warehouse such as Redshift.
  • Scale the memory cache for popular content and spikes. The read replicas may struggle with the cache misses.
  • 400 writes per second on average, with higher peaks, may be too much for one primary. Consider SQL scaling patterns: federation, sharding, denormalization, SQL tuning. Consider moving suitable data to a NoSQL store such as DynamoDB.
  • Do non-real-time work asynchronously: the application server puts a job on a queue such as SQS, and workers on EC2 or Lambda process it. In the primer's photo example, the worker makes the thumbnail, updates the database and stores the result in S3.

The diagram shows the end of this sequence.

Users1× instanceDNS2× instancesCDN2× instancesLoad balancer10× instancesWeb servers (reverseproxy)10× instancesApplication servers42× instancesMemory cache5× instancesMySQL read replicas2× instancesMySQL primary andstandby2× instancesQueue2× instancesWorkers2× instancesObject store1× instanceData warehouse2× instancesUsers1× instanceDNS2× instancesCDN2× instancesLoad balancer10× instancesWeb servers (reverseproxy)10× instancesApplication servers42× instancesMemory cache5× instancesMySQL read replicas2× instancesMySQL primary andstandby2× instancesQueue2× instancesWorkers2× instancesObject store1× instanceData warehouse2× instances
Figure 1

The primer’s design at its last step: DNS and a CDN, a load balancer, web and application servers, a memory cache, a MySQL primary with read replicas, a queue with workers, an object store and a data warehouse.

Follow a read, then a job. DNS sends each user to the CDN, which serves static content, and some dynamic content, from its copies, and fetches static files from the object store. Other requests go to the load balancer, which spreads them over the web servers, and each web server passes its requests to an application server. An application server reads from the memory cache, and a miss goes to the MySQL read replicas; writes go to the MySQL primary, which replicates them to the replicas and has a standby in another zone. Work that need not happen at once goes on the queue, and workers take it from there. Older data moves from MySQL to the data warehouse. Each tier of servers is an Auto Scaling group across several availability zones. The primer does not say how many reads the CDN answers, so the application servers are counted for all of its 40,000 reads and 400 writes a second, at the lab’s 1,000 requests a second a server, plus one spare; the load balancer and the web servers at 5,000 a second, plus one spare; the memory cache at 12,000 a second a node, plus one spare. The primer gives no counts for the rest, so each is drawn with two.

  • Users: 1 × Users
  • DNS: 2 × DNS
  • CDN: 2 × CDN
  • Load balancer: 10 × Load balancer
  • Web servers (reverse proxy): 10 × Load balancer
  • Application servers: 42 × App server
  • Memory cache: 5 × Cache
  • MySQL read replicas: 2 × Read replica
  • MySQL primary and standby: 2 × Relational database
  • Queue: 2 × Queue
  • Workers: 2 × App server
  • Object store: 1 × Object store
  • Data warehouse: 2 × Relational database
  • Users sends requests to DNS
  • DNS sends requests to CDN
  • CDN sends requests to Load balancer
  • CDN sends requests to Object store
  • Load balancer sends requests to Web servers (reverse proxy)
  • Web servers (reverse proxy) sends requests to Application servers
  • Application servers sends requests to Memory cache
  • Memory cache sends requests to MySQL read replicas
  • Application servers sends requests to MySQL primary and standby
  • MySQL read replicas keeps a copy of MySQL primary and standby
  • Application servers sends requests to Queue
  • Queue sends requests to Workers
  • Workers sends requests to MySQL primary and standby
  • Workers sends requests to Object store
  • MySQL primary and standby sends requests to Data warehouse

Key takeaways

  • Every step follows the same loop: benchmark, profile, fix the bottleneck, repeat; the primer marks steps with Users+ symbols because the thresholds depend on the workload.
  • Separate first (S3 for static content, RDS for MySQL, a VPC), then scale the web tier behind a load balancer across zones with a standby database and a CDN.
  • For the 100:1 read load, add a cache and read replicas; moving sessions into the cache makes web servers stateless, so Auto Scaling can follow the daily curve.
  • Then move history to a warehouse, consider SQL scaling patterns for writes, and put non-real-time work on queues.
Sources2