Lesson 1.1• 4 min read

Performance versus scalability

Why making a system fast for one user does not make it ready for a million.

What you will learn

  • Tell a performance problem from a scalability problem.
  • Weigh scaling up one machine against scaling out to many.
  • Say what has to change before web servers can scale out.

The fundamental distinction

A system has a performance problem if it is slow for a single user under zero load. Common causes include database queries that use no index, synchronous blocking I/O, heavy cryptographic hashing on the request path, or unoptimized rendering routines.

A system has a scalability problem if it is fast for one user, but degrades or fails as traffic increases. Adding concurrent users causes thread pool exhaustion, database connection saturation, lock contention, or network bandwidth depletion.

Werner Vogels defines a service as scalable if adding resources increases its performance in proportion to the resources added. Ideally, doubling the servers doubles the throughput:

ΔPerformanceΔResources≈constant\frac{\Delta \text{Performance}}{\Delta \text{Resources}} \approx \text{constant}what its symbols mean

Vertical scaling (scaling up)

Scaling vertically means adding more CPU cores, RAM, and faster NVMe storage to a single server instance.

  • Advantages: Zero architectural changes; maintains strict ACID transactions without distributed consensus; simple single-node debugging.
  • Disadvantages: Hard hardware ceiling; diminishing returns due to bus and memory contention; the largest machines cost disproportionately more; one machine remains a single point of failure (SPOF).

Horizontal scaling (scaling out)

Scaling horizontally means adding more discrete compute nodes and distributing requests across them via load balancing. Scaling out on commodity machines is usually more cost-efficient and more available than scaling up one expensive server, but it adds complexity.

  • Stateless compute: To scale web servers horizontally, session state must be removed from local server memory and kept in a shared store (a database, or a cache such as Redis) or in signed client-side tokens (JWTs).
  • Downstream load: As the web tier scales out, caches and databases behind it must handle more simultaneous connections.
  • The load balancer itself: It can become a bottleneck, and a single load balancer is a single point of failure, so production systems run more than one.
Clients1× instanceLoad balancer2× instancesWeb servers4× instancesDatabase1× instanceClients1× instanceLoad balancer2× instancesWeb servers4× instancesDatabase1× instanceClients1× instanceLoad balancer2× instancesWeb servers4× instancesDatabase1× instance
Figure 1

Web servers scaled out behind a load balancer.

Follow a request from the clients: the load balancer sends it to one of four web servers, and that server reads and writes the one database. This is scaling out: to serve more users you add web servers instead of buying a bigger machine, which works only once no session state is kept in a server’s memory. Notice what adding web servers does not multiply: every request still passes the load balancer, and the database carries the load of all the web servers.

  • Clients: 1 × Users
  • Load balancer: 2 × Load balancer
  • Web servers: 4 × App server
  • Database: 1 × Relational database
  • Clients sends requests to Load balancer
  • Load balancer sends requests to Web servers
  • Web servers sends requests to Database

Key takeaways

  • Slow for one user at no load is a performance problem; fast for one user but failing as traffic grows is a scalability problem.
  • A service is scalable when adding resources adds performance in proportion.
  • Scaling up needs no redesign but hits a hardware ceiling and stays a single point of failure; scaling out is cheaper and more available but adds complexity.
  • To scale out, keep session state out of server memory, and expect the load balancer, caches and databases behind the web tier to carry more.
Sources5