Browser, CDN edge, reverse proxies, distributed Redis, and DB buffer pools.
What you will learn
Place caches along the request path, from the client to the database, and say what each tier holds.
Compute the average response time from the hit ratio, and how the hit ratio changes database load.
Size an application cache for its working set.
The memory hierarchy and the cost of disk I/O
Data access times span many orders of magnitude (chapter 2's latency numbers, from about 2012): a main memory reference takes about 100 ns, a random 4 KBSSD read about 150μswhat its symbols mean, and a round trip within a datacenter about 500μs. Data held in RAM is much faster to reach than data on disk, which is why in-memory caches such as Memcached and Redis sit between applications and their databases.
Caching works because accesses are not uniform: recently used data tends to be used again, and popular items take a large share of requests. A cache in front of a database also absorbs uneven load: databases do best when reads and writes spread evenly across partitions, and popular items skew that distribution.
The caching hierarchy
Caches can sit at every stage of the request path:
Tier
Where
Examples
What it typically holds
1. Client
The OS or browser
HTTP cache (Cache-Control, ETag)
Static files, recently fetched responses
2. CDN
Edge locations near users
CloudFront
Static media and other cacheable responses
3. Web server
Reverse proxy or web cache
NGINX, Varnish
Static and dynamic responses, without contacting the application servers
4. Application cache
In-memory key-value stores between the app and its storage
Memcached, Redis
Sessions, fully rendered pages, activity streams, user graph data
5. Database
The database's own memory
PostgreSQL shared buffers, InnoDBbuffer pool
Recently used pages; usually some caching is on by default, and tuning it for your workload helps further
Within the application cache you can cache at two levels: query results (hash the query as the key; hard to invalidate, because one changed cell can affect many cached queries) or objects (assemble the data into an object and cache that; remove it when its underlying data changes). Avoid file-based caching, which makes cloning and autoscaling servers harder.
Cache hit ratio mathematics: The effective response time equation
The performance of any cached system is strictly governed by its Hit Ratio (hwhat its symbols mean):
h=Cache Hits+Cache MissesCache Hitswhat its symbols mean
The average response time (Teffectivewhat its symbols mean)clients see is:
Teffective=h⋅Tcache+(1−h)⋅(Tcache+Tdb)what its symbols mean
Why 95% vs. 99% hit ratio matters
Consider a system where the cache answers in Tcache=2 mswhat its symbols mean and the database in Tdb=40 mswhat its symbols mean:
At h=80%: Teffective=0.80(2)+0.20(42)=1.6+8.4=10.0 ms (Database absorbs 20% of traffic).
At h=95%: Teffective=0.95(2)+0.05(42)=1.9+2.1=4.0 ms (Database absorbs 5% of traffic).
At h=99%: Teffective=0.99(2)+0.01(42)=1.98+0.42=2.4 ms (Database absorbs only 1% of traffic).
A drop from 99% to 95% hit ratio multiplies database query load by 5× (1%→5%).
Working set estimation & sizing formula
The working set is the subset of data actively requested over a time window (e.g. 24 hours). Size the cache to hold it:
MRAM=Working set keys×Average item size×(1+Overhead margin)what its symbols mean
Assume, for illustration, that 20% of a catalog's items receive most of its reads (an 80/20 split; measure your own traffic to find the real share):
Follow a request from a shopper: the browser’s own cache answers first, then the CDN, and what reaches the origin goes through the load balancer to an app server, which asks the cache before the database. Each tier answers what it can, so only misses travel further. The hit ratio at the cache decides both the average response time and how much load reaches the database.
Shoppers: 1 × Users
DNS: 1 × DNS
CDN: 1 × CDN
Load balancer: 3 × Load balancer
App servers: 4 × App server
Cache: 4 × Cache
Database: 1 × Relational database
Shoppers sends requests to DNS
DNS sends requests to CDN
DNS sends requests to Load balancer
CDN sends requests to Load balancer
Load balancer sends requests to App servers
App servers sends requests to Cache
Cache sends requests to Database
App servers sends requests to Database
Key takeaways
Caching works because accesses are not uniform: recent and popular items take most requests.
Cache objects rather than query results where you can: an object is easy to invalidate when its data changes.
Teffective=h⋅Tcache+(1−h)(Tcache+Tdb)what its symbols mean; falling from 99% to 95% hits sends five times the queries to the database.
Size RAM as working-set keys × item size × (1 + overhead), using the share of items your own traffic shows to be hot.