Lesson 6.1• 8 min read

Caching tiers and cache placement

Browser, CDN edge, reverse proxies, distributed Redis, and DB buffer pools.

What you will learn

  • Place caches along the request path, from the client to the database, and say what each tier holds.
  • Compute the average response time from the hit ratio, and how the hit ratio changes database load.
  • Size an application cache for its working set.

The memory hierarchy and the cost of disk I/O

Data access times span many orders of magnitude (chapter 2's latency numbers, from about 2012): a main memory reference takes about 100 ns100\text{ ns}, a random 4 KB SSD read about 150 μs150\,\mu\text{s}what its symbols mean, and a round trip within a datacenter about 500 μs500\,\mu\text{s}. Data held in RAM is much faster to reach than data on disk, which is why in-memory caches such as Memcached and Redis sit between applications and their databases.

Caching works because accesses are not uniform: recently used data tends to be used again, and popular items take a large share of requests. A cache in front of a database also absorbs uneven load: databases do best when reads and writes spread evenly across partitions, and popular items skew that distribution.


The caching hierarchy

Caches can sit at every stage of the request path:

Tier Where Examples What it typically holds
1. Client The OS or browser HTTP cache (Cache-Control, ETag) Static files, recently fetched responses
2. CDN Edge locations near users CloudFront Static media and other cacheable responses
3. Web server Reverse proxy or web cache NGINX, Varnish Static and dynamic responses, without contacting the application servers
4. Application cache In-memory key-value stores between the app and its storage Memcached, Redis Sessions, fully rendered pages, activity streams, user graph data
5. Database The database's own memory PostgreSQL shared buffers, InnoDB buffer pool Recently used pages; usually some caching is on by default, and tuning it for your workload helps further

Within the application cache you can cache at two levels: query results (hash the query as the key; hard to invalidate, because one changed cell can affect many cached queries) or objects (assemble the data into an object and cache that; remove it when its underlying data changes). Avoid file-based caching, which makes cloning and autoscaling servers harder.


Cache hit ratio mathematics: The effective response time equation

The performance of any cached system is strictly governed by its Hit Ratio (hhwhat its symbols mean):

h=Cache HitsCache Hits+Cache Missesh = \frac{\text{Cache Hits}}{\text{Cache Hits} + \text{Cache Misses}}what its symbols mean

The average response time (TeffectiveT_{\text{effective}}what its symbols mean) clients see is:

Teffective=h⋅Tcache+(1−h)⋅(Tcache+Tdb)T_{\text{effective}} = h \cdot T_{\text{cache}} + (1 - h) \cdot (T_{\text{cache}} + T_{\text{db}})what its symbols mean

Why 95% vs. 99% hit ratio matters

Consider a system where the cache answers in Tcache=2 msT_{\text{cache}} = 2\text{ ms}what its symbols mean and the database in Tdb=40 msT_{\text{db}} = 40\text{ ms}what its symbols mean:

  • At h=80%h = 80\%: Teffective=0.80(2)+0.20(42)=1.6+8.4=10.0 msT_{\text{effective}} = 0.80(2) + 0.20(42) = 1.6 + 8.4 = \mathbf{10.0\text{ ms}} (Database absorbs 20% of traffic).
  • At h=95%h = 95\%: Teffective=0.95(2)+0.05(42)=1.9+2.1=4.0 msT_{\text{effective}} = 0.95(2) + 0.05(42) = 1.9 + 2.1 = \mathbf{4.0\text{ ms}} (Database absorbs 5% of traffic).
  • At h=99%h = 99\%: Teffective=0.99(2)+0.01(42)=1.98+0.42=2.4 msT_{\text{effective}} = 0.99(2) + 0.01(42) = 1.98 + 0.42 = \mathbf{2.4\text{ ms}} (Database absorbs only 1% of traffic).

A drop from 99%99\% to 95%95\% hit ratio multiplies database query load by 5×5\times (1%→5%1\% \to 5\%).


Working set estimation & sizing formula

The working set is the subset of data actively requested over a time window (e.g. 24 hours). Size the cache to hold it:

MRAM=Working set keys×Average item size×(1+Overhead margin)M_{\text{RAM}} = \text{Working set keys} \times \text{Average item size} \times (1 + \text{Overhead margin})what its symbols mean

Assume, for illustration, that 20% of a catalog's items receive most of its reads (an 80/20 split; measure your own traffic to find the real share):

Example: 10M products×20%×2 KB×1.25=2M×2.5 KB=5.0 GB RAM\text{Example: } 10\text{M products} \times 20\% \times 2\text{ KB} \times 1.25 = 2\text{M} \times 2.5\text{ KB} = \mathbf{5.0\text{ GB RAM}}
Shoppers1× instanceDNS1× instanceCDN1× instanceLoad balancer3× instancesApp servers4× instancesCache4× instancesDatabase1× instanceShoppers1× instanceDNS1× instanceCDN1× instanceLoad balancer3× instancesApp servers4× instancesCache4× instancesDatabase1× instance
Figure 1

Caches at each tier of the request path.

Follow a request from a shopper: the browser’s own cache answers first, then the CDN, and what reaches the origin goes through the load balancer to an app server, which asks the cache before the database. Each tier answers what it can, so only misses travel further. The hit ratio at the cache decides both the average response time and how much load reaches the database.

  • Shoppers: 1 × Users
  • DNS: 1 × DNS
  • CDN: 1 × CDN
  • Load balancer: 3 × Load balancer
  • App servers: 4 × App server
  • Cache: 4 × Cache
  • Database: 1 × Relational database
  • Shoppers sends requests to DNS
  • DNS sends requests to CDN
  • DNS sends requests to Load balancer
  • CDN sends requests to Load balancer
  • Load balancer sends requests to App servers
  • App servers sends requests to Cache
  • Cache sends requests to Database
  • App servers sends requests to Database

Key takeaways

  • Caching works because accesses are not uniform: recent and popular items take most requests.
  • Cache objects rather than query results where you can: an object is easy to invalidate when its data changes.
  • Teffective=h⋅Tcache+(1−h)(Tcache+Tdb)T_{\text{effective}} = h \cdot T_{\text{cache}} + (1 - h)(T_{\text{cache}} + T_{\text{db}})what its symbols mean; falling from 99% to 95% hits sends five times the queries to the database.
  • Size RAM as working-set keys × item size × (1 + overhead), using the share of items your own traffic shows to be hot.
Sources3