Lesson 9.4• 9 min read

Redirects, caching and visit analytics

HTTP 301 versus 302 and 307 redirects, caching in the browser, at the CDN and in the application, and counting visits from web server logs or from click events.

What you will learn

  • Choose between a 301 and a 302 redirect by what each does to load, counting and control.
  • Cache redirects in the browser, at the CDN and in an application cache.
  • Count visits with batch MapReduce over web server logs, or near real time with a stream.

301 or 302?

A shortener answers a click with a redirect and a Location header. Which status code it uses decides whether later clicks reach it at all. From RFC 9110:

Status Meaning Caching Method on redirect
301 Moved Permanently New permanent URI; "future references … ought to use" it Heuristically cacheable: a cache may reuse it without asking again, unless told otherwise A client may change POST to GET
308 Permanent Redirect Like 301 Heuristically cacheable Method preserved
302 Found Resource "resides temporarily under a different URI"; keep using the original Not cached unless the response gives explicit freshness A client may change POST to GET
307 Temporary Redirect Like 302 Like 302 Method preserved

The trade-off:

  • With a 301, a browser may answer repeat clicks from its own cache. That saves the round trip and server load, but those clicks never reach you. You cannot count them, and you cannot change or expire the link for anyone who has it cached.
  • With a 302 or 307, every click comes back to you unless you send explicit cache headers. Counting is accurate and links stay editable, at the cost of serving every click.

Services choose differently, depending on whether they care more about load or about counting and control. Ask which one the interviewer wants.


Caching redirects and content

The primer's scaled design puts the popular content in a memory cache, which also absorbs uneven traffic and spikes. Read replicas handle the misses, and a CDN serves static content.

A redirect can be cached at three levels:

  1. Browser: a 301, or any redirect sent with explicit freshness such as Cache-Control: max-age=60 (RFC 9111).
  2. CDN: a 301 (RFC 9110 lets any cache store it heuristically, not only the browser), or any redirect that allows shared caching, for example Cache-Control: public, max-age=60. Clicks the CDN answers never reach your servers. They appear only in the CDN's access logs.
  3. Application cache (Redis or Memcached): key shortlink, value the target URL or paste location, evicted by LRU or LFU (chapter 6). Your servers still see every click it answers.

A short max-age is a compromise: most of the load goes to the edge, and changes take effect within a minute.


Counting visits

The primer's analytics are deliberately simple. The requirement is monthly visit statistics, not real-time ones. So it runs MapReduce over the web server logs, counting hits per shortlink per month:

def mapper(self, _, line):
    url = self.extract_url(line)
    period = self.extract_year_month(line)
    yield (period, url), 1

def reducer(self, key, values):
    yield key, sum(values)

The results go to an analytics database. The primer suggests a data warehouse such as Amazon Redshift or Google BigQuery.

If analytics must be near real time, publish a click event to a log such as Kafka after sending the redirect, so the click never waits on it. Then aggregate it with a stream processor (chapters 7 and 15). Either way, clicks answered by a browser cache are invisible, and clicks answered by a CDN appear only in the CDN's logs. Count those logs too, or choose redirect headers that send clicks to you.

Who sees a click?

Any cache, the browser or a CDN, may reuse a 301; a 302 is cached only if the response says so. A redirect that a cache answers never reaches your servers. Here every click falls within the cached copy’s lifetime.

Redirect status
Clicks
0
Seen by your servers
0
Only in CDN logs
0
Never seen
0
ClickVisitorAnswered byClick event
Browsers1× instanceCDN1× instanceLoad balancer2× instancesRedirect servers2× instancesApplication cache2× instancesLinks database2× instancesMessage broker (clickevents)1× instanceStream processors2× instancesBrowsers1× instanceCDN1× instanceLoad balancer2× instancesRedirect servers2× instancesApplication cache2× instancesLinks database2× instancesMessage broker (clickevents)1× instanceStream processors2× instances
Figure 4

Redirects answered at the CDN or by the redirect servers, with click events published for counting near real time.

Follow a click. The CDN answers it if it holds a cached redirect for the link, and then the click appears only in the CDN’s access logs. Otherwise the load balancer passes it to a redirect server, which looks the short link up through the application cache, asking the database when the cache misses, and answers with the redirect. After answering, the redirect server publishes a click event to the message broker, so the click never waits on counting, and stream processors count the events. A click the browser answers from its own cache goes no further than the browser.

  • Browsers: 1 × Users
  • CDN: 1 × CDN
  • Load balancer: 2 × Load balancer
  • Redirect servers: 2 × App server
  • Application cache: 2 × Cache
  • Links database: 2 × Relational database
  • Message broker (click events): 1 × Queue
  • Stream processors: 2 × App server
  • Browsers sends requests to CDN
  • CDN sends requests to Load balancer
  • Load balancer sends requests to Redirect servers
  • Redirect servers sends requests to Application cache
  • Application cache sends requests to Links database
  • Redirect servers sends requests to Message broker (click events)
  • Message broker (click events) sends requests to Stream processors

Key takeaways

  • A 301 can be cached by browsers and CDNs, saving load but losing the clicks and the power to change the link; a 302 brings every click back to you.
  • A short max-age sends most of the load to the edge while changes still take effect within a minute.
  • Monthly statistics need only MapReduce over web server logs into a warehouse.
  • For near-real-time counts, publish a click event after sending the redirect, then aggregate the stream; clicks answered by a browser cache are invisible, and clicks answered by a CDN appear only in the CDN's logs.
Sources5