Lesson 8.2• 10 min read

Circuit breakers, retries and bulkheads

Stopping cascading failures and retry storms: three-state circuit breakers, exponential backoff with full jitter, and bulkheads.

What you will learn

  • Use Little's law to see how one slow dependency starves a caller's threads.
  • Follow a circuit breaker through its closed, open and half-open states.
  • Stop retry storms with retry limits and jitter, and contain failures with bulkheads.

Cascading failure dynamics and thread pool starvation

In a call tree, failure travels upward. Consider an orchestrator with 100100 worker threads that calls several downstream services and holds a thread for each call in flight.

Suppose one non-critical dependency, Recommendations, slows from 20 ms20\text{ ms} to 4,000 ms4{,}000\text{ ms}. By Little's law (chapter 1), the threads busy with it equal the call rate times the call duration. So at 4 s4\text{ s} per call, just 100/4=25100 / 4 = 25 calls per second to Recommendations hold all 100100 threads. At 20 ms20\text{ ms}, the same threads could serve 5,0005{,}000 calls per second.

Once every thread is waiting on Recommendations, checkout and payment requests have no thread to run on either. Fowler puts it this way: with many callers on an unresponsive supplier "you can run out of critical resources leading to cascading failures across multiple systems."


The circuit breaker (three states)

Fowler: "You wrap a protected function call in a circuit breaker object, which monitors for failures."

             failure rate ≥ threshold
   ┌────────┐ ───────────────────────> ┌────────┐
   │ CLOSED │                          │  OPEN  │  calls rejected
   └────────┘ <──────┐        ┌─────── └────────┘  without a network call
                     │        │  trial calls      │
       trial calls   │        │  fail too often   │ wait duration
       mostly succeed│        │                   │ elapses
                     │   ┌───────────┐ <──────────┘
                     └── │ HALF-OPEN │
                         └───────────┘

Exact behavior varies by library. The numbers below are resilience4j's documented defaults:

  1. Closed: calls pass through, and outcomes are recorded in a sliding window. In the default count-based window, that is the last 100100 calls. No failure rate is computed until a minimum number of calls (100100 by default) is recorded. When the failure rate reaches the threshold (50%50\% by default), the breaker opens.
  2. Open: calls are rejected at once with a CallNotPermittedException, so the caller can serve a fallback without waiting on the network. The breaker stays open for a configured wait (60 s60\text{ s} by default).
  3. Half-open: a limited number of trial calls (1010 by default) are let through. If their failure rate is still at or above the threshold, the breaker opens again; otherwise it closes. Fowler's original description uses a single trial call.

Slow calls count too. A timeout lets a caller stop waiting for an answer it no longer expects, and a call that times out has failed. Release It! notes that a circuit breaker can count timeouts and trip once too many occur, and that breakers guard against slow responses as well as errors. So a dependency that is slow, not down, also opens the breaker, and while it is open the caller stops waiting on it at all.

The AWS Builders' Library adds a caution: circuit breakers "introduce modal behavior into systems that can be difficult to test".


Retry storms and backoff with jitter

When calls fail because the service is overloaded, "retries can make the overload worse instead of better" (AWS Builders' Library). If every call fails, each layer that retries multiplies the load:

Load=λbase×(1+MaxRetries)\text{Load} = \lambda_{\text{base}} \times (1 + \text{MaxRetries})what its symbols mean

So 8,000 RPS8{,}000\text{ RPS} with MaxRetries=3\text{MaxRetries} = 3 becomes 32,000 RPS32{,}000\text{ RPS} at the failing service. Retries at several layers multiply too. AWS's example is a five-layer stack where each layer tries three times: the database sees 35=2433^5 = 243 times the load, "making it unlikely to ever recover". Retry only at one layer, and cap retries, for example with a token bucket that allows retries only while tokens remain.

Full jitter (Marc Brooker's AWS post, chapter 7) spreads the retries that remain:

tsleep=random(0, min⁡(M, B⋅2i))t_{\text{sleep}} = \text{random}\left(0,\ \min(M,\ B \cdot 2^i)\right)what its symbols mean

Here BBwhat its symbols mean is the base delay, MMwhat its symbols mean the cap and iiwhat its symbols mean the retry number. With the example values B=100 msB = 100\text{ ms} and M=10 sM = 10\text{ s}, the ceiling doubles 100, 200, 400 … ms until it reaches 10 s at the eighth retry. Clients that all failed at the same moment then retry at different moments instead of in lockstep.


Bulkheads

The pattern is named after the watertight compartments of a ship's hull: "if the hull of a ship is compromised, only the damaged section fills with water" (Azure Architecture Center). A consumer gives each dependency its own pool, such as a connection pool, thread pool or semaphore limit. Then "if a service begins to fail, it only affects the connection pool assigned for that service." In the example above, a separate pool for Recommendations would leave checkout's threads free.

The price is less efficient use of resources. Azure suggests combining bulkheads with retries, circuit breakers and throttling.

Circuit breaker and retries

Closed, open and half-open transitions over a count-based window, and how retries multiply downstream calls.

The 10-call window and 4 trial calls are sized for this demo; real defaults are larger (resilience4j: 100 calls and 10 trials).

Breaker state
Closed
Failure rate, last 0 of 10 calls
0%
Downstream calls per request
0.00×
Requests failed / rejected
0 / 0 of 0
Retries (up to 3)
Recent requests
RequestOutcomeDownstream callsSleeps before retries
Closed. The breaker judges the last 10 downstream calls.
Clients1× instanceLoad balancer2× instancesCheckout service(circuit breaker)4× instancesInventory service2× instancesLast known stock2× instancesClients1× instanceLoad balancer2× instancesCheckout service(circuit breaker)4× instancesInventory service2× instancesLast known stock2× instancesClients1× instanceLoad balancer2× instancesCheckout service(circuit breaker)4× instancesInventory service2× instancesLast known stock2× instances
Figure 2

A checkout service with a circuit breaker on its calls to a slow inventory service, and a fallback.

Follow a checkout request: the load balancer passes it to the checkout service, which wraps its calls to the inventory service in a circuit breaker. While the inventory service answers normally, the breaker is closed and lets calls through. Once too many calls fail, it opens: calls then fail at once, and the checkout service answers from the fallback instead of holding a thread for each slow call. A bulkhead would go further and give the inventory calls a pool of threads of their own.

  • Clients: 1 × Users
  • Load balancer: 2 × Load balancer
  • Checkout service (circuit breaker): 4 × App server
  • Inventory service: 2 × App server
  • Last known stock: 2 × NoSQL store
  • Clients sends requests to Load balancer
  • Load balancer sends requests to Checkout service (circuit breaker)
  • Checkout service (circuit breaker) sends requests to Inventory service
  • Checkout service (circuit breaker) sends requests to Last known stock

Key takeaways

  • Threads tied up equal call rate × call duration, so a dependency that slows to 4 s can take every thread and stop unrelated requests.
  • A circuit breaker counts failures, rejects calls at once while open so the caller can fall back, and lets trial calls through when half-open.
  • Every layer that retries multiplies the load on a failing service; limit retries and spread them with full jitter.
  • Bulkheads give each dependency its own pool, so one failure fills only its own section.
Sources7