Stopping cascading failures and retry storms: three-state circuit breakers, exponential backoff with full jitter, and bulkheads.
What you will learn
Use Little's law to see how one slow dependency starves a caller's threads.
Follow a circuit breaker through its closed, open and half-open states.
Stop retry storms with retry limits and jitter, and contain failures with bulkheads.
Cascading failure dynamics and thread pool starvation
In a call tree, failure travels upward. Consider an orchestrator with 100 worker threads that calls several downstream services and holds a thread for each call in flight.
Suppose one non-critical dependency, Recommendations, slows from 20 ms to 4,000 ms. By Little's law (chapter 1), the threads busy with it equal the call rate times the call duration. So at 4 s per call, just 100/4=25 calls per second to Recommendations hold all 100 threads. At 20 ms, the same threads could serve 5,000 calls per second.
Once every thread is waiting on Recommendations, checkout and payment requests have no thread to run on either. Fowler puts it this way: with many callers on an unresponsive supplier "you can run out of critical resources leading to cascading failures across multiple systems."
The circuit breaker (three states)
Fowler: "You wrap a protected function call in a circuit breaker object, which monitors for failures."
Exact behavior varies by library. The numbers below are resilience4j's documented defaults:
Closed: calls pass through, and outcomes are recorded in a sliding window. In the default count-based window, that is the last 100 calls. No failure rate is computed until a minimum number of calls (100 by default) is recorded. When the failure rate reaches the threshold (50% by default), the breaker opens.
Open: calls are rejected at once with a CallNotPermittedException, so the caller can serve a fallback without waiting on the network. The breaker stays open for a configured wait (60 s by default).
Half-open: a limited number of trial calls (10 by default) are let through. If their failure rate is still at or above the threshold, the breaker opens again; otherwise it closes. Fowler's original description uses a single trial call.
Slow calls count too. A timeout lets a caller stop waiting for an answer it no longer expects, and a call that times out has failed. Release It! notes that a circuit breaker can count timeouts and trip once too many occur, and that breakers guard against slow responses as well as errors. So a dependency that is slow, not down, also opens the breaker, and while it is open the caller stops waiting on it at all.
The AWS Builders' Library adds a caution: circuit breakers "introduce modal behavior into systems that can be difficult to test".
Retry storms and backoff with jitter
When calls fail because the service is overloaded, "retries can make the overload worse instead of better" (AWS Builders' Library). If every call fails, each layer that retries multiplies the load:
Load=λbase×(1+MaxRetries)what its symbols mean
So 8,000 RPS with MaxRetries=3 becomes 32,000 RPS at the failing service. Retries at several layers multiply too. AWS's example is a five-layer stack where each layer tries three times: the database sees 35=243 times the load, "making it unlikely to ever recover". Retry only at one layer, and cap retries, for example with a token bucket that allows retries only while tokens remain.
Full jitter (Marc Brooker's AWS post, chapter 7) spreads the retries that remain:
tsleep=random(0,min(M,B⋅2i))what its symbols mean
Here Bwhat its symbols mean is the base delay, Mwhat its symbols mean the cap and iwhat its symbols mean the retry number. With the example values B=100 ms and M=10 s, the ceiling doubles 100, 200, 400 … ms until it reaches 10 s at the eighth retry. Clients that all failed at the same moment then retry at different moments instead of in lockstep.
Bulkheads
The pattern is named after the watertight compartments of a ship's hull: "if the hull of a ship is compromised, only the damaged section fills with water" (Azure Architecture Center). A consumer gives each dependency its own pool, such as a connection pool, thread pool or semaphore limit. Then "if a service begins to fail, it only affects the connection pool assigned for that service." In the example above, a separate pool for Recommendations would leave checkout's threads free.
The price is less efficient use of resources. Azure suggests combining bulkheads with retries, circuit breakers and throttling.
Circuit breaker and retries
Closed, open and half-open transitions over a count-based window, and how retries multiply downstream calls.
The 10-call window and 4 trial calls are sized for this demo; real defaults are larger (resilience4j: 100 calls and 10 trials).
Breaker state
Closed
Failure rate, last 0 of 10 calls
0%
Downstream calls per request
0.00×
Requests failed / rejected
0 / 0 of 0
Retries (up to 3)
Recent requests
Request
Outcome
Downstream calls
Sleeps before retries
Closed. The breaker judges the last 10 downstream calls.
A checkout service with a circuit breaker on its calls to a slow inventory service, and a fallback.
Follow a checkout request: the load balancer passes it to the checkout service, which wraps its calls to the inventory service in a circuit breaker. While the inventory service answers normally, the breaker is closed and lets calls through. Once too many calls fail, it opens: calls then fail at once, and the checkout service answers from the fallback instead of holding a thread for each slow call. A bulkhead would go further and give the inventory calls a pool of threads of their own.
Clients: 1 × Users
Load balancer: 2 × Load balancer
Checkout service (circuit breaker): 4 × App server
Inventory service: 2 × App server
Last known stock: 2 × NoSQL store
Clients sends requests to Load balancer
Load balancer sends requests to Checkout service (circuit breaker)
Checkout service (circuit breaker) sends requests to Inventory service
Checkout service (circuit breaker) sends requests to Last known stock
Key takeaways
Threads tied up equal call rate × call duration, so a dependency that slows to 4 s can take every thread and stop unrelated requests.
A circuit breaker counts failures, rejects calls at once while open so the caller can fall back, and lets trial calls through when half-open.
Every layer that retries multiplies the load on a failing service; limit retries and spread them with full jitter.
Bulkheads give each dependency its own pool, so one failure fills only its own section.