Retrying a failed request seems like an obviously safe, sensible default — if something didn’t work, try again. In a distributed system under real load, naive retries are one of the most common ways a small, contained problem turns into a full outage.
How a retry storm happens
Consider a service that’s starting to struggle — slow, but not fully down, perhaps because it’s approaching a saturation limit, discussed in The Four Golden Signals. Callers experiencing timeouts or errors from it retry automatically, as retry logic is designed to do. But those retries land on the exact same struggling service, adding to its load at precisely the moment it can least handle more — which makes it slower still, causing more timeouts, causing more retries, in a feedback loop that can turn a service that was merely struggling into one that’s fully overwhelmed and failing completely. This pattern is commonly called a retry storm, and it’s a well-documented failure mode in distributed systems specifically because the “obvious” fix — retry on failure — is exactly what drives it.
Backoff with jitter: spreading retries out over time
Exponential backoff addresses part of the problem by increasing the delay between successive retry attempts for a given request — retry after 1 second, then 2, then 4, and so on — rather than retrying immediately and repeatedly. This reduces sustained pressure on a struggling dependency, but on its own it has a subtler problem: if many clients all started retrying at the same moment (because they all experienced the same failure at the same moment), exponential backoff alone means they all retry again at the same subsequent moments too, in synchronized waves that can still overwhelm a recovering service each time they land together.
Jitter — adding randomness to the exact retry delay, rather than a fixed exponential schedule every client follows identically — spreads those retries out across time instead of letting them arrive in synchronized bursts. None of this is safe to do at all, though, unless the operation being retried is actually idempotent — retrying a non-idempotent operation risks applying its effect more than once, independent of how well-behaved the retry timing is. AWS’s Builders’ Library article on timeouts, retries, and backoff with jitter covers this pattern directly, and specifically recommends combining exponential backoff with randomized jitter rather than using either alone, precisely because unjittered exponential backoff still produces synchronized retry waves.
Circuit breakers: stop calling a failing dependency entirely, for a while
A circuit breaker takes a different, complementary approach: rather than retrying a failing call indefinitely with increasing delays, it tracks failures to a given dependency, and once failures cross a defined threshold, it “opens” — failing subsequent calls immediately, without even attempting to reach the struggling dependency, for a cooldown period. Martin Fowler’s description of the circuit breaker pattern frames the core benefit directly: this gives a failing service breathing room to actually recover, instead of every caller continuing to hammer it with fresh requests and retries while it’s already struggling. After the cooldown period, the circuit breaker allows a small number of test requests through to check whether the dependency has recovered, and only resumes normal traffic once it has.
Why both matter together
Backoff with jitter and circuit breakers solve overlapping but distinct parts of the same problem: backoff with jitter reduces how aggressively an individual caller retries and spreads that retry load out over time, while a circuit breaker can stop calling a dependency altogether once it’s clear that dependency is genuinely failing, rather than continuing to retry a service that backoff alone won’t help. Systems that implement one without the other are still exposed to a meaningful version of the retry storm problem — jittered backoff without a circuit breaker still eventually sends every client’s retries to a dependency that may never actually recover on its own without the load being cut off; a circuit breaker without sensible backoff on the underlying calls still allows a burst of simultaneous, unjittered retries in the moments before the circuit opens.
Where this fits in the broader reliability picture
This is a specific, concrete instance of the broader principle behind chaos engineering: a dependency failure that seems like it should be handled gracefully — “we retry on failure” — can actually make things meaningfully worse under real load and real timing, and the only way to know for certain is to test it deliberately, rather than assume the theoretical design is correct.
Key takeaway
Naive, unlimited, immediate retries can turn a struggling service into a fully failed one by piling additional load onto exactly the dependency that’s already overwhelmed. Exponential backoff with jitter spreads retries out over time to avoid synchronized retry waves, and circuit breakers stop calling a failing dependency entirely for a cooldown period — together, they’re what actually prevents a small, contained failure from cascading into a full outage.
This article explains general distributed systems reliability patterns; specific implementation details vary by language and platform. See our disclaimer.