Retries, backoff and circuit breakers

At 8 p.m. on the first day of a sale, the payment provider's API slows down for five seconds. Then it is fine again. Your checkout stays down for forty minutes. The provider's status page is green the whole time, and every graph points at them. The thing keeping you down after the first minute is your own retry code.

Retries are the right answer to partial failure. Done naively, they are also the most reliable way to turn a short blip into a long outage. This chapter is about the difference.

Retries multiply load#

A retry is another request. When a dependency is healthy and one request fails by bad luck, one extra request costs nothing. When the dependency is failing because it is overloaded, every retry adds load to exactly the thing that cannot take more.

The numbers get large fast. Three retries means up to four attempts per request, so a dependency at 80% of its capacity can suddenly be asked for 320%. Stack the retries and it multiplies: if each of five services in a call chain retries three times, one user request can become 4 × 4 × 4 × 4 × 4 = 1,024 calls at the bottom.

So the first rule is: retry at one layer only, usually the one closest to the caller, and let every layer below fail fast.

Watch a storm#

Here is a dependency running at 80% of its capacity. It slows down for a few seconds. Pick a retry policy and watch what happens after the slowdown ends.

With instant retries the outage outlives its cause. There are two effects feeding each other. Retries multiply the load, and past capacity the dependency wastes its time: requests wait in queues until their callers time out, so it keeps finishing work that nobody is waiting for any more. Little gets done, so more requests fail, so more retries arrive.

This has a name: a metastable failure, from a 2021 paper by Nathan Bronson and colleagues. The system has a healthy stable state and a broken stable state, and a short trigger can push it from one into the other. Once there, it stays, because the thing keeping it there (the retries) is the system's own reaction. Restarting the dependency does not help. Load has to drop.

Backoff and jitter#

Exponential backoff waits longer after each failure: 100 ms, then 200, then 400, up to a cap. It gives a struggling dependency room to recover instead of hitting it again at once.

Jitter makes each wait random. Without it, a thousand clients that failed at the same moment all retry at the same moment, 100 ms later, then again 200 ms later: synchronised waves that look like a denial of service attack. Marc Brooker's 2015 post on the AWS architecture blog compared the options, and "full jitter" (wait a random time between zero and the backoff value) came out as a simple, strong choice. AWS SDKs back off with jitter by default.

// Full jitter: wait a random time between 0 and the exponential backoff, capped.
function backoffMs(attempt: number, baseMs = 100, capMs = 5_000): number {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt)
  return Math.random() * ceiling
}
 
async function withRetries<T>(call: () => Promise<T>, maxRetries = 3): Promise<T> {
  for (let attempt = 0; ; attempt += 1) {
    try {
      return await call()
    } catch (err) {
      if (attempt >= maxRetries || !isRetryable(err)) throw err
      await new Promise((resolve) => setTimeout(resolve, backoffMs(attempt)))
    }
  }
}

As the widget shows, backoff alone is not enough. It spreads retries out, and every failed request still comes back three times, so sustained overload still builds up. Backoff helps a blip. It does not cap the total.

Retry budgets#

The fix that caps the total is a retry budget: retries may add at most a fixed share on top of normal traffic. Google's SRE book describes two limits used together: at most three attempts per request, and retries at most 10% of all requests from a client. When the budget is spent, a failure is returned to the caller straight away instead of being retried.

In the widget, that one rule turns a permanent outage into one that ends the moment the dependency recovers. During the slowdown some users get errors. That is the trade: a few errors now, instead of everyone's errors for forty minutes.

Two more habits keep retries honest:

Retry only what can succeed. A timeout, a dropped connection, a 503 or a 429 might work next time. A 400, 401, 403, 404 or 422 will fail exactly the same way. And never retry a request that changes something unless it carries an idempotency key.

Pass the deadline down. If the user's request has 2 seconds left, every downstream call should know that, and nobody should start work for a caller that has already given up. gRPC propagates deadlines between services for exactly this reason.

Circuit breakers#

A circuit breaker sits in front of a dependency and watches its failures. Michael Nygard named the pattern in his 2007 book Release It!, after the electrical part.

A breaker does not make the dependency work. It makes failure fast and cheap for the caller, and quiet for the dependency. The design question that remains is the fallback: what does the user see while it is open? Recommendations can disappear. A price cannot be made up.

Bulkheads and shedding load#

Two more ideas finish the set.

Bulkheads separate resources per dependency, like the watertight compartments of a ship. Give the recommendations client its own small pool of 20 connections, and when recommendations hang, they can tie up 20 connections, not every thread in your checkout service.

Load shedding is the server's side of the same problem: when it is past capacity, it rejects some requests early and cheaply (a fast 503) so it can finish the rest properly. Shed the least important traffic first: background refreshes before searches, searches before checkouts. Rejecting 20% of requests at once beats timing out 100% of them slowly.

Interview replay#

Checkpoint#