Failure Modes and Resilience Patterns: Timeouts, Retries, Breakers and Bulkheads (2026)

At 2 a.m. nothing is down. A single recommendations service has become slow: it answers, but in eight seconds instead of eighty milliseconds. Twenty minutes later the checkout page is failing, the login service is timing out, and the dashboard says every service is up.

Nothing crashed. The services that call the slow one are holding a worker for every waiting request, and workers are finite. This lesson is about how that spreads, and the five tools that stop it. One idea runs through all of them: fail fast, fail small. The follow-up lesson, failure modes: follow-ups, covers the interview half: “what happens when X is down?”, health checks, hedged requests, error budgets and chaos testing.

How One Slow Service Takes Down Everything#

A request to service A calls B, and B calls C. Each service has a limited number of workers: threads, connections, or in-flight slots. When C slows down, B’s workers wait on C. When all of B’s workers are waiting, B stops answering, so A’s workers wait on B. A and B are healthy code. They have simply run out of workers.

Microsoft’s Circuit Breaker guidance describes exactly this: blocked requests hold critical resources such as memory, threads and database connections, and exhausting them can fail unrelated parts of the system. Google’s SRE book defines a cascading failure as one that grows over time through positive feedback. Overloaded servers miss deadlines, the work they did is wasted, and clients retry, which adds more load.

The slow one. A dead dependency makes callers fail quickly and free their workers. A slow one holds the workers. That is why the first tool is not about errors at all.

Timeouts and Deadlines#

Amazon’s Builders’ Library states the rule plainly: set a timeout on any remote call, covering both the connection and the request. Microsoft’s retry guidance adds the order: set timeouts on every outbound call before you add retry logic, because a retry strategy is only as good as the timeout on each attempt.

The hard part is the number:

• Too high: workers are still held while waiting.
• Too low: healthy calls fail and get retried, which adds load.

Amazon’s method is to pick an acceptable rate of false timeouts, an assumed 0.1% in the article’s example, and read the matching latency percentile of the downstream service, p99.9p99.9: the latency that 99.9% of requests beat, so only 1 in 1,000 is slower in that example. It does not work for calls over the internet, where network latency has to be added, or when p99.9 is close to the median, where some padding helps.

Deadlines travel down the chain#

A timeout covers one call. A deadline is the time the original caller will wait in total, and each hop should know how much is left. Google’s SRE book says servers should propagate deadlines instead of inventing new ones, and check the time left before each stage of work. Amazon does the same, passing the remaining time at each hop so the last service knows how long its answer is useful.

The reason is wasted work. A client gives up after an assumed 10 seconds, the server reaches its request at second 11 and does the work anyway, and nobody reads the answer. The client has already retried, so under overload the waste feeds the loop.

Retries: Selfish by Design#

Retries multiply across layers#

Amazon’s example makes the danger concrete:

• A five-deep stack of services, ending in a database.
• Three tries at each layer, and each layer retries on its own.
• The database sees 3 × 3 × 3 × 3 × 3 = 243 times the original load.

Microsoft’s guidance gives a smaller case with the same shape: two layers with three retries each make nine attempts on the target.

A toy test shows the effect at a smaller scale. Assume 1,000 requests a second, a server that can serve 1,200 a second, and a five-second period when it fails everything.

The toy server only drops excess work. It does not slow down or collapse, so it understates the damage of a real overload. It is enough to show where the extra load comes from, and what removes it.

Backoff, jitter and a budget#

Four rules keep retries from becoming the outage.

1. Wait between tries, and wait longer each time. Exponential backoff multiplies the delay after each failure, up to a cap. gRPC’s connection backoff, which governs how a client reconnects to a server, uses a 1 second start, a 1.6 multiplier, a 120 second cap and 20% jitter.
2. Add jitter, a random amount of delay. Without it, clients that failed together retry together. Amazon explains that backing off alone often does not help under overload because of correlation: calls that failed at the same moment come back at the same moment.
3. Stop after a few tries. Microsoft’s Retry Storm antipattern is the endless while(true) retry: requests are only valid for a limited time, and retrying forever stops the service recovering.
4. Keep a retry budget. Both Microsoft and Google suggest a limit across all requests in a process, with 60 retries a minute as the example. When the budget is spent, fail instead of retrying. Amazon limits retries locally with a token bucket, and says the AWS SDK has done so since 2016.

A second toy test shows what jitter does. Assume 1,000 clients all fail at the same instant and retry three times, with a 1 second base that doubles each time. Without jitter, the busiest 100 ms slot gets ≈ 1,000 retries, all of them. With full jitter, it gets ≈ 154, and the retries spread across ≈ 66 slots instead of 3.

What to retry, and where#

• Only retry what is safe. Amazon’s view is that operations with side effects are not safe to retry unless they are idempotentidempotent: repeating the operation has the same effect as doing it once. Some APIs accept an idempotency token for exactly this reason. The message delivery lesson shows how a server dedupes repeats.
• Only retry what can succeed. A 400 Bad Request will not improve on a second try. Retry timeouts, connection failures and 503 responses, and honour a Retry-After header when the server sends one.
• Retry at one layer. Amazon’s advice is to retry at a single point in the stack for most operations, and to switch off built-in retries in the layers below.

Circuit Breakers#

A circuit breaker wraps a call and counts failures. Microsoft’s description has three states:

• Closed: calls pass through, and failures are counted over a time window.
• Open: after too many failures the breaker rejects calls at once, without calling the dependency.
• Half-open: after a wait, a few trial calls go through. If they succeed the breaker closes, and if one fails it opens again.

Breakers are not free. Amazon’s engineers point out that circuit breakers introduce modal behaviour into a system that can be difficult to test, and can add significant time to recovery. They prefer limiting retries locally with a token bucket. Microsoft lists more things to decide:

• Scope. One breaker for a dependency with several independent parts, each on its own server, can hide a healthy part behind a failing one.
• Fallback. The caller needs something to do when the breaker is open: a cached answer, a default, or a clear error.
• Not for everything. A breaker adds overhead around local in-memory resources, and message-driven systems often rely on retry queues and dead-letter queues instead.

Bulkheads#

A ship’s hull is divided into watertight sections so one breach does not sink it. The Bulkhead pattern does the same for software: give each dependency its own pool of workers or connections, so a failure in one fills only its own pool.

Microsoft’s examples include a connection pool per downstream service, a service instance per client, and separate queues per workload. The costs are the usual ones for isolation: unused capacity in one pool cannot help another, and every partition needs sizing and monitoring. Microsoft also notes that AI and inference workloads often need strict bulkheads, because model deployments have quotas and concurrency limits.

Sizing a pool: Little’s law#

How big should the recommendations pool be? Use Little’s lawLittle's law: the average number of requests in flight equals the arrival rate times the average time each takes. All numbers here are assumptions. Assume the page calls recommendations 100 times a second and a healthy call takes 80 ms.

in flight = rate x time
          = 100/s x 0.08 s = 8
pool of 20 = 2.5x headroom
sick: 100/s x 8 s = 800 wanted
cap at 20, rest fail fast

In a shared pool of 20, the sick dependency would want 800 workers and take all 20 from payments.

In its own pool it takes 20 of its own. Those 20 workers finish only 2.5 calls a second at 8 s each, so about 97 of every 100 calls get an immediate rejection or a fallback, and payments keeps its workers.

Load Shedding and Graceful Degradation#

The tools so far protect a caller from a dependency. Load shedding protects a server from its callers. Amazon’s article on overload separates throughput, the requests sent, from goodput, the requests handled correctly and quickly enough to be useful. When a server is past capacity, latency crosses the client’s timeout, goodput falls, and clients retry.

Shedding fixes this by rejecting the excess early. The requests the server accepts are answered quickly, so only the surplus traffic is lost. Practical forms:

• Limit work in flight. Google’s SRE book suggests returning 503 when more than a set number of requests are in flight.
• Keep queues short. The same book suggests queue lengths of ≈ 50% or less of the thread pool for steady traffic, so a saturated server rejects early. Amazon bounds how long a request may wait in a queue and drops stale ones.
• Drop doomed work. If the deadline has already passed, skip the request.
• Degrade on purpose. Return a cheaper answer, for example search over a cached subset, instead of failing. Google warns that a degraded mode you never exercise is likely to be broken, so run it regularly.

When Not to Use Each Tool#

Every tool has a cost, so name when you would skip it. These are judgement calls, backed where noted by the sources.

The last row is the one people forget. A cached recommendation list is a fine fallback. A cached account balance that lets someone overspend is not.

Putting It Together: One Protected Call#

For one outbound call, the layers sit in a sensible order.

Check the remaining deadline first, then take a slot in the dependency’s pool. If the breaker is open, skip the call. Otherwise make one attempt with a timeout. If it fails and the budget has tokens, retry after a jittered delay. When nothing works, return the fallback. Every failure feeds the breaker and spends budget.

Where AI Systems Hit This#

Try It: Count the Retries#

4 hops x 3 attempts each
= 3 x 3 x 3 x 3 = 81 attempts
200 calls/s x 81 = 16,200/s
Fix: retry at one layer only
Cap total with a retry budget
Breaker stops calls entirely

Check Yourself#

In One Picture#

Slow   holds workers, spreads up
Time   timeout every call,
  pass the deadline down
Retry  one layer, backoff +
  jitter, few tries, budget
Break  stop calling a sick
  dependency, probe to recover
Wall   a pool per dependency
Shed   reject excess early,
  degrade on purpose

Fail fast, fail small: limit the waiting, limit the retrying, and limit what any one failure can reach.