The design is drawn. Then the interviewer points at one box and asks: “What happens when this is down?” Candidates who list every pattern they know sound rehearsed. Candidates who walk one failure from the first error to the recovery sound like they have been on call.
This lesson is the second half of Failure modes and resilience patterns, which explains timeouts, retries, breakers, bulkheads and load shedding. Here the same tools are turned into answers.
- Failure modes and resilience patterns: the five tools and the retry arithmetic.
- Load balancers for where health checks act, and rate limiting for protecting a server from callers.
Five layers each retry three times. What load does the database see, and which single change helps most? It sees 243 times the original load. Retrying at one layer only is the biggest single fix, and a retry budget caps what is left.
- Answer “what if X is down?” in four steps: detect, contain, degrade, recover. Name the real timeout, pool and fallback.
- Liveness checks the process, readiness checks it can serve. A liveness probe that checks a dependency turns one outage into a restart storm.
- Hedge only idempotent, slow-tail reads, after about the p95 latency, so extra load stays small.
- Shed by importance and bound queue wait. Health checks and finishing work in progress come first.
- An error budget turns “be reliable” into a number. Test failure paths on purpose, because an unused fallback is probably broken.
Answer “what happens when X is down?” for any box in a design, set health checks that help instead of hurt, decide when to hedge and when to shed by priority, link resilience to an error budget, and state what a mid, senior and staff answer sounds like.
The Four-Step Answer#
An interviewer says: “The payment provider is down.” Name the first thing you would say. Is it a pattern, or a question?
The strong first move is a question: “Is it down, or slow, and for which operations?” Slow and down need different handling, as the core lesson showed. Then walk four steps.
- Detect. A timeout on every call, an error-rate alert, and a breaker state change that pages someone.
- Contain. A bulkhead so the sick dependency cannot take every worker, a breaker so callers stop waiting, and a retry budget so retries do not become the load.
- Degrade. The user gets a reduced but honest answer: cached, queued for later, or a clear message.
- Recover. Half-open probes, jittered backoff and gradual traffic, so the dependency is not flattened the moment it returns.
Dependency Walk-Throughs#
These rows are design judgement, not measurements. Each applies the four steps to a box an interviewer is likely to pick.
| Dependency down | Contain | Degrade |
|---|---|---|
| Cache | Cap database load with a short timeout and request coalescing | Serve from the database at lower rate, shed low-priority reads |
| Primary database | Fail writes fast, keep reads on replicas | Read-only mode with a banner, queue writes if they can wait |
| Auth service | Cache validated tokens for their lifetime | Existing sessions continue, new logins fail with a clear message |
| Payment provider | Own pool and breaker, no blind retry | Accept the order as pending, charge later with an idempotency key |
| Recommendations | Own pool, tight timeout | Show a static popular list |
| Message broker | Local buffer with a size cap | Accept work to a retry queue, tell the user it is delayed |
Two details separate a good answer from a list. First, name what you refuse to degrade: the payment row accepts an order but never claims money was taken. Second, say who is told. A silent fallback hides an outage from the people who could fix it.
A fallback that skips an authorisation check or serves a stale balance is a security or correctness bug, not resilience. Fail closed where the wrong answer costs more than no answer.
Health Checks That Do Not Cause Outages#
A load balancer or orchestrator needs to know which instances can take traffic. The Google SRE book separates two questions: process health (“is this binary responding at all?”) and service health (“can it respond to this class of request right now?”). The first suits the scheduler, the second the load balancer.
Kubernetes names three probes in its documentation:
| Probe | Question | On failure |
|---|---|---|
| Startup | Has the app finished starting? | Container killed and restarted |
| Liveness | Is it stuck, for example deadlocked? | Container restarted |
| Readiness | Can it take traffic now? | Removed from Service endpoints, not restarted |
The trap is wiring a dependency into liveness. If the database goes down and every pod’s liveness probe checks it, Kubernetes restarts every pod together, each starts cold, and the outage grows. The same page says liveness probes must truly indicate unrecoverable failure, and that incorrect ones can cause cascading failures. Put dependency checks in readiness, and even there prefer checking a flag set by your breaker over a fresh call to the database from every probe.
The SRE book warns that a freshly started process is often slower until its caches fill, so a restart storm hits an already stressed system.
Hedged Requests and Tail Latency#
Retries answer failure. Hedging answers slowness. The Tail at Scale paper (Dean and Barroso, CACM 2013) shows why tails matter: if a request must collect answers from 100 servers and each is slow one time in a hundred, 63% of requests see at least one slow answer.
A hedged request sends a second copy to another replica once the first has waited longer than about the 95th-percentile latency, then uses whichever answers first and cancels the other. The paper reports that this caps extra load near 5%. In its BigTable benchmark, reading 1,000 keys across 100 servers with a 10 ms hedge cut the 99.9th-percentile latency from 1,800 ms to 74 ms for 2% more requests.
Hedging has rules, which are design judgement built on the paper:
- Hedge only idempotent operations, usually reads.
- Hedge only when the dependency has spare capacity. Hedging an overloaded service is a retry storm with a better name.
- Count hedges inside the retry budget, so the two do not add up.
Shed by Importance, Bound the Wait#
The core lesson covered load shedding in general. Amazon’s overload article adds the detail interviewers probe:
- Health checks come first. If the server misses the load balancer’s pings, the balancer stops sending it traffic, so the server sits idle while overloaded. Keep a path for them.
- Finish work before starting more. The article prioritises
end()calls overstart()calls, and later pages of a paginated request over the first page. - Bound queue wait. It recommends an upper limit on how long a request sits in a queue, and discarding anything older.
- Watch the median. In the same article, a service shedding 60% of its traffic can show a flattering median latency, so measure goodput and the shed rate too.
Backpressure is the same idea pointed upstream: when a queue is full, push back on the sender, with a rejection or a slower accept rate, instead of buffering without limit. An unbounded queue only moves the failure to memory and adds delay no client will wait for. This definition is design judgement, not from a single source.
Retries Need Idempotency#
Retries and hedges send the same operation twice, so the operation must survive it. Amazon’s article notes that EC2’s RunInstances API accepts an explicit token so a repeat does not create a second instance. Azure’s guidance says a repeated increment or a duplicate queue message can corrupt state unless each step is idempotent.
For a payment, the pattern looks like this:
client: one key per intent
every attempt sends the key
server: store result under key
repeat key -> return stored resultThe server answers a repeat from the stored result. The message delivery lesson covers the dedupe table.
Resilience Has a Budget#
An SLO sets how much failure is acceptable. The Google SRE Workbook defines the error budget as 100% minus the SLO.
SLO 99.9% over 30 days
minutes = 30 x 24 x 60 = 43,200
budget = 0.1% x 43,200 = 43.2 minIf a recommendations outage returns errors on the whole product page, 43 minutes of it spends the entire month. If a fallback serves a static list, the same outage spends almost none. That is the business case for a fallback, and a number to quote when someone asks why the team spent a sprint on one.
Alert on how fast the budget burns, not just on outages. The Workbook describes an error budget policy: when the budget is gone, reliability work comes before features.
Test the Failure Paths#
A fallback that has never run is a guess. The SRE book says “the code path you never use is the code path that (often) doesn’t work”, and suggests running a small subset of servers near overload to exercise degradation. It also advises load testing components until they break, with both gradual and sudden load.
Azure’s retry guidance suggests a mock dependency that returns the errors you handle, and fault injection, such as Azure Chaos Studio, for latency and process kills. A simple game day plan:
- State a hypothesis: “If recommendations is slow, checkout stays within its error budget.”
- Limit the blast radius: one cell, a small slice of traffic, a named stop button.
- Inject the fault, watch the breaker, queue depth and error budget.
- Fix what surprised you, and rerun.
Try It: Walk One Failure#
A product page calls a search service for “related items”. Assume search’s p99 rises from 80 ms to 6 s during an incident. The page has a 2 s deadline and 300 requests a second, each calling search once. Answer in the four steps:
- What is the pool size needed in normal times, by Little’s law, and what happens at 6 s?
- What does the user see?
- What do you alert on?
Hint
Start with in flight = rate x time. Then ask what a 2 s deadline does to the 6 s calls.
One answer
Normal: 24 in flight (300 x 0.08)
Sick: 300/s x 2 s deadline = 600
Pool of 50 caps it; rest fail fast
Breaker opens: no related itemsThe deadline matters. The page never waits past 2 s, so each call holds a worker for at most 2 s, not 6.
A pool of about 50 (roughly twice normal, a judgement call) stops search taking every worker. The breaker opens once errors pass its threshold, and the page renders without the related-items row. The user sees a complete page minus one section. Alert on breaker state, timeout rate and fallback rate, and spend of the error budget. Add jittered half-open probes so recovery is gradual.
What Each Level Should Say#
This table is design judgement about interview expectations, not a published standard.
| Level | A strong answer |
|---|---|
| Mid | Names timeouts, retries with backoff and a breaker, and says why retries can hurt |
| Senior | Adds deadlines, a retry budget at one layer, a bulkhead per dependency, a fallback with its limits, and what to alert on |
| Staff | Adds blast radius and cells, SLO and error budget trade-offs, readiness vs liveness, safe recovery, game days, and what not to degrade |
Strong: asks whether the dependency is down or slow, quantifies with a pool or retry calculation, names what the user sees, and says how the fallback is tested. Weak: a list of pattern names, retries everywhere, a fallback that hides failures, or no way to know it is happening.
The 45-Minute Plan for a Resilience Round#
This pacing is a suggestion for a round that centres on failure handling.
| Minutes | Do |
|---|---|
| 0 to 5 | Draw the call graph, mark critical and optional dependencies |
| 5 to 15 | Timeouts, deadlines and where retries live, with the multiplication |
| 15 to 25 | Breaker, bulkhead and fallback for the two riskiest dependencies |
| 25 to 35 | Shedding, health checks, recovery |
| 35 to 45 | SLO, alerts, game day, and follow-ups |
| They ask | Answer in one line |
|---|---|
| The dependency is slow, not down? | Deadline, bulkhead, breaker on latency |
| Retries made it worse? | One layer, jitter, budget |
| It recovers and falls over again? | Half-open probes, gradual ramp, jitter |
| Pods restart in a loop? | Liveness checked a dependency; move it to readiness |
| Tail latency on a fan-out? | Hedge reads after p95, count in the budget |
| Overload with a full queue? | Bound the wait, shed by priority |
| Duplicate charges? | Idempotency key stored with the result |
| How do you know the fallback works? | Game day, plus a small live slice |
| Why not a breaker everywhere? | Modal behaviour; local resources and queues do not need one |
Check Yourself#
1.Why is a database check in the liveness probe dangerous?
When the database fails, every pod fails liveness together and is restarted, which adds cold starts to an outage. Liveness should show only that the process is stuck. Readiness can reflect dependencies.
2.When is hedging a bad idea?
When the operation is not idempotent, or when the dependency is already overloaded, because the extra copy adds load to a service that is struggling.
3.A 99.9% SLO over 30 days: how many minutes of full outage does it allow?
0.1% of 43,200 minutes, which is 43.2 minutes. Partial degradation spends less, which is why fallbacks protect the budget.
4.Why bound how long a request waits in a queue?
A request that waits past the client’s timeout is work nobody will read. Dropping it frees capacity for requests that still matter.
Pick any system you know. For each arrow in its diagram write the four words: detect, contain, degrade, recover. Any blank is a follow-up question waiting for you.
In One Picture#
Down? ask: down or slow, which op
Steps detect, contain, degrade,
recover
Probe liveness = process stuck
readiness = can serve
Hedge idempotent reads, after p95
Shed health checks and finishing
work first, bound the wait
Budget 99.9% / 30 days = 43.2 min
Test game day, then repeatName the failure, quantify it, say what the user sees, and say how you know it works.
- Back to the tools: failure modes and resilience patterns for timeouts, retries, breakers and bulkheads.
- Where checks and shedding run: load balancers and API gateways.
- Asynchronous failure handling: event-driven architecture covers retry queues and dead-letter queues.
