# LLM System Design Interview Follow-Ups: API, Failure Modes and Deep Dives (2026)
> The production half of the LLM system design round: the API contract, data model, what breaks when each part dies, regions, abuse, a 45-minute plan, level expectations, and four deep dives on KV-cache math, semantic-cache invalidation, tenant fairness and a provider outage.
*Source: https://www.infowok.com/llm-system-design-interview-2026-follow-ups/ · Sukhveer Kaur · Published October 24, 2026*
---
The five layers are on the whiteboard and the interviewer says: fine, now the API. Then: what if the cache is wrong? What if one customer sends ten times the traffic? What if the model provider is down for twenty minutes at launch?
None of that changes the core design. All of it decides whether the design survives production, and interviewers often spend the second half of the round here. This lesson is the second half of the [LLM system design interview](/llm-system-design-interview-2026/), which builds the five-layer answer.
- [The core lesson](/llm-system-design-interview-2026/): the five layers, the running support chatbot and its capacity table (about 9 QPS and 27 requests in flight on launch day).
- [Rate limiting explained](/rate-limiting-explained/) and [Design a rate limiter](/design-a-rate-limiter/) for token buckets, and [caching strategies](/caching-strategies/) for invalidation.
Why does the cache check sit before the router, and which cache lever needs an invalidation plan? Before, because a hit should skip everything expensive. The semantic cache needs the plan, because the source articles change.
- State the contract and the data model first. A streaming API needs an idempotency key and a clear error for every failure, including "slow down" versus "stop".
- KV-cache arithmetic tells whether memory or something else limits the pool: tokens that fit equal free memory divided by bytes per token.
- Invalidate a semantic cache by version (tenant, model, prompt, source documents), not by guessing a time.
- Limit tenants by tokens, then share spare capacity fairly. Request counts hide the cost.
- A provider outage is a ladder, not a switch: retry, fail over, serve stale, retrieve only, queue. Exercise each rung before it is needed.
- Judgement calls here are labelled. Numbers are assumptions unless a source is linked.
Extend the five-layer answer into a full 45-minute round: write the API contract and data model, say what breaks when each part dies, work the KV-cache memory arithmetic, keep a semantic cache correct, share a pool fairly between tenants, and degrade gracefully when the provider is down.
## What to ask and assume before the contract
The core lesson asks about growth, budget and uptime. Two more questions matter before the API is drawn, and each changes the design.
- **Who calls it?** A browser chat widget needs streaming and a session. A partner calling from a backend needs an API key, quotas and a bill per tenant.
- **Is the model hosted or self-run?** A hosted API means limits and outages belong to the provider. Self-run means owning GPU memory, which is the KV-cache dive below.
State the assumptions aloud: one hosted provider today and a second one wired for failover, multiple tenants on shared infrastructure, answers streamed, and conversation history kept for 30 days (all assumptions, to be replaced by the interviewer's answers).
| | In scope | Out of scope |
|---|---|---|
| Functional | Ask a question, stream an answer with sources, keep history, give feedback | Model training, the retrieval index build, billing UI |
| Non-functional | 3 s p95 to the first token, 99.9% uptime, tenant isolation, a per-tenant budget | Compliance certification, on-device models |
## What does the API contract look like?
A chat endpoint looks easy until it fails halfway through a stream. The contract below is a design judgement, not a standard.
```text
POST /v1/chats/{id}/messages
Authorization: Bearer
Idempotency-Key:
body: {"text": "...",
"max_output_tokens": 500}
200 text/event-stream
event: token
data: {"t":"Hel"}
event: sources
data: [{"id":"a42"}]
event: done
data: {"usage":
{"in":2000,"out":310}}
GET /v1/chats/{id}/messages
?cursor=&limit=50
```
Four details separate a strong contract from a sketch:
- **Idempotency key.** When a call times out mid-stream, the client retries. The same key returns the stored answer instead of paying for a second generation.
- **Output cap.** `max_output_tokens` bounds cost and tail latency per request. The server enforces a ceiling above the client's value.
- **Usage in the final event.** The gateway needs real token counts to charge the tenant's budget, so the stream ends with them.
- **Cursor pagination** for history, because offset pages shift when new messages arrive.
| Status | Meaning | Client should |
|---|---|---|
| 400 / 422 | Bad body or input over the size limit | Fix the request, do not retry |
| 401 / 403 | Missing key, or key not allowed for this tenant | Stop |
| 429 | This tenant is over its rate or token budget | Wait for `Retry-After`, then retry |
| 503 | The service is overloaded or the provider is down | Retry with jitter, or show the fallback |
| 504 | The model did not answer in time | Retry once with the same idempotency key |
[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110) defines `503` as a temporary overload or maintenance condition and lets the server send a `Retry-After` header with it. A `429` for a monthly cap and a `429` for a per-minute limit need different messages, as the [rate limiter follow-ups](/design-a-rate-limiter-follow-ups/) explain.
## Data model: what is stored and where
Start from the access patterns, then pick the store. Every row of the table is a judgement.
| Data | Shape | Store, and why |
|---|---|---|
| Conversations and messages | Read by conversation id, newest first | A key-value or document store keyed by tenant and conversation: single-key reads, simple sharding |
| Usage ledger | Append-only rows: tenant, tokens in and out, model, time | An append log, aggregated per tenant per minute: the budget check reads aggregates, not rows |
| Cache entries | Embedding, answer, tenant, versions, source ids | A vector index with a tenant filter, plus a key-value store for the answer text |
| Document index | Chunks and embeddings for retrieval | The RAG layer's vector database, owned by Part 1's design |
| Eval samples | Sampled question, answer, feedback | A warehouse table: read in batches, never on the request path |
Sharding by tenant keeps one tenant's reads together and makes tenant deletion a single operation. The cost is a hot tenant, handled in the fairness dive.
## What breaks when each part dies?
Kill the parts of the final design one at a time. An interviewer asks this to see whether the candidate designs for failure or only for the happy path.
| Part | What happens | The answer already in the design |
|---|---|---|
| Gateway instance | In-flight streams drop | Clients retry with the same idempotency key to another instance |
| Cache store | Every request is a miss, so load and cost jump | Serve through; alert; the pool was sized for a 0% hit rate |
| Router or classifier | Cannot pick a tier | Default to the larger model for hard cases, the safe tier |
| Serving replica | Its requests fail | Load balancer drains it; clients retry; autoscaler adds a replacement |
| Vector database | No context for the prompt | Answer from the cached or fallback path, or refuse; never invent context |
| Provider | Errors and timeouts | The degradation ladder below |
| Usage ledger | Budgets cannot be read | Fail open for small spends against a local counter, closed past a hard ceiling (judgement) |
In CAP terms (the theorem that a system split by a network failure must choose between staying consistent and staying available), chat history leans consistent inside one region, because a user must read their own last message. The cache and usage counters lean available: a stale cache entry or a counter a few tokens off is cheaper than an outage. That is a judgement, said with its reason.
## Deep dive 1: KV-cache and batching math
Memory sets how many requests a pool can batch. The arithmetic is short enough to do aloud.
### Memory per token
For every token in flight, the model stores a key and a value in every layer. The [vLLM paper](https://arxiv.org/abs/2309.06180) gives the formula and a case: for the 13-billion-parameter OPT model, one token needs 800 KB, from 2 (key and value) × 5,120 (hidden size) × 40 layers × 2 bytes. It also reports that on a 40 GB A100 about 65% of memory holds the weights and close to 30% holds this cache.
Take Llama 3 8B, which has 32 layers and only 8 key-value heads of size 128, per its [paper](https://arxiv.org/abs/2407.21783) (grouped-query attention: many query heads share a few KV heads). Is its per-token cache bigger or smaller than OPT-13B's 800 KB, and by roughly how much?
### The same card, a newer model
Much smaller. [Llama 3's paper](https://arxiv.org/abs/2407.21783) lists 32 layers, 4,096 model dimension, 32 attention heads and 8 key-value heads for the 8B model, so the head size is 4,096 ÷ 32 = 128 (derived). Per token: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB, about a sixth of 800 KB.
Assume an 80 GiB card, an 8 GiB reserve and a 2,500-token request (2,000 in, a few hundred out). The budget is then:
```text
Weights: 8B x 2 B = ~16 GiB
Reserve (10%) = 8 GiB
KV cache: 80-16-8 = 56 GiB
Per token = 128 KiB
Tokens: 56 GiB/128 KiB = 458,752
Requests: 458,752/2,500 = 183
```
Compare that with the assumed 27 requests in flight at the launch-day peak (from the core lesson's capacity table). Memory is not the limit at this size. Three things move it:
- **Context length.** A 20,000-token conversation takes eight times the memory of a 2,500-token one, so a few long chats shrink the batch.
- **KV heads.** A design with 32 KV heads instead of 8 needs 4 times the memory per token and fits about 45 requests on the same card.
- **Fragmentation.** The same paper says earlier systems wasted 60% to 80% of this memory (as reported on the [vLLM blog](https://vllm.ai/blog/vllm)), which is what paging fixed.
### Batching and the trade-off
Batching decides how well that memory is used. [Orca](https://www.usenix.org/conference/osdi22/presentation/yu) introduced scheduling per model iteration, so a finished request leaves the batch and a waiting one joins at once. That is continuous batching. Speculative decoding, where a small model drafts tokens and the large one checks them, was reported to speed up T5-XXL by [2x to 3x with identical outputs](https://arxiv.org/abs/2211.17192); whether it pays off depends on how often drafts are accepted, which must be measured.
The honest trade-off: bigger batches raise tokens per second and cost per token falls, but each request waits longer for its next token. Set the batch from the p95 target, not from peak throughput.
## Deep dive 2: how a semantic cache stays correct
### Three ways it goes wrong
A semantic cache returns an old answer for a new question. Three things can make that answer wrong: the question only looked similar, the documents changed, or the model or prompt changed.
| Failure | Example | Fix |
|---|---|---|
| False hit | "Cancel my plan" matches "Cancel my order" | Threshold tuned on labelled pairs; a second check for high-stakes intents |
| Stale answer | The refund policy page was edited this morning | Entry stores source ids and versions; a version change is a miss |
| Version skew | A new system prompt changed the tone and the rules | Prompt and model version are part of the key |
| Cross-tenant leak | Tenant A's private answer is returned to tenant B | Tenant id is part of the key and the vector filter; never a shared index |
### Threshold and provider caches
The threshold is a measured number. Collect labelled question pairs (same intent or not), score them, and pick the cutoff at the false-hit rate the business accepts. Start strict, since a miss costs one model call and a false hit costs trust. Research on [semantic caching](https://arxiv.org/abs/2604.20021) frames the cost-versus-reuse choice over a continuous embedding space, which is why the cutoff is tuned on data, not copied.
Provider prompt caches have their own limits. OpenAI's docs say cached prompt prefixes stay for about 30 minutes by default on GPT-5.6 and later (or 12 hours at a higher write cost); Anthropic's default is 5 minutes, and its caches are isolated per workspace. Both are prefix caches, so neither replaces a semantic cache; they cut the price of a miss.
If cache entries do not record which documents they came from, a policy fix cannot be rolled out safely: the only remedy is to clear everything, which spikes load and cost. Store source ids and versions from day one.
## Deep dive 3: multi-tenant limits and fairness
### Two layers, two jobs
One shared pool, many tenants. A rate limit alone does not make this fair, because a request is not a fixed unit: the [core rate limiter lesson](/design-a-rate-limiter/) counts requests, and an LLM pool must count tokens.
Two layers do different jobs:
- **Admission (per tenant, at the gateway).** A token bucket charged in tokens: an estimate on arrival, corrected when usage is known. It stops a tenant exceeding its plan and protects the budget.
- **Scheduling (inside the pool).** Decides who runs next when demand exceeds capacity. A limit cannot do this, since every tenant may be under its limit while the pool is still full.
### Fair sharing, worked
Max-min fairness is the simple rule for the second layer: no tenant gets more than it asked for, and spare capacity is split equally among the tenants that want more. A worked example, with all numbers assumed:
```text
Pool: 6,000 output tokens/s
Demand: A 5,000 B 1,000 C 500
Equal share: 6,000/3 = 2,000
C needs 500 -> gets 500
B needs 1,000 -> gets 1,000
A gets the rest: 6,000-1,500
= 4,500 (not 5,000)
```
The [Virtual Token Counter paper](https://arxiv.org/abs/2401.00588) turns this into a scheduler for continuous batching. It counts the input and output tokens each client has used and serves the client with the least service first. The authors prove that the service difference between two backlogged clients is bounded by 2x, and note that request-rate limits can leave capacity idle.
Add three practical rules (judgement): reserve a small share for interactive chat so batch jobs cannot crowd it out, give each tenant a queue cap so one tenant's backlog cannot grow without bound, and alert on the share of 429s per tenant to catch a limit set too tight.
## Deep dive 4: the provider is down
### The ladder
A hosted model is a dependency with its own failures. The design answer is a ladder, tried in order, where each step gives up some quality to stay available.
### Keeping the ladder safe
Two rules keep the ladder from backfiring. The [Google SRE book](https://sre.google/sre-book/addressing-cascading-failures/) says to always use randomized exponential backoff and to limit retries per request. It also warns against retrying at several layers: its example is a database receiving 64 attempts (4 to the power 3) from one user action. Retry once, at one layer.
The same chapter warns that degraded modes are rarely exercised, so the code path you never use often does not work. Run a game day: switch the provider off in staging and walk down every rung.
Availability arithmetic needs a caveat. Assume the provider offers 99.9%, and that two providers fail independently:
```text
One provider: 0.1% of 43,200 min
= ~43 min a month
Two, independent:
1 - 0.001 x 0.001
= 99.9999%
```
Independence is the weak assumption: a shared cloud region, a shared rate limit or a shared prompt format can make both fail together. Quote the arithmetic, then say why the real figure is lower.
Failover is not free. The second model needs its own prompt tuning and an eval run, its price differs, and its limits may be lower. That last point is the exercise below.
## Multi-region: one pool or several?
One region is the simplest answer and the right first one. A second region is for two reasons: latency for distant users, and surviving a region loss. Both have costs.
- **Serving pools are per region.** GPU capacity is the scarce part, so each region needs its own pool, sized for its share plus an agreed failover load.
- **History needs a home region per tenant.** The stance from earlier holds: reading your own last message matters more than cross-region speed, so a conversation is written in its tenant's home region and replicated for recovery.
- **Caches are regional.** A cache that converges slowly is fine, since a stale entry is checked against document versions anyway.
- **Data residency can force the layout.** If a tenant's data must stay in one region, the tenant's pool, cache and logs stay there too, and failover stops at the border.
The trade-off to state: active-active regions double the idle capacity bill. Active-passive is cheaper and recovers more slowly. Recommend active-passive for chat with a tested failover, and move to active-active only when latency or a contract requires it (judgement).
## Security and abuse
An LLM service is attacked through its text, its cost and its tenants. The [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/) gives the vocabulary an interviewer expects.
| Threat | Why it matters here | Control |
|---|---|---|
| Prompt injection (LLM01) | A retrieved help article contains hidden instructions | Treat retrieved text as data; strip tool access from the answer path; filter output |
| Sensitive information disclosure (LLM02) | The model repeats another customer's data or personal data from logs | Tenant-scoped retrieval and cache; redact before logging |
| System prompt leakage (LLM07) | The system prompt holds internal rules | Keep secrets out of prompts; assume the prompt can be read |
| Unbounded consumption (LLM10) | One caller drives a huge bill or denial of service | Token budgets, output caps, input size limits, queue caps, timeouts |
| Vector and embedding weaknesses (LLM08) | Poisoned or cross-tenant documents in the index | Per-tenant indexes or filters; review what is ingested |
OWASP's own guidance on unbounded consumption lists rate limiting and quotas, input limits, timeouts, monitoring and graceful degradation, the same controls this design already has. Privacy: conversation text is personal data in many places, so keep it for the stated 30 days, make tenant deletion one operation, and keep raw text out of metrics.
## Observability, cost and rollout
Watch signals that map to the levers from the core lesson:
- **Latency:** TTFT and p95 total time, per model tier.
- **Capacity:** queue depth, KV-cache usage, and 429 rate per tenant.
- **Cost:** tokens per question (in and out), cache hit rate, and spend per tenant against budget.
- **Quality:** golden-set score, feedback rate and the share of answers served from cache or fallback.
Roll out any change (model, prompt, threshold) the same way: shadow it on a sample, compare against the golden set, then ramp in steps (assume 5%, 25%, 100%) with a rollback that is one switch. A cache or router change is a model change for this purpose, because it alters answers.
## The 45-minute plan
The core lesson fits the first 40 minutes of a round. Many rounds run 45, and the extra time goes to follow-ups. Here is one way to split it (a pacing suggestion, not a rule):
| Minutes | Do |
|---|---|
| 0 to 5 | Questions, assumptions, requirements, out of scope |
| 5 to 10 | Capacity table, API contract, data model |
| 10 to 20 | Request path and serving: cache, router, batching, scaling |
| 20 to 30 | Cost levers with numbers, then failure modes per part |
| 30 to 40 | The deep dive the interviewer picks (KV cache, semantic cache, fairness or outage) |
| 40 to 45 | Regions, security, rollout, recap and open questions |
If time is short, drop the regions before the failure table. Failure modes are what interviewers remember.
## What each level is expected to show
| Level | Expected | Strong signal |
|---|---|---|
| Mid | A working design in the five-layer order; names caching, batching and a fallback | Does the capacity math and the cost lever with one number |
| Senior | Contract, data model and failure per part; explains the trade-off of each lever | Picks a deep dive, states the assumption, and says what would change the answer |
| Staff | Frames cost and risk across teams: tenant budgets, vendor concentration, rollout policy | Proposes what to measure first, sets the order of work, and names what not to build yet |
## Practice: plan a provider outage
Using the launch-day peak of **9 QPS**, assume a **20% cache hit rate**, a second provider allowed **4 QPS**, and a retrieval-only fallback for the rest. Write the plan:
1. How many requests per second reach a model?
2. How many are served by the second provider, and how many fall back?
3. What do you tell users, and what do you watch?
Remove cache hits first. Then fill the second provider up to its limit.
```text
Peak: 9 QPS
Cache hits: 20% x 9 = 1.8
To a model: 9 - 1.8 = 7.2
Provider B: 4.0
Fallback: 7.2 - 4.0 = 3.2 QPS
3.2 / 9 = 36% of users
```
About 36% of requests get a retrieval-only answer (article links, no generated text), marked as such. Retry the failed provider once at one layer, with jitter. Watch the fallback share, the second provider's 429 rate, queue depth and TTFT. Ask the second provider for a higher limit before launch, since 36% is the number that shrinks if it says yes. All figures are assumptions to replace with real limits.
Pick one deep dive and explain it aloud in 90 seconds with only the arithmetic and the failure case, no notes. Do the KV-cache one first: it has the most numbers to forget.
## The follow-up bank
Expect follow-ups. Answer each with a trade-off, not a tool name:
| They ask | Answer in one line |
|---|---|
| Client retries mid-stream? | Idempotency key returns the stored answer |
| How many requests fit on a GPU? | Free memory / KV bytes per token / tokens per request |
| Why is the cache wrong? | Key by tenant, versions and source ids; threshold from labelled pairs |
| One tenant sends 10x? | Token bucket at the gateway, fair scheduling in the pool |
| Provider down? | Retry once, fail over, stale, retrieval only, queue |
| Second region? | Per-region pools, home region for history, active-passive first |
| Prompt injection? | Retrieved text is data; no tools on the answer path |
| Cache store dies? | Serve through; the pool is sized for no hits |
| Budget doubles? | Measure tokens per question first, then the cheapest lever |
| How do you roll out a model change? | Shadow, golden set, 5% then 25% then 100% |
## Check yourself
56 GiB is left for the cache. 56 GiB ÷ 128 KiB = 458,752 tokens, and 458,752 ÷ 2,500 is about 183 requests.
Each entry stores the ids and versions of the documents it was built from. The edit bumps the document's version, so every entry built from it fails the version check and becomes a miss.
The small tenants get what they asked for (1,000 and 500), and the large one gets the rest: 6,000 - 1,500 = 4,500, not its full 5,000.
Model traffic is 7.2 QPS, so the second provider could take all of it and the retrieval-only fallback would not be needed. The plan then adds a watch on that provider's own limit and cost.
## In one picture
```text
Contract stream, idempotency key
Data by tenant, ledger apart
Failure a fallback per part
KV cache free mem / bytes per tok
Cache keys carry versions
Tenants tokens in, fair out
Outage retry, switch, stale
```
## Conclusion
The core design is five boxes in a row. The production answer is the contract that survives a retry, memory arithmetic that says what limits the pool, a cache that knows what it was built from, a fair share for every tenant, and a ladder for when the provider disappears. Say the assumption and the trade-off each time, and the follow-ups stop being surprises.
**Which deep dive would catch you out in a live round?** Tell us in the comments.
- **Back to the core answer:** the [five-layer LLM system design lesson](/llm-system-design-interview-2026/).
- **Where the limits live:** [Design a rate limiter](/design-a-rate-limiter/) and its [follow-ups](/design-a-rate-limiter-follow-ups/).
- **Next in the course:** [AI gateway design](/ai-gateway-design/), where routing, budgets and failover become one component.