LLM System Design Interview Follow-Ups: API, Failure Modes and Deep Dives (2026)

The five layers are on the whiteboard and the interviewer says: fine, now the API. Then: what if the cache is wrong? What if one customer sends ten times the traffic? What if the model provider is down for twenty minutes at launch?

None of that changes the core design. All of it decides whether the design survives production, and interviewers often spend the second half of the round here. This lesson is the second half of the LLM system design interview, which builds the five-layer answer.

What to ask and assume before the contract#

The core lesson asks about growth, budget and uptime. Two more questions matter before the API is drawn, and each changes the design.

• Who calls it? A browser chat widget needs streaming and a session. A partner calling from a backend needs an API key, quotas and a bill per tenant.
• Is the model hosted or self-run? A hosted API means limits and outages belong to the provider. Self-run means owning GPU memory, which is the KV-cache dive below.

State the assumptions aloud: one hosted provider today and a second one wired for failoverfailover: switching to the standby provider or server when the main one fails, multiple tenants on shared infrastructure, answers streamed, and conversation history kept for 30 days (all assumptions, to be replaced by the interviewer’s answers).

What does the API contract look like?#

A chat endpoint looks easy until it fails halfway through a stream. The contract below is a design judgement, not a standard.

POST /v1/chats/{id}/messages
  Authorization: Bearer <key>
  Idempotency-Key: <uuid>
  body: {"text": "...",
    "max_output_tokens": 500}
200 text/event-stream
  event: token
    data: {"t":"Hel"}
  event: sources
    data: [{"id":"a42"}]
  event: done
    data: {"usage":
      {"in":2000,"out":310}}
GET /v1/chats/{id}/messages
  ?cursor=<opaque>&limit=50

Four details separate a strong contract from a sketch:

• Idempotency key. When a call times out mid-stream, the client retries. The same key returns the stored answer instead of paying for a second generation.
• Output cap. max_output_tokens bounds cost and tail latency per request. The server enforces a ceiling above the client’s value.
• Usage in the final event. The gateway needs real token counts to charge the tenant’s budget, so the stream ends with them.
• Cursor pagination for history, because offset pages shift when new messages arrive.

RFC 9110 defines 503 as a temporary overload or maintenance condition and lets the server send a Retry-After header with it. A 429 for a monthly cap and a 429 for a per-minute limit need different messages, as the rate limiter follow-ups explain.

Data model: what is stored and where#

Start from the access patterns, then pick the store. Every row of the table is a judgement.

Sharding by tenant keeps one tenant’s reads together and makes tenant deletion a single operation. The cost is a hot tenant, handled in the fairness dive.

What breaks when each part dies?#

Kill the parts of the final design one at a time. An interviewer asks this to see whether the candidate designs for failure or only for the happy path.

In CAP terms (the theorem that a system split by a network failure must choose between staying consistent and staying available), chat history leans consistent inside one region, because a user must read their own last message. The cache and usage counters lean available: a stale cache entry or a counter a few tokens off is cheaper than an outage. That is a judgement, said with its reason.

Deep dive 1: KV-cache and batching math#

Memory sets how many requests a pool can batch. The arithmetic is short enough to do aloud.

Memory per token#

For every token in flight, the model stores a key and a value in every layer. The vLLM paper gives the formula and a case: for the 13-billion-parameter OPT model, one token needs 800 KB, from 2 (key and value) × 5,120 (hidden size) × 40 layers × 2 bytes. It also reports that on a 40 GB A100 about 65% of memory holds the weights and close to 30% holds this cache.

The same card, a newer model#

Much smaller. Llama 3’s paper lists 32 layers, 4,096 model dimension, 32 attention heads and 8 key-value heads for the 8B model, so the head size is 4,096 ÷ 32 = 128 (derived). Per token: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, or 128 KiB, about a sixth of 800 KB.

Assume an 80 GiB card, an 8 GiB reserve and a 2,500-token request (2,000 in, a few hundred out). The budget is then:

Weights: 8B x 2 B   = ~16 GiB
Reserve (10%)       =   8 GiB
KV cache: 80-16-8   =  56 GiB
Per token           = 128 KiB
Tokens: 56 GiB/128 KiB = 458,752
Requests: 458,752/2,500 = 183

Compare that with the assumed 27 requests in flight at the launch-day peak (from the core lesson’s capacity table). Memory is not the limit at this size. Three things move it:

• Context length. A 20,000-token conversation takes eight times the memory of a 2,500-token one, so a few long chats shrink the batch.
• KV heads. A design with 32 KV heads instead of 8 needs 4 times the memory per token and fits about 45 requests on the same card.
• Fragmentation. The same paper says earlier systems wasted 60% to 80% of this memory (as reported on the vLLM blog), which is what paging fixed.

Batching and the trade-off#

Batching decides how well that memory is used. Orca introduced scheduling per model iteration, so a finished request leaves the batch and a waiting one joins at once. That is continuous batching. Speculative decoding, where a small model drafts tokens and the large one checks them, was reported to speed up T5-XXL by 2x to 3x with identical outputs; whether it pays off depends on how often drafts are accepted, which must be measured.

The honest trade-off: bigger batches raise tokens per second and cost per token falls, but each request waits longer for its next token. Set the batch from the p95 target, not from peak throughput.

Deep dive 2: how a semantic cache stays correct#

Three ways it goes wrong#

A semantic cache returns an old answer for a new question. Three things can make that answer wrong: the question only looked similar, the documents changed, or the model or prompt changed.

Threshold and provider caches#

The threshold is a measured number. Collect labelled question pairs (same intent or not), score them, and pick the cutoff at the false-hit rate the business accepts. Start strict, since a miss costs one model call and a false hit costs trust. Research on semantic caching frames the cost-versus-reuse choice over a continuous embedding space, which is why the cutoff is tuned on data, not copied.

Provider prompt caches have their own limits. OpenAI’s docs say cached prompt prefixes stay for about 30 minutes by default on GPT-5.6 and later (or 12 hours at a higher write cost); Anthropic’s default is 5 minutes, and its caches are isolated per workspace. Both are prefix caches, so neither replaces a semantic cache; they cut the price of a miss.

Deep dive 3: multi-tenant limits and fairness#

Two layers, two jobs#

One shared pool, many tenants. A rate limit alone does not make this fair, because a request is not a fixed unit: the core rate limiter lesson counts requests, and an LLM pool must count tokens.

Two layers do different jobs:

• Admission (per tenant, at the gateway). A token bucket charged in tokens: an estimate on arrival, corrected when usage is known. It stops a tenant exceeding its plan and protects the budget.
• Scheduling (inside the pool). Decides who runs next when demand exceeds capacity. A limit cannot do this, since every tenant may be under its limit while the pool is still full.

Fair sharing, worked#

Max-min fairness is the simple rule for the second layer: no tenant gets more than it asked for, and spare capacity is split equally among the tenants that want more. A worked example, with all numbers assumed:

Pool: 6,000 output tokens/s
Demand: A 5,000  B 1,000  C 500
Equal share: 6,000/3 = 2,000
C needs 500  -> gets 500
B needs 1,000 -> gets 1,000
A gets the rest: 6,000-1,500
   = 4,500 (not 5,000)

The Virtual Token Counter paper turns this into a scheduler for continuous batching. It counts the input and output tokens each client has used and serves the client with the least service first. The authors prove that the service difference between two backlogged clients is bounded by 2x, and note that request-rate limits can leave capacity idle.

Add three practical rules (judgement): reserve a small share for interactive chat so batch jobs cannot crowd it out, give each tenant a queue cap so one tenant’s backlog cannot grow without bound, and alert on the share of 429s per tenant to catch a limit set too tight.

Deep dive 4: the provider is down#

The ladder#

A hosted model is a dependency with its own failures. The design answer is a ladder, tried in order, where each step gives up some quality to stay available.

Keeping the ladder safe#

Two rules keep the ladder from backfiring. The Google SRE book says to always use randomized exponential backoff and to limit retries per request. It also warns against retrying at several layers: its example is a database receiving 64 attempts (4 to the power 3) from one user action. Retry once, at one layer.

The same chapter warns that degraded modes are rarely exercised, so the code path you never use often does not work. Run a game day: switch the provider off in staging and walk down every rung.

Availability arithmetic needs a caveat. Assume the provider offers 99.9%, and that two providers fail independently:

One provider: 0.1% of 43,200 min
            = ~43 min a month
Two, independent:
  1 - 0.001 x 0.001
  = 99.9999%

Independence is the weak assumption: a shared cloud region, a shared rate limit or a shared prompt format can make both fail together. Quote the arithmetic, then say why the real figure is lower.

Failover is not free. The second model needs its own prompt tuning and an eval run, its price differs, and its limits may be lower. That last point is the exercise below.

Multi-region: one pool or several?#

One region is the simplest answer and the right first one. A second region is for two reasons: latency for distant users, and surviving a region loss. Both have costs.

• Serving pools are per region. GPU capacity is the scarce part, so each region needs its own pool, sized for its share plus an agreed failover load.
• History needs a home region per tenant. The stance from earlier holds: reading your own last message matters more than cross-region speed, so a conversation is written in its tenant’s home region and replicated for recovery.
• Caches are regional. A cache that converges slowly is fine, since a stale entry is checked against document versions anyway.
• Data residency can force the layout. If a tenant’s data must stay in one region, the tenant’s pool, cache and logs stay there too, and failover stops at the border.

The trade-off to state: active-active regions double the idle capacity bill. Active-passive is cheaper and recovers more slowly. Recommend active-passive for chat with a tested failover, and move to active-active only when latency or a contract requires it (judgement).

Security and abuse#

An LLM service is attacked through its text, its cost and its tenants. The OWASP Top 10 for LLM Applications 2025 gives the vocabulary an interviewer expects.

OWASP’s own guidance on unbounded consumption lists rate limiting and quotas, input limits, timeouts, monitoring and graceful degradation, the same controls this design already has. Privacy: conversation text is personal data in many places, so keep it for the stated 30 days, make tenant deletion one operation, and keep raw text out of metrics.

Observability, cost and rollout#

Watch signals that map to the levers from the core lesson:

• Latency: TTFT and p95 total time, per model tier.
• Capacity: queue depth, KV-cache usage, and 429 rate per tenant.
• Cost: tokens per question (in and out), cache hit rate, and spend per tenant against budget.
• Quality: golden-set score, feedback rate and the share of answers served from cache or fallback.

Roll out any change (model, prompt, threshold) the same way: shadow it on a sample, compare against the golden set, then ramp in steps (assume 5%, 25%, 100%) with a rollback that is one switch. A cache or router change is a model change for this purpose, because it alters answers.

The 45-minute plan#

The core lesson fits the first 40 minutes of a round. Many rounds run 45, and the extra time goes to follow-ups. Here is one way to split it (a pacing suggestion, not a rule):

If time is short, drop the regions before the failure table. Failure modes are what interviewers remember.

What each level is expected to show#

Practice: plan a provider outage#

Peak: 9 QPS
Cache hits: 20% x 9 = 1.8
To a model: 9 - 1.8 = 7.2
Provider B: 4.0
Fallback: 7.2 - 4.0 = 3.2 QPS
         3.2 / 9 = 36% of users

The follow-up bank#

Check yourself#

In one picture#

Contract  stream, idempotency key
Data      by tenant, ledger apart
Failure   a fallback per part
KV cache  free mem / bytes per tok
Cache     keys carry versions
Tenants   tokens in, fair out
Outage    retry, switch, stale

Conclusion#

The core design is five boxes in a row. The production answer is the contract that survives a retry, memory arithmetic that says what limits the pool, a cache that knows what it was built from, a fair share for every tenant, and a ladder for when the provider disappears. Say the assumption and the trade-off each time, and the follow-ups stop being surprises.

Which deep dive would catch you out in a live round? Tell us in the comments.