Every LLM system design interview eventually asks some version of the same prompt: design a system that serves this model to a million users, within budget, without blowing the latency target. The round tests something a classic backend round does not.
Cost and latency move together here. Getting a design wrong isn’t a scaling problem you patch later. It’s a decision made up front.
Series so far: Part 1 covered the RAG interview round. Part 2, the agents round, is still being written. This post picks up with the system design round, using the same chatbot example at real scale.
This is Part 3 of the AI Engineer Interview Prep series. The hub post covers every round; this one goes deep on system design. Inside: a five-step answer order with time boxes, a capacity estimate you can do out loud, a glossary for the serving terms interviewers expect you to know, and eight questions answered twice, once weakly and once well.
- Use one fixed skeleton: scope → request path → serving and scaling → cost and latency levers → eval and monitoring.
- Time-box it. The whole answer fits in about 40 minutes, and serving plus cost and latency trade-offs get the biggest share.
- Do the capacity math out loud in the first five minutes, then keep the same scenario running to the end.
- The senior signal is trade-offs with numbers: naming the lever, the number it moves, and what it costs elsewhere.
What the LLM system design round actually tests#
Most LLM system design interview questions are not checking whether you can draw a box labeled load balancer and a box labeled model. They are checking whether you can hold three numbers in your head at once: latency, throughput, and cost. Can you show how one design decision moves one of those numbers without wrecking the other two?
Everyone can sketch client, gateway, model, response. Few candidates can say which part of that path they would change first when the bill doubles. The same question is graded differently by level — here is what a passing answer looks like at each level:
| What they probe | Junior / L4 passes with… | Senior / L5+ passes with… |
|---|---|---|
| Scoping | Asks how many users there are | Asks about growth, budget ceiling, uptime target, and what a slow or wrong answer costs |
| Request path | Draws client → model → response | Explains where caching, routing, and the eval hook sit on that path, and why |
| Serving | ”Add more GPUs” | Names continuous batching, an autoscaling signal, and queueing behavior under load |
| Cost and latency | ”It should be cheaper” | Names the lever (cache, cascade, token budget), the number it moves, and its trade-off |
| Ownership | Follows the interviewer’s prompts | Drives the deep dives and states trade-offs before being asked |
The right-hand column is the senior bar. For a first AI role, hit the left column cleanly, and touch the right column once or twice to stand out.
Terms you’ll hear in this round#
Interviewers use these words without defining them. You don’t need the internals, but you should be able to say what each one is and which number it moves.
| Term | What it means | Why it matters here |
|---|---|---|
| p95 / p99 latency | The response time that 95% (or 99%) of requests beat. p99 is the slowest 1%. | Averages hide slow users. Latency targets are almost always set on p95 or p99. |
| QPS | Queries per second, the request rate the system has to absorb. | Drives how much serving capacity you need. |
| TTFT | Time to first token: how long until the first word of the answer streams back. | With streaming, a user judges speed by TTFT, not by when the last word arrives. |
| Token | The unit a model reads and writes, roughly three-quarters of a word. Input tokens are your prompt and context; output tokens are the answer. | Hosted APIs bill per token, so token counts are the cost model. |
| Continuous batching | The serving engine slots new requests into the batch as soon as old ones finish, instead of waiting for a whole batch to complete. | Keeps the GPU busy, which raises throughput without adding hardware. |
| KV cache | Memory the model uses to hold its working state for every request in flight. Long prompts and long answers use more of it. | When it fills up, new requests queue. That makes it a better scaling signal than GPU utilization. |
| Exact-match cache | Stores answers keyed by the exact request text. | Free hits for repeated questions, but misses anything worded differently. |
| Semantic cache | Stores answers keyed by the meaning of the question (an embedding), so reworded duplicates also hit. | Catches far more repeats, at the risk of serving a wrong match. |
| Prompt (prefix) caching | The model provider or serving engine reuses the processed form of a prompt prefix that repeats across requests, such as a long system prompt. | Cuts cost and TTFT for the part of every request that never changes. |
| Model routing / cascade | Send easy questions to a small, cheap model and hard ones to a large model. | Lowers average cost per question without lowering quality on hard cases. |
| Graceful degradation / load shedding | Under overload, deliberately drop to a cheaper mode or reject low-priority work instead of failing everything. | Keeps the core feature alive during a spike. |
| Golden set | A fixed set of questions with known-good answers, used to test every change. | The basis of an offline eval gate before deploy. |
| Drift | Quality or behavior slowly changing in production with no code change, because the data, traffic, or a dependency moved. | Why monitoring needs alerts on trends, not only on errors. |
The Five-Layer LLM System Answer#
Strong answers to an LLM system design interview question tend to follow the same order, so memorize the order, not the content. I call that order the Five-Layer LLM System Answer. State the five layers out loud at the start so the interviewer knows where you are going, then walk them one at a time.
The time boxes matter: roughly 5 minutes for scope, 8 for the request path, 10 each for serving and for cost and latency, and 7 for eval and monitoring. Candidates who spend fifteen minutes drawing the request path never reach cost and latency, the layer where seniors separate from everyone else. If the interviewer pulls you into an early deep dive, follow them, but keep a mental clock and steer back to layer four.
A system design answer sounds complete long before it is. The gap shows up in the numbers. Saying “add caching” names a technique.
Saying “a 30% cache hit rate removes 30% of model calls, so the token bill drops by about that share, but p95 barely moves because the slowest requests are still cache misses” operates at a different level entirely. I’d lead with the lever, then the number, then the trade-off, every time.
The running example#
Use one concrete scenario for the whole round. This one continues directly from Part 1’s support chatbot, now scaled up:
The same support chatbot from Part 1, now growing fast. It handles 50,000 questions a day, growing roughly 3x over the next two quarters, on a fixed monthly compute budget. The 3-second p95 target holds, and uptime needs to stay above 99.9%. Launch-day spikes can hit five times a normal day’s traffic.
Those numbers force real decisions. A fixed budget with 3x growth means cost per question has to drop; total spend can’t simply grow with traffic. A 5x spike means the design needs headroom or graceful degradation rather than servers added after the fact. The uptime target rules out a design with one model-serving pool and no fallback.
That diagram is where the answer ends up. Build it on the whiteboard one layer at a time; the question bank below follows the same order.
Do the capacity math out loud#
Before drawing anything, turn the scenario into load numbers. These come straight from the scenario above, so you can derive them in the room:
| Number | Math | What it tells you |
|---|---|---|
| Average load today | 50,000 ÷ 86,400 seconds ≈ 0.6 QPS | Raw throughput is not the hard part at this size. |
| Average after 3x growth | 150,000 ÷ 86,400 ≈ 1.7 QPS | Still modest on average. |
| Launch day after growth | 5 × 1.7 ≈ 9 QPS averaged over the day, higher in the busiest hour | Size capacity for this, not for the average. |
| Requests in flight at peak | 9 QPS × 3 s ≈ 27 at once (Little’s law: rate × time in system) | How many generations the pool must batch concurrently. |
| Input tokens per day, today | 50,000 × ~2,000 (five retrieved chunks plus the prompt) ≈ 100M | Multiply by your model’s per-million-token price to get the input bill. |
| Downtime budget at 99.9% | 0.1% of a 30-day month ≈ 43 minutes | One bad deploy on a single pool with no fallback can use all of it. |
Saying these out loud does two jobs. It shows the interviewer you size before you build, and it tells you where to spend the rest of the round: cost per question and tail latency during spikes, not raw scale.
LLM system design interview questions: weak vs strong answers#
Each question below maps to one of the five layers. Read the weak answer first: it sounds reasonable, and that is why it fails. Then the strong answer, then the follow-up an interviewer uses to check whether you memorized it.
Layer 1 — “What would you ask before designing anything?”#
“How many users do we have?” Then straight into architecture.
“What’s the growth curve, not just today’s volume? Is the compute budget fixed, or does it grow with revenue? What’s the uptime target, and what does a slow or wrong answer cost us: an annoyed user, a refund, or a safety incident? And is the 3-second target measured to the first streamed token or to the full answer?”
The follow-up: “Why does a fixed budget change your design?” I’d push back on “add more GPUs” as the default answer to growth. A fixed budget means the design has to make each question cheaper, not just add more capacity.
Layer 2 — “Walk me through a request end to end.”#
“The request hits the model and it responds.”
“It hits a gateway, which checks an exact-match and a semantic cache before anything expensive happens. A miss goes to a router that picks a model tier by task difficulty, then into an autoscaled serving pool running continuous batching. The response streams back, and a monitoring hook logs latency, cost, and a quality signal on the way out.”
The follow-up: “Would you put the cache before or after the router?” Before. A cache hit should never touch the router or the model at all. Putting the cache after the router wastes the one layer that’s nearly free.
Layer 3 — “How do you serve the model efficiently at this volume?”#
“Run it on a big GPU and scale horizontally.”
“Continuous batching, so the serving engine keeps the GPU full by merging requests mid-generation instead of waiting for a batch to complete together (the approach behind vLLM, whose PagedAttention manages the KV cache in small pages). Autoscale the pool on queue depth and KV-cache usage, not request count or GPU utilization. With continuous batching the GPU looks busy almost all the time, so utilization says little about headroom, and a handful of long generations can fill the KV cache of a pool that looks quiet by request count.”
The follow-up: “Your p95 holds but p99 is terrible. What do you check?” A few long-tail requests are probably queued behind long generations. Look at queue depth and tail latency per replica, not just the averages.
Layer 3 — “Traffic spikes 5x on launch day. What happens?”#
“Autoscaling handles it.”
“Autoscaling has a ramp-up delay, so the real plan is headroom plus graceful degradation: pre-warm extra capacity ahead of a known launch, queue requests with a visible “still working” state instead of timing out, and fall back to a cheaper, faster model for the spike window where quality can flex more than latency can.”
The follow-up: “What if the spike is unplanned?” Shed load by tier. Deprioritize or batch non-interactive requests first, and keep the interactive chat path protected as long as possible.
Layer 4 — “The bill is too high. What do you cut first?”#
“Switch to a cheaper model.”
“Measure where the tokens actually go first. Here, five chunks plus the prompt is roughly 2,000 input tokens per question against a few hundred output tokens. Output tokens usually cost more each, so check both sides, but at that ratio the input side is big enough to attack directly. Cache whole answers first, since a real hit skips the model call entirely. Turn on prompt caching second: the system prompt is identical on every request, and hosted APIs and vLLM can reuse that shared prefix instead of reprocessing it. Route easy questions to a cheaper model third. Only then trim retrieved context, since that lever is the most likely to hurt answer quality.”
The follow-up: “How do you know a question is ‘easy’ enough to route to the cheaper model?” Classify it first with a cheap heuristic or small model: question length, topic match, retrieval confidence. Fall back to the bigger model whenever that classifier is unsure.
Layer 4 — “Two users ask near-identical questions seconds apart. How do you avoid paying twice?”#
“Cache the exact question text.”
“Exact-match caching only catches identical strings, so add semantic caching: embed the question, and serve a cached answer when a new one is close enough in meaning to one already answered, even when the wording differs. That brings back the problem caching invalidation always comes back to: these articles update daily, so a cached answer has to expire or get checked against the source article’s last-updated timestamp, or the bot will confidently serve a stale answer.”
The follow-up: “How close is ‘close enough’ for a semantic cache hit?” Set the threshold conservatively at first. Measure how often a near-duplicate gets missed, and how often a real difference gets wrongly merged. Tune the threshold from there instead of guessing a number.
Layer 5 — “How do you know a change didn’t make it worse?”#
“We’ll monitor it after we ship.”
“Gate it before ship: run an offline eval suite against a golden set, and block the deploy if quality or latency regresses past a threshold. After ship, watch the same metrics live, plus cost per question, and alert on drift, not just outright failure.”
The follow-up: “Your offline evals all passed, but the new model version times out under load. What do you check?” Offline evals usually run one request at a time, so they miss serving-level problems. Check batch behavior and GPU memory under concurrent load. Also check whether the new model’s generation length changed enough to shift its latency profile.
Layer 5 — “An answer was fine for weeks and is suddenly wrong. What’s your first move?”#
“Roll back the last deploy.”
“Check what changed that isn’t the code: the underlying documents, the traffic mix, or a silently-updated embedding or serving dependency. A rollback only helps if the last deploy actually caused it; if the cause is upstream data or traffic drift, rolling back buys time without fixing anything.”
The follow-up: “Nothing you deployed changed. What now?” Compare retrieval and generation metrics over the drift window to find which stage changed. Then check for a shift in question types or source-document edits before assuming the model itself regressed.
Red flags that sink the round#
Interviewers rarely fail you for one wrong detail on an LLM system design interview question. They fail you for patterns, not details, and these five are the ones that end the round:
- Jumping straight to “add more GPUs” or “use a bigger model” before asking about budget or growth.
- Treating cost as an afterthought instead of a first-class constraint, same as latency.
- Naming a caching strategy without mentioning invalidation, especially when the source data changes on a schedule.
- No eval or monitoring plan beyond “we’ll keep an eye on it.”
- Designing for today’s traffic with no answer for what breaks at 3x or 5x.
Learn it by building#
You cannot fake these follow-ups with memorized talking points. Candidates who have built the pieces handle follow-ups that memorizers can’t. These posts walk through each piece in order:
- System Design Building Blocks: A Decision Map, Not a Parts List — the decision framework behind layers two and three.
- Caching Strategies: Why Invalidation Is the Hard Part — the semantic-cache invalidation question, in full.
- Load Balancers Explained: Health Checks Do the Real Work — the request-path layer, hands-on.
- Horizontal vs Vertical Scaling: Scale Up Until You Can’t — the autoscaling question, in depth.
- Rate Limiting Explained: Token Bucket, Leaky Bucket, and When Each Wins — protecting the serving pool under a spike.
- RAG Interview Questions & Answers (2026): Strong vs Weak — Part 1 of this series, the retrieval layer this system relies on.
- How to Evaluate an AI Agent: Metrics & Frameworks — the metrics layer five expects you to name.
Start with system-design-building-blocks and caching-strategies for the concepts, then rate-limiting and load-balancers for the request path. Scaling comes last, once you have something worth scaling.
What to take into the round#
The LLM system design interview rewards numbers over vocabulary. Walk the five layers in order, keep one example running through all of them, and spend your best minutes on cost and latency trade-offs: that is where offers are decided. Which layer feels shakiest for you right now: serving, caching, or monitoring?
Next in the series: the evals round, the newest round and the easiest to under-prepare for. In the meantime, the AI Engineer Interview Questions hub summarizes all five rounds on one page.
