“Design a RAG system for our customer-support chatbot.” Of all the RAG interview questions in an AI engineer loop, this is the most common opening — and where most candidates quietly lose. Not because they don’t know RAG (retrieval-augmented generation: fetching relevant documents so an LLM answers from real sources), but because they answer with buzzwords instead of a design.
This is Part 1 of the AI Engineer Interview Prep series. The hub post covers every round; this one goes deep on RAG. You get one answer framework, one example threaded through every step, and a question bank that shows each answer twice: the weak version and the strong one.
- Use one fixed skeleton: scope → ingest and chunk → retrieve and rerank → generate with citations → evaluate and diagnose.
- Time-box it. The whole answer fits in about 40 minutes, and the last box (evaluation) deserves the most time.
- Thread one example through every step instead of answering in the abstract.
- The senior signal is measurement: naming metrics, building a golden set, and splitting retrieval failures from generation failures.
What the RAG round actually tests#
Most RAG interview questions are not checking whether you have heard of vector databases. They are checking whether you treat RAG as a production system with failure modes, costs and a way to measure quality. Everyone can draw “docs → embeddings → vector DB → LLM.” Few candidates can explain what breaks, how they would notice, and what they would change.
The same question is graded differently by level. Here is what a passing answer looks like at each:
| What they probe | Junior / L4 passes with… | Senior / L5+ passes with… |
|---|---|---|
| Scoping | Asks how many documents there are | Asks about freshness, permissions, latency budget and the cost of a wrong answer |
| Chunking | A sensible fixed size with overlap | Structure-aware chunks plus metadata, tuned against an eval set |
| Retrieval | Vector search with top-k | Hybrid keyword + vector search with filters, justified by real query types |
| Evaluation | ”We’d check the answers are accurate” | Named metrics, a golden set, and a retrieval-vs-generation failure split |
| Ownership | Follows the interviewer’s prompts | Drives the deep dives and states trade-offs before being asked |
If you are aiming for senior, the right-hand column is the bar. If you are aiming for a first AI role, hitting the left column cleanly (and touching the right column once or twice) is what stands out.
The Five-Box RAG Answer#
Strong RAG answers tend to follow the same order, so memorize the order, not the content. I call that order the Five-Box RAG Answer. Say the five boxes out loud at the start so the interviewer knows where you are going, then walk them one at a time.
The time boxes matter. Candidates who spend 20 minutes on embedding models never reach evaluation, the box where seniors are separated from everyone else. If the interviewer pulls you into an early deep dive, follow them, but keep a mental clock and steer back to box five.
A RAG pipeline looks like it works until you measure it. A golden set of even 50 real questions usually surfaces answers that cite the wrong article or miss a step. Tell that story — you measured, found the gap, fixed it. Not built one yet? The RAG in Python series gets you there in a weekend.
The running example#
Use one concrete prompt for the whole round. Here is the one this series uses, and the one you should rehearse:
Design a customer-support chatbot over our help center. There are about 12,000 articles, updated daily, across several product versions. Answers must cite their source article, and the bot should respond in under 3 seconds at p95 (for 95% of requests).
Those few numbers change the design, which is the point. Daily updates mean you need incremental indexing. Multiple product versions mean metadata filters. Mandatory citations mean your chunks must keep their source URL. A 3-second budget limits how heavy your reranker can be.
This is the design you are steering towards. You do not draw it all at once; you build it box by box, and the question bank below follows the same order.
RAG interview questions: weak vs strong answers#
Each question below maps to one of the five boxes. For each one you get the weak answer (which sounds fine and still fails), the strong answer, and the follow-up the interviewer uses to catch a memorized response.
Box 1 — “What would you ask before designing anything?”#
“How many documents are there?” Then straight into architecture.
“How fresh does it need to be? Are some articles restricted to certain customers? What’s the latency budget? And what does a wrong answer cost: an annoyed user, or a refund we didn’t owe?”
The follow-up: “Why does the cost of a wrong answer matter?” Because it decides whether the bot may say “I don’t know,” whether you need human handoff, and how much you spend on reranking and evaluation.
Box 2 — “How would you chunk these articles?”#
“Split everything into 512-token chunks with some overlap.”
“Help articles already have structure, so I’d split on headings and keep each chunk under a size limit. Every chunk keeps metadata (article URL, product, version, last-updated date) because citations and filters depend on it. Then I’d tune chunk size against a small eval set rather than guess.”
The follow-up: “What happens to a step-by-step procedure or a table that gets split across two chunks?” Retrieval returns half the steps and the answer is confidently incomplete. Strong candidates mention keeping procedures and tables whole, or adding a short context line to each chunk, the idea behind Anthropic’s contextual retrieval.
Box 2 — “Which embedding model would you use?”#
“Whichever is top of the leaderboard.”
“I’d shortlist two or three, then compare recall@k (how often the right article lands in the top k results) on our own support questions, since leaderboards don’t know our product names. I’d also weigh cost per million tokens, vector size, and whether we need multilingual support.”
The follow-up: “Six months later you switch embedding models. What happens?” Every vector in the index is now incompatible, so you re-embed the whole corpus. Strong candidates version the index and run old and new side by side before cutting over.
Box 2 — “An article is edited or deleted. How does the index stay correct?”#
“Re-index everything every night.”
“Re-indexing 12,000 articles nightly is wasteful and still leaves the index up to a day stale. I’d key every chunk by article ID and store a content hash. When an article changes, re-chunk and replace only its chunks; when it’s deleted, delete every chunk with that article ID. A last-updated field in the metadata lets the bot prefer the newest version.”
The follow-up: “You also cache answers. The article changes, now what?” Tag each cached answer with the article IDs it cited and invalidate those entries when an article changes. Candidates who forget the cache serve stale answers for hours.
Box 3 — “How do you retrieve the right chunks?”#
“Vector search, top 5.”
“Hybrid search. Support questions are full of exact strings (error codes, product names, setting labels), and keyword search (BM25) catches those while vector search catches paraphrases. I’d merge both lists with reciprocal rank fusion, take the top 50, and apply metadata filters for product version and access rights before anything reaches the model.”
The follow-up: “A user types error E-4012 on checkout. What does pure vector search do?” It tends to return generic checkout articles, because an error code carries little semantic meaning. This one question exposes candidates who have never debugged real retrieval.
If they ask how the two lists are merged, reciprocal rank fusion fits on a whiteboard. Each document scores 1 / (k + rank) in every list it appears in, and the scores add up:
def reciprocal_rank_fusion(result_lists, k=60):
"""Merge ranked lists of doc IDs. k=60 is the common default."""
scores = {}
for results in result_lists:
for rank, doc_id in enumerate(results, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
bm25_hits = ["kb-4012", "kb-0188", "kb-0931"] # exact match on "E-4012"
vector_hits = ["kb-0931", "kb-4012", "kb-2210"] # semantic matches
print(reciprocal_rank_fusion([bm25_hits, vector_hits]))
# ['kb-4012', 'kb-0931', 'kb-0188', 'kb-2210']Articles that rank well in both lists rise to the top, and you never have to compare a BM25 score with a vector similarity score. That is the whole reason RRF is popular.
Filter for permissions at retrieval time, never by asking the LLM to ignore restricted content. If a chunk reaches the prompt, assume it can reach the user.
Box 3 — “Why add a reranker, and what does it cost?”#
“Rerankers improve accuracy.”
“Retrieval is tuned for recall, so the top 50 is noisy. A cross-encoder reranker scores each question–chunk pair together, which is far more precise, and I keep the top 5. The cost is latency, so I’d measure it against our 3-second budget.”
The follow-up: “Your p95 latency doubled after adding the reranker. Now what?” Rerank fewer candidates (30 instead of 50), use a smaller reranker, cache answers to frequent questions, or run it only when retrieval scores are close together.
Box 4 — “How do you stop the model from making things up?”#
“Tell it not to hallucinate in the prompt.”
“Instruct it to answer only from the provided context, cite the article for every claim, and say it doesn’t know when the context doesn’t cover the question. Put the most relevant chunks first and last, because models attend less to the middle of long contexts. Then measure faithfulness instead of trusting the prompt.”
The follow-up: “Two retrieved articles contradict each other, an old version and a new one. What happens?” Without version metadata, the model may blend them. Strong answers filter or rank by version and date, and surface the conflict instead of averaging it away.
Box 4 — “Can this actually hit 3 seconds? What does it cost?”#
Senior candidates put numbers on the whiteboard here instead of saying “it should be fast.” Budget the latency stage by stage. The figures below are illustrative ballparks; the point is the method, and you should measure your own stack:
| Stage | Ballpark (p95) | What moves it |
|---|---|---|
| Embed the query | 50–100 ms | Hosted API vs a local model |
| Hybrid retrieval + filters | 30–100 ms | Index size, filter selectivity |
| Rerank 50 candidates | 100–400 ms | Model size, GPU vs CPU, candidate count |
| LLM answer (~250 tokens) | 1.5–2.5 s | Model, output length, provider load |
| Total | ~1.7–3.1 s | Tight against a 3 s budget |
The table tells you what to say next: the LLM dominates, so stream the answer. The user sees text after the first token, well under a second, and keep answers short.
Cost follows the same logic. Five 300-token chunks plus the prompt is roughly 2,000 input tokens per question, so 50,000 questions a day is about 100 million input tokens a day. That line item, not the vector database, dominates the bill, so caching frequent questions and trimming context are your first cost levers.
Box 4 — “The user asks a follow-up: what about on mobile?”#
“Send the new message to retrieval like any other question.”
“On its own, what about on mobile? retrieves nothing useful. Before retrieval I’d rewrite it into a standalone question using the conversation so far (how do I reset my password in the mobile app?) with a small, cheap model, then retrieve on the rewritten query.”
The follow-up: “Isn’t that an extra LLM call on every turn?” Yes, so use a small model, skip the rewrite when the message is already self-contained, and keep only the last few turns of history.
Box 5 — “How would you evaluate this system?”#
This is the question that decides senior offers, so give it the most time.
“We’d test it and check the answers look right.”
“I’d build a golden set of 100–200 real questions from support tickets, each with the correct source article. For retrieval I’d track recall@k and MRR (mean reciprocal rank — how near the top the right article appears). For the full pipeline I’d use Ragas-style metrics: context precision and context recall on the retrieval side, faithfulness and response relevancy on the generation side. Every change (chunk size, embedding model, reranker) has to move those numbers before it ships.”
The follow-up: “Where does your ground truth come from?” Resolved support tickets, plus subject-matter experts reviewing a sample. Synthetic questions generated by an LLM are fine to bootstrap, but a human checks them before they become the bar.
Box 5 — “An answer is wrong. Walk me through debugging it.”#
“I’d try a better prompt or a bigger model.”
“First I look at the retrieved chunks. If the right article isn’t there, it’s a retrieval failure: chunking, embeddings, filters or the query itself. If it is there and the answer still drifts, it’s a generation failure: prompt, context order or the model. I’d tag every failure in the eval set with one of those buckets so I fix the stage that actually breaks most.”
The follow-up: “Retrieval is fine but the question is vague, like it’s not working.” This is where agentic RAG comes in: an agent that rewrites or clarifies the query and retrieves again. Know when it earns its extra latency and cost, and when one retrieval pass is enough. The agentic RAG guide covers that trade-off.
Red flags that sink the RAG round#
Interviewers rarely fail you for one wrong detail on RAG interview questions. They fail you for patterns, and these five are the ones that end the round:
- Jumping to “embed everything and use a vector DB” before asking a single scoping question.
- Never mentioning evaluation, or describing it as “checking that answers look right.”
- Treating every failure as a prompt problem instead of separating retrieval from generation.
- Ignoring permissions, freshness or document versions in a multi-product help center.
- Naming tools (a specific vector database or framework) instead of explaining the trade-off behind the choice.
Learn it by building, not by memorizing#
You cannot fake the follow-ups. Every “strong” answer above comes from having built a pipeline and watched it fail, so the fastest prep is to build the running example yourself. These posts walk through it in order:
- What Is RAG? A Practical Developer’s Guide — the mental model behind box one.
- Embeddings and Vector Search: A Primer — what you need for the embedding questions.
- Build a RAG System in Python From Scratch (Part 1) — the end-to-end skeleton.
- RAG Chunking & Retrieval Quality (Part 2) — box two, hands-on.
- Semantic Chunking & Re-Ranking (Part 3) — box three’s reranker in practice.
- Vector Database for RAG (Part 4) — when the index needs to grow up.
- How to Evaluate an AI Agent: Metrics & Frameworks — the metrics box five expects you to name.
Build Parts 1–3 over a weekend, add a 50-question golden set, then break the pipeline on purpose — shrink the chunks, drop the reranker — and watch the numbers move. After that, the follow-ups stop feeling like traps.
The bottom line#
RAG interview questions reward structure and measurement, not vocabulary. Walk the five boxes in order, keep one example running through all of them, and spend your best minutes on evaluation and failure diagnosis: that is where offers are decided. Which box feels shakiest for you right now, chunking, retrieval, or evaluation?
Next in the series: the agents round — the one most candidates fail. Until it’s live, the AI Engineer Interview Questions hub has every round at a glance.
