“We changed the prompt and it feels better. Can we ship?” Every AI engineer interview has a version of this question, and most candidates answer it with “let’s try some examples.”
That answer ends the round early. Anyone can try five examples and like the result. The evaluation round checks whether you can turn “feels better” into a number you trust, and whether you know where that number lies to you. This is Part 3 of the AI Engineer Interview Prep series. Part 1 covered RAG and Part 2 covered agents. This one covers how you prove either of them works.
- You have built a RAG pipeline or an agent, even a small one.
- You have read the agent evaluation guide or are about to.
An agent run went wrong. What do you read first: the prompt or the trace? The trace. Evaluation is that same habit applied to many runs at once instead of one.
- Use one skeleton: goal, dataset, graders, calibration, gate, monitoring.
- Start with 20 to 50 real failures, not a thousand invented questions.
- Match the grader to the question: code for exact facts, a calibrated LLM judge for judgment, humans for the gold labels.
- Flaky systems need repeated trials. One pass proves little, and pass^k shows how fast reliability decays.
- Reading transcripts is part of the job. It is how you find out your grader is wrong.
Answer “how would you evaluate this system?” with six steps, pick the right grader for a given check, explain pass@k vs pass^k with arithmetic, and say how an eval blocks a bad pull request.
What does the evals round actually test?#
It tests whether you treat quality as something you measure, not something you feel. Interviewers watch for three habits: you start from real failures, you know every metric has a blind spot, and you wire the measurement into the release process so it cannot be skipped.
Here is what passing looks like at two levels:
| What they probe | Junior answer that passes | Senior answer that passes |
|---|---|---|
| Dataset | ”A set of example questions with answers” | Built from real failures, versioned, split into regression and capability sets |
| Graders | Names one metric | Chooses code, model or human per check and says why |
| LLM judges | ”We use GPT to score it” | Calibrates the judge against humans and tests for bias |
| Flakiness | Reruns until it passes | Runs repeated trials and reports pass^k |
| Release | ”We check before deploy” | A required CI check with a stated threshold |
The six-step evals answer#
Say the steps out loud at the start: goal, dataset, graders, calibration, gate, monitoring.
Notice the dashed line. An eval is not a one-time test. It is a loop, and the loop is what interviewers want to hear about.
The running example#
Use one prompt for the whole round:
A support bot answers questions from a help center, and can also issue refunds. The team wants to ship a new system prompt and a new embedding model this week. How do you decide whether that is safe?
Two changes at once is on purpose. A strong candidate points out that they should be evaluated separately, or the result cannot be traced to either one.
LLM evaluation interview questions: weak vs strong answers#
How do you know the new prompt is better?#
“I would try some examples and see if the answers look better.”
“I would run the old and new versions on the same fixed set of tasks and compare scores, one change at a time. The set comes from real failures. A change ships only if the score does not drop on the regression set and improves on the cases it was meant to fix.”
The follow-up: “You changed the prompt and the embedding model together. What can you conclude?” Almost nothing. Two changes confound each other, so a better score cannot be credited to either. Run them separately, or at least run all four combinations.
Where does the dataset come from, and how big is it?#
A fixed set of tasks with expected outcomes is called a golden set. The other lessons in this series use the same name.
“We ask an LLM to generate a few thousand questions.”
“I would start with 20 to 50 tasks taken from real failures: support tickets, thumbs-down feedback, bad traces. Each task has an input and an expected outcome. I would version the set, and add every new production failure to it.”
The follow-up: “Why not generate thousands?” Generated questions drift toward what the generator finds easy, and nobody has checked their answers. Anthropic’s guide to agent evals recommends starting with 20 to 50 simple tasks drawn from real failures, and turning user-reported failures into test cases. Synthetic data is fine to widen coverage once a human reviews it.
Split the set in two. A regression set holds tasks the system already passes, and it should stay near 100%. A capability set holds hard tasks that start with a low pass rate and measure progress. Anthropic’s same article describes both.
Which metrics would you track?#
“Accuracy, and maybe BLEU.”
“It depends on the layer. For retrieval: did the right passage come back, and how much of what came back was useful. For generation: is the answer supported by the passages, and does it address the question. For the agent: was the right refund issued, and did it stay within cost and step budgets.”
The follow-up: “Which of those can you check without a model?” Whether the right document was retrieved, whether the refund record exists, whether the output matches a schema. Those are exact checks. The RAG lesson names the retrieval and generation metrics, and the metrics guide covers the framework options.
If the interviewer asks for definitions, the Ragas library gives precise ones in its docs. Faithfulness is the share of claims in the answer that the retrieved context supports. Context recall is the share of claims in a reference answer that the retrieved context supports, so it needs a reference. Context precision checks whether relevant chunks rank above irrelevant ones. Response relevancy checks whether the answer addresses the question, without judging whether it is true.
Code, LLM or human: which grader do you use?#
“An LLM judge, because it is easiest.”
“Code first, because it is fast, cheap and reproducible, but it breaks on valid variations in wording. An LLM judge handles nuance and scales, but it is non-deterministic and costs more, so it needs calibration against human labels. Humans are the gold standard and too slow for everything, so I use them to label a sample and to audit the judge.”
The follow-up: “Give an example of each in this system.” Code: the refund record exists and the amount matches. Judge: is the tone appropriate and the explanation clear. Human: whether the judge’s “good” matches what a support lead would accept. These trade-offs follow Anthropic’s description of the three grader types.
How do you trust an LLM judge?#
“GPT-4 is smart, so I trust its scores.”
“I would write a rubric with specific criteria, label a sample by hand, and measure how often the judge agrees with the labels before using it. I would let it answer Unknown when evidence is missing, swap the order when comparing two answers to check for position bias, and read a sample of its grades regularly.”
The follow-up: “What biases do judges have?” The MT-Bench paper by Zheng et al. reported that strong LLM judges can reach over 80% agreement with human preferences, and also documented position bias, where a judge favors an answer because of where it appears (paper). Anthropic’s advice matches: calibrate judges with human experts, and give the judge a way out by letting it return “Unknown” when it lacks information.
Using the same model family to write the answers and grade them, with no human labels. The judge shares the generator’s blind spots, so scores rise while real quality does not.
The same test passes and fails on different runs. Now what?#
Models are non-deterministic, so one run is one sample. Anthropic defines two numbers worth knowing. pass@k is the chance of at least one success in k attempts. pass^k is the chance that all k attempts succeed. Their example: a 75% per-trial success rate gives about 42% for three trials in a row.
You can check that on a napkin, assuming each trial succeeds independently:
p = 0.75 per trial
pass^1 = 0.75
pass^2 = 0.75 x 0.75 = 0.56
pass^3 = 0.75^3 = 0.42
pass@3 = 1 - 0.25^3 = 0.98The same system looks 98% reliable or 42% reliable, depending on which question you ask (the napkin numbers above, from the 75% assumption). For a coding helper where one good attempt is enough, pass@k fits. For a refund bot that must work every time, pass^k is the honest number.
Do you grade the answer or the steps?#
“Both, strictly. The agent must call the tools in the expected order.”
“I grade the outcome first: was the right refund issued, is the final state correct. Strict step checks break when the agent finds a valid different route, so I use the path as a diagnostic, not as the pass condition.”
The follow-up: “When would you check the path?” For safety and cost: a forbidden tool was never called, the step count stayed under budget. Anthropic says it is often better to grade what the agent produced, not the path it took.
Do you read the outputs, or just the scores?#
A score is a summary of many decisions you have not looked at. Anthropic’s advice is blunt: you will not know whether graders work unless you read the transcripts and grades from many trials. In an interview, say how often. A good answer is a fixed sample every week plus every failure the gate flags.
How does an eval block a bad change?#
“We run the evals before release and look at the numbers.”
“The regression set runs in CI on every pull request and is a required check. It fails the build if the pass rate drops below a threshold. Because model calls are noisy, each task runs several times and the comparison uses rates, not single results. The capability set runs on a schedule and informs planning but does not block.”
The follow-up: “The eval is flaky and the team starts ignoring it.” Then it has failed. Fix the flakiness by tightening graders, adding trials or removing ambiguous tasks, never by retrying until it passes. The series shows a working gate in AI Agent Evals in CI, with the pytest basics in the CI primer.
The score is 100% and users still complain. Why?#
Three usual causes. The set is too easy: Anthropic calls this eval saturation, when an agent passes all the solvable tasks and no room for improvement is left. The set is stale: real questions moved on and nobody added them. Or the grader measures the wrong thing, such as format instead of correctness.
The fix is the dashed line in the loop diagram. Sample live traces, turn each new failure into a test case, and retire tasks that every version passes.
Red flags that sink the evals round#
- “I tried some examples and it looked fine” as the whole method.
- A dataset made only of generated questions nobody reviewed.
- An LLM judge with no human calibration.
- Reporting a single run of a non-deterministic system.
- Never reading the actual outputs behind the scores.
- Changing two things at once and crediting the result to one.
Pick one system you built. Write down ten real failures it has had. Turn each into a task with an expected outcome, then say aloud which grader would check it and why. Do it again next week with fresh failures.
Take the support bot from the series, or any small LLM feature. Write ten tasks as a list of inputs and expected outcomes, then add one code grader:
def grade(task, output):
got = output.get("refund")
return got == task["want"]
def pass_rate(task, run, n=3):
wins = 0
for _ in range(n):
out = run(task)
ok = grade(task, out)
wins += ok
return wins / nRun each task three times. Count tasks that pass all three, and tasks that pass at least once.
Hint
Expected output
The shape to aim for, not output from a real run:
10 tasks x 3 trials
pass^3: 6 of 10 = 60%
pass@3: 9 of 10 = 90%
never passed: 1 taskThe never-passed task is the most valuable one: read it before changing anything.
1.Name the six steps of the evals answer, in order.
Goal, dataset, graders, calibration, gate, monitoring.
2.A refund bot succeeds on 90% of single trials. What is the chance it succeeds on five trials in a row, and what does that tell you?
0.9 to the power 5 is about 0.59, so about 59%. A bot that looks reliable per run still fails often across repeated use, which is why pass^k matters for anything that must work every time.
3.Your LLM judge scores everything 9 out of 10. What do you check first?
Calibrate it against human labels on a sample and read its grades. Check that the rubric has specific criteria, that it can answer Unknown, and that it is not grading its own model family’s output.
4.What would change in your eval design if the bot only answered FAQs and never touched money?
Fewer strict gates and cheaper checks. A wrong refund costs money, but a slightly off FAQ answer costs less, so you can rely more on sampled review than on a hard CI block.
In one picture#
Goal one change at a time
Dataset 20-50 real failures
Graders code, judge, human
Calibrate read transcripts
Gate required CI check
Monitor failures become tasks- How to Evaluate an AI Agent: Metrics and Frameworks: the metric names in depth.
- AI Agent Observability and Evals (Part 6): trace runs and score them.
- AI Agent Evals in CI (Part 7): the gate from this lesson, built.
- Evaluate an AI Agent on a Local LLM: the capstone, free and offline.
- Portfolio projects that get interviews: turn your golden set into evidence.
Next in the course: serving and system design, where the same eval hooks guard cost and latency changes.
