A reviewer opens a repository called rag-chatbot. The README lists the libraries used, shows one screenshot, and ends. There is no number, no live link and no sign of anything going wrong. The reviewer closes the tab and learns nothing about the person.
Now the same project with a two-line summary, a demo that loads, a hit rate before and after a chunking change, and a note saying what broke on day two. Same code. Very different evidence. This is Part 4 of the AI Engineer Interview Prep series, and it shows how to build the second version.
- You have followed at least one capstone in the course, or built something similar.
- You can run Python and git locally.
Where should an eval dataset come from, and how big should it start? Real failures, 20 to 50 tasks. The portfolio uses the same idea: your own failures are the evidence.
- A portfolio project is evidence for interview answers, not a list of libraries.
- Build one project in four layers: knowledge (RAG), actions (agent), proof (evals), live (deployment).
- Each layer reports one measured number with its setup and what it does not prove.
- A failure log is the strongest part of the README, because it cannot be copied from a tutorial.
- Finish small. One project that runs and is measured beats four that almost work.
Plan one portfolio project in four layers, write the README a reviewer reads in order, and name the number, the failure and the interview round each layer supports.
What does a portfolio project have to prove?#
It has to prove that you can do what the interview asks about. Every round in this course asks a version of “how would you know it works, and what breaks?” A project gives you real answers instead of rehearsed ones.
That changes how to choose. Do not pick the project with the most impressive name. Pick the one that lets you answer a question with your own number.
| Round or topic | The question | Evidence your project gives |
|---|---|---|
| RAG | How do you know retrieval works? | A hit rate on your own questions, before and after a change |
| Agents | What happens when a tool fails or it loops? | A guard firing in a logged run |
| Evals | How do you know a change helped? | A golden set and a failing check in CI |
| System design | How would you run this for real users? | A live URL with a cost cap and logs |
Anthropic’s advice for agent builders fits here too: it recommends finding “the simplest solution possible” and adding complexity only when needed. For a portfolio that means a small system you fully understand.
One project, four layers#
The course has four capstones. Instead of four separate repos, treat them as layers of one support assistant that answers questions from documents and can take a limited action.
Each layer maps to a capstone you can follow:
- Knowledge: chat with your documents, fully local.
- Actions: a support agent with tools, memory and escalation.
- Proof: an eval suite that runs free on a local model.
- Live: deploy the agent to Cloud Run or Fly.io.
One honest caveat. The capstones use different libraries, so combining them takes some adaptation. Pick one stack for the portfolio and keep it. A project that runs on one stack and is measured is better than a clever mix that nobody can reproduce.
What must each layer show?#
- Knowledge. Show the documents you used and a number. The chunking lesson measures a hit rate: how often the right passage appears in the top results. Report it on at least 20 real questions, before and after one change such as chunk size. Say what the number does not prove, for example that your questions came from one document set.
- Actions. Show a run where a guard fires. A step cap that stops a loop, or an argument check that rejects a bad refund, is more convincing than a demo that always works. The guardrails lesson builds the checks.
- Proof. Show the golden set and the gate. Anthropic’s guide to agent evals suggests 20 to 50 simple tasks drawn from real failures. Put them in the repo, run them in CI, and show one pull request that the check blocked. The CI lesson shows how.
- Live. Show a URL that loads, with secrets kept out of the repo, a spending cap and request logs. A free tier may sleep, so say how long the first request takes.
Each layer has the same three parts: a thing that runs, one number, and one failure. If a layer lacks any of the three, it is not ready to show.
What goes in the README, and in what order?#
A reviewer reads from the top and stops when they have an answer or run out of patience. Put the proof first and the setup last.
Here is what the top of a README can look like, with placeholders for your own facts:
Support assistant (local, free)
Answers questions from <N> help
docs; can issue small refunds.
Demo: <link> Hit rate: <x>/20
Stack: <one stack, pinned>The failure log deserves the most care. It is the one section that cannot be copied from a tutorial, and it is exactly what the interviewer will ask about. A good entry has four parts:
What happened: agent refunded
the same order twice.
Cost: 40 minutes, one bad test.
Fix: idempotency key + step cap.
Now: check runs in CI.Keep a running file from day one. Writing it from memory at the end produces vague entries like “had some bugs”.
How big should it be?#
Smaller than you think. Decide the scope before you start and write it down.
Pick one document set, one or two tools, one golden set, one deployment. Add a feature only when a layer’s number or failure log needs it. A useful rule: if you cannot explain a part of the system out loud in two minutes, cut it or learn it before showing it.
- A tutorial copied with no change, and no sign of what you learned.
- No measurement, so every claim about quality is an opinion.
- A demo link that is dead by the time the reviewer clicks it.
- API keys or
.envfiles committed to the repository. - Claiming scale or accuracy you did not measure.
Before you share it#
A short checklist keeps small mistakes from sinking a good project:
- Run the README instructions on a clean machine or a fresh clone.
- Search the repo and its git history for keys and personal data.
- Use documents you are allowed to share, or public ones.
- Pin versions so the project runs next month.
- Check the demo link from a private browser window.
Pick one capstone you built. Without looking at the README, say aloud the number, the failure and the round it supports. If you stall, that part of the README is missing something. Fix it, then try again tomorrow.
For one layer of your project, fill in this block in your README:
Layer:
Question it answers:
Number + how measured:
What it does NOT show:
Failure + fix:Then say it aloud in under a minute as if answering “tell me about this project”.
Hint
Expected output
The shape to aim for:
Layer: Knowledge
Q: does retrieval work?
Number: hit rate on 20
questions, before/after
a chunk-size change
Not shown: other doc sets
Fix: overlap, re-measured1.Name the four layers of the portfolio project and the interview round each supports.
Knowledge for the RAG round, actions for the agents round, proof for the evals questions, live for the system design round.
2.Your project has a RAG pipeline and a demo, but no number. What do you add first?
A small set of real questions with the correct source for each, then a hit rate before and after one change. A number with its setup is what turns the demo into evidence.
3.Why is the failure log more valuable than a feature list?
It cannot be copied from a tutorial, it shows you debugged something real, and it gives the interviewer the follow-up questions you are best prepared for.
4.You have two weeks. Do you build a fifth feature or measure what you have?
Measure. A finished layer with a number and a failure log is better evidence than another unmeasured feature.
In one picture#
Layers knowledge, actions,
proof, live
Each runs + one number + one
failure
README top lines, demo, number,
failures, code
Scope one project, finished- Local RAG: chat with your documents: the knowledge layer.
- Build a customer support AI agent: the actions layer.
- Evaluate an AI agent on a local LLM: the proof layer.
- Deploy an AI agent to Cloud Run or Fly.io: the live layer.
- AI Engineer Interview Questions: every round at a glance.
Next in the course: presenting the project in the behavioral round, then a prep plan that schedules the practice.
