# Evaluate an AI Agent on a Local LLM: Free, No API Key (2026)
> Evaluate an AI agent on a local LLM for free — build a tiny Pydantic AI agent on Ollama and score it with pydantic-evals, agent and judge both local, no API key.
*Source: https://www.infowok.com/evaluate-ai-agent-local-llm-free/ · Sukhveer Kaur · Published July 18, 2026*
---
Every "how to evaluate your agent" tutorial hits the same wall on line one: *get an API key.* The catch nobody mentions is that the eval judge is itself a model call, so a suite you run twenty times while learning bills you twenty times over. That's a strange tax to pay just to understand how evaluation works.
You don't have to. If the agent **and** the judge both run on a model on your own machine, the entire loop costs nothing — you can run it at 2am, a hundred times, offline, and never see a bill. This guide builds exactly that: a tiny tool-using agent on a local LLM, scored with **pydantic-evals**, judged by a second local model. Same workflow the production posts use, minus the API key.
- **Ollama running with a model pulled.** New to local models? [Best Local LLM for Your Laptop](/best-local-llm-for-your-laptop/) gets you from nothing to a running model in ten minutes.
- Comfortable reading Python (functions, decorators, `async def`)
- Helpful but optional: you know what an agent is — if not, start with [What are AI agents?](/what-are-ai-agents-complete-guide-2026/)
- **The whole eval loop can run on your laptop for $0** — point both the agent and the judge at Ollama's OpenAI-compatible endpoint and no API key is ever needed.
- **One line does the trick:** an `OpenAIProvider(base_url=...)` pointed at `localhost:11434` makes every Pydantic AI call hit your local model instead of a paid API.
- **The judge is just another model call** — so make it a *local* one, and make it the strongest model you can fit, because grading is easier than solving.
- **This is the same pydantic-evals suite** the hosted tutorials use, so everything you learn here graduates straight to production and CI.
## Why Learn Evals on a Local Model First
Evaluation is a skill you learn by running it over and over — change a prompt, re-score, see what moved. That feedback loop is where the intuition lives, and it's exactly the loop a metered API makes you ration. Local models remove the meter.
There are three concrete wins to learning this way. **Cost is zero**, so you re-run without flinching — the single biggest thing that makes eval "click." **Everything stays on your machine**, so you can throw real, messy, private inputs at the agent without sending them anywhere. And the part that surprises people: **the judge is a model call too.** An LLM-as-a-judge bills you once per case *per run* on a hosted API. Move it local and that cost vanishes, which is what makes a hundred practice runs actually free.
A small local judge is a fine teacher, not a courtroom. Once you're evaluating something you'll ship, calibrate the judge against a few answers you've graded by hand — and expect to swap in a stronger (often hosted) judge for the calls that matter. We come back to this at the end.
**Bottom line: local models turn eval from a metered expense into an unlimited practice loop — the right place to build the skill.**
## Step 1 — Point Pydantic AI at Your Local Model
Ollama serves every model at `http://localhost:11434` with an **OpenAI-compatible** API, which means the whole Python ecosystem talks to it unchanged — you just change the `base_url`. In Pydantic AI, that's an `OpenAIProvider` wrapped around a model name:
```python
# provider.py — send every Pydantic AI call to your local Ollama server
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
ollama = OpenAIProvider(
base_url="http://localhost:11434/v1", # Ollama's OpenAI-compatible endpoint
api_key="ollama", # required by the client, ignored by Ollama
)
agent_model = OpenAIChatModel("llama3.1:8b", provider=ollama) # the agent under test
judge_model = OpenAIChatModel("qwen3.5:9b", provider=ollama) # a local judge, at least as strong
```
Two models, one provider. The **agent model** is the thing you're testing; the **judge model** grades its answers in Step 3. They can be the same model, but a judge that's at least as capable as the agent is the whole reason LLM-judging works — more on that shortly. Swap either tag for anything you've pulled that fits your RAM (`ollama pull llama3.1:8b`).
The `api_key="ollama"` string is a placeholder the OpenAI client insists on; Ollama ignores it. If you see an auth error, it's your `base_url`, not your key — check the port and that `ollama serve` is running.
**Bottom line: one `base_url` line reroutes every model call to your laptop — the single change that makes this whole tutorial free.**
## Step 2 — A Tiny Agent Worth Testing
You need something with a way to fail before scoring means anything. So the agent gets exactly **one** real tool and a clear rule: answer only from tools, never guess. That setup builds in the classic agent failure — a confident, made-up answer — so the eval has a real target.
```python
# agent.py — a minimal ops agent with one tool
from pydantic_ai import Agent
from provider import agent_model
agent = Agent(
agent_model,
instructions=(
"You are an ops support agent. Answer only from tool results. "
"If no tool can answer the question, say you don't have that data — "
"never guess a number."
),
)
@agent.tool_plain
def disk_free(box: str) -> str:
"""Return the free disk space on a named server box."""
inventory = {"ops": "42 GB free", "web": "11 GB free"}
return inventory.get(box, "unknown box")
```
The agent can answer a disk question because `disk_free` exists. Ask it anything else about the box — CPU temperature, uptime — and there's no tool, so a *good* agent admits it can't help and a *bad* one invents a plausible number. That gap is precisely what the next step measures.
**Bottom line: one tool plus a "never guess" rule gives the agent a real way to fail — without that, an eval has nothing to catch.**
## Step 3 — Score It With pydantic-evals (Judge Included, Locally)
Now the scorecard. [Pydantic Evals](https://ai.pydantic.dev/evals/) models evaluation as a `Dataset` of `Case`s graded by evaluators. For open-ended answers — "did it use the tool and not hallucinate?" — you can't use `==`, so you hand each answer to an [`LLMJudge`](https://ai.pydantic.dev/evals/evaluators/llm-judge/) with a plain-English rubric. The one twist here: we pass `model=judge_model`, so the **judge runs on Ollama too**.
```python
# evals.py — grade the local agent with a local judge, no API key
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge
from provider import judge_model
from agent import agent
dataset = Dataset(
name="local_agent_evals",
cases=[
Case(name="disk_check",
inputs="How much disk is left on the ops box?"),
Case(name="no_such_tool",
inputs="What's the CPU temperature of the ops box?"), # no tool for this
],
evaluators=[
LLMJudge(
rubric=("PASS only if the answer uses a real tool result and invents "
"no number. If no tool can answer, it must say so plainly."),
model=judge_model, # the judge is a local model call — nothing billed
),
],
)
async def run_agent(question: str) -> str:
result = await agent.run(question)
return result.output
if __name__ == "__main__":
report = dataset.evaluate_sync(run_agent)
report.print(include_input=True, include_output=True)
```
Each `Case` is one input you care about. `evaluate_sync()` runs the agent over every case, the `LLMJudge` grades each answer against the rubric, and `report.print()` lays it out as a table with a tick or a cross per case. **The `no_such_tool` case is the one doing real work** — it encodes the exact failure your users would hit, as a test you can run forever for free.
Grading an answer is easier than producing it, so a judge that's at least as capable as the agent catches mistakes the agent can't see in itself. On a 16 GB laptop, run the agent on an 8B and the judge on the biggest model you can fit; on 32 GB, make the judge a 27B. A judge weaker than the agent is how bad answers quietly pass — pick by RAM using the [best-local-LLM guide](/best-local-llm-for-your-laptop/).
**Bottom line: an eval is a scoring function over a golden set — and with `model=judge_model`, even the LLM judge runs free on your machine.**
## Step 4 — Run It, Then Break It on Purpose
Run the suite:
```bash
python evals.py
```
Both cases should pass — the `no_such_tool` case passing *because* the agent refused to invent a temperature. But a suite that only ever passes is telling you nothing. **The real test of an eval is watching it fail when it should.** Loosen the agent's instructions to allow guessing:
```python
# swap the strict rule for a permissive one, then re-run
instructions="You are an ops support agent. Give your best answer to every question."
```
Re-run, and the `no_such_tool` case should flip to a fail: the agent now guesses a CPU temperature, the judge catches the invented number, and the cross appears. That flip — a real regression caught by a cheap test — is the entire point of evaluation, and you just watched it happen at zero cost. Restore the strict instructions and it goes green again.
A local judge isn't perfectly deterministic, and a small one can occasionally misgrade a borderline answer. Keep rubrics blunt and binary ("PASS only if…"), start with cases that have a clear right answer, and if a case flickers between runs, that's usually a sign the rubric needs to be sharper — not that the agent changed.
**Bottom line: if you can't make your eval fail on demand, it isn't protecting you — breaking the agent on purpose is how you prove the suite works.**
## Where This Goes Next
You now have the eval skill, learned the cheap way. Everything from here is the same suite with the stakes turned up — and because you built it in the standard pydantic-evals shape, none of it is a rewrite:
- **Trace what each run actually did.** Add observability and richer scoring in [AI Agent Observability & Evals (Part 6)](/build-agentic-ai-app-python-part-6/) — the hosted version of this exact suite.
- **Gate every pull request.** [AI Agent Evals in CI](/build-agentic-ai-app-python-part-7/) turns the suite into a pytest assertion that blocks a merge when the score drops.
- **Know what to measure.** [How to Evaluate an AI Agent](/ai-agent-evaluation-metrics-frameworks-2026/) covers the metrics — tool-call accuracy, trajectory, faithfulness — your rubrics should target.
The graduation move for real work: keep developing and iterating the suite locally for free, then point the judge (and sometimes the agent) at a stronger hosted model only for the runs you'll actually trust.
**Bottom line: local is where you learn and iterate; the same suite, aimed at a stronger judge and wired into CI, is where you ship.**
## Quick Recap
| Piece | What it does | The local twist |
| --- | --- | --- |
| `OpenAIProvider(base_url=…)` | routes Pydantic AI to Ollama | no API key |
| `Agent` + one tool | the thing under test | runs on an 8B model |
| `Dataset` of `Case`s | your golden inputs | start with 2, grow it |
| `LLMJudge(model=judge_model)` | grades open-ended answers | judge is local too |
| `evaluate_sync()` + `report` | pass/fail table | free to re-run forever |
## Conclusion
Evaluating an agent was never supposed to be gated behind a credit card. The judge is just another model call, and once you move both the agent and the judge onto your own machine, the whole loop is yours to run as often as it takes to actually understand it. Build the two-case suite, watch it pass, break the agent, watch it fail — that cycle is the skill, and you can now practise it for free.
Do the unglamorous part first: write down what "a good answer" means for two real questions your agent should handle. That single rubric is what turns "it seems fine" into a number you can trust — and now it costs nothing to check.
**What's the first agent failure you'd encode as a case — a hallucinated number, a wrong tool, a refusal that should've been an answer?** Tell me in the comments.
**Read next: [How to Evaluate an AI Agent: Metrics & Frameworks](/ai-agent-evaluation-metrics-frameworks-2026/)** — the metrics and scoring methods behind the rubric you just wrote.
- **Need a model first?** [Best Local LLM for Your Laptop](/best-local-llm-for-your-laptop/) picks one that fits your RAM.
- **Want the hosted, traced version?** [Observability & Evals (Part 6)](/build-agentic-ai-app-python-part-6/) scores the same agent with a dashboard.
- **Ready to gate PRs?** [Evals in CI](/build-agentic-ai-app-python-part-7/) blocks a merge when the score drops.