Evaluate an AI Agent on a Local LLM: Free, No API Key (2026)

Every “how to evaluate your agent” tutorial hits the same wall on line one: get an API key. The catch nobody mentions is that the eval judge is itself a model call, so a suite you run twenty times while learning bills you twenty times over. That’s a strange tax to pay just to understand how evaluation works.

You don’t have to. If the agent and the judge both run on a model on your own machine, the entire loop costs nothing — you can run it at 2am, a hundred times, offline, and never see a bill. This guide builds exactly that: a tiny tool-using agent on a local LLM, scored with pydantic-evals, judged by a second local model. Same workflow the production posts use, minus the API key.

Why Learn Evals on a Local Model First#

Evaluation is a skill you learn by running it over and over — change a prompt, re-score, see what moved. That feedback loop is where the intuition lives, and it’s exactly the loop a metered API makes you ration. Local models remove the meter.

There are three concrete wins to learning this way. Cost is zero, so you re-run without flinching — the single biggest thing that makes eval “click.” Everything stays on your machine, so you can throw real, messy, private inputs at the agent without sending them anywhere. And the part that surprises people: the judge is a model call too. An LLM-as-a-judgeUsing a language model to grade another model's output against a plain-English rubric, instead of an exact-match check. bills you once per case per run on a hosted API. Move it local and that cost vanishes, which is what makes a hundred practice runs actually free.

Bottom line: local models turn eval from a metered expense into an unlimited practice loop — the right place to build the skill.

Step 1 — Point Pydantic AI at Your Local Model#

Ollama serves every model at http://localhost:11434 with an OpenAI-compatible API, which means the whole Python ecosystem talks to it unchanged — you just change the base_url. In Pydantic AI, that’s an OpenAIProvider wrapped around a model name:

# provider.py — send every Pydantic AI call to your local Ollama server
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
 
ollama = OpenAIProvider(
    base_url="http://localhost:11434/v1",  # Ollama's OpenAI-compatible endpoint
    api_key="ollama",                       # required by the client, ignored by Ollama
)
 
agent_model = OpenAIChatModel("llama3.1:8b", provider=ollama)  # the agent under test
judge_model = OpenAIChatModel("qwen3.5:9b",  provider=ollama)  # a local judge, at least as strong

Two models, one provider. The agent model is the thing you’re testing; the judge model grades its answers in Step 3. They can be the same model, but a judge that’s at least as capable as the agent is the whole reason LLM-judging works — more on that shortly. Swap either tag for anything you’ve pulled that fits your RAM (ollama pull llama3.1:8b).

Bottom line: one base_url line reroutes every model call to your laptop — the single change that makes this whole tutorial free.

Step 2 — A Tiny Agent Worth Testing#

You need something with a way to fail before scoring means anything. So the agent gets exactly one real tool and a clear rule: answer only from tools, never guess. That setup builds in the classic agent failure — a confident, made-up answer — so the eval has a real target.

# agent.py — a minimal ops agent with one tool
from pydantic_ai import Agent
from provider import agent_model
 
agent = Agent(
    agent_model,
    instructions=(
        "You are an ops support agent. Answer only from tool results. "
        "If no tool can answer the question, say you don't have that data — "
        "never guess a number."
    ),
)
 
@agent.tool_plain
def disk_free(box: str) -> str:
    """Return the free disk space on a named server box."""
    inventory = {"ops": "42 GB free", "web": "11 GB free"}
    return inventory.get(box, "unknown box")

The agent can answer a disk question because disk_free exists. Ask it anything else about the box — CPU temperature, uptime — and there’s no tool, so a good agent admits it can’t help and a bad one invents a plausible number. That gap is precisely what the next step measures.

Bottom line: one tool plus a “never guess” rule gives the agent a real way to fail — without that, an eval has nothing to catch.

Step 3 — Score It With pydantic-evals (Judge Included, Locally)#

Now the scorecard. Pydantic Evals models evaluation as a Dataset of Cases graded by evaluators. For open-ended answers — “did it use the tool and not hallucinate?” — you can’t use ==, so you hand each answer to an LLMJudge with a plain-English rubric. The one twist here: we pass model=judge_model, so the judge runs on Ollama too.

# evals.py — grade the local agent with a local judge, no API key
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge
from provider import judge_model
from agent import agent
 
dataset = Dataset(
    name="local_agent_evals",
    cases=[
        Case(name="disk_check",
             inputs="How much disk is left on the ops box?"),
        Case(name="no_such_tool",
             inputs="What's the CPU temperature of the ops box?"),  # no tool for this
    ],
    evaluators=[
        LLMJudge(
            rubric=("PASS only if the answer uses a real tool result and invents "
                    "no number. If no tool can answer, it must say so plainly."),
            model=judge_model,   # the judge is a local model call — nothing billed
        ),
    ],
)
 
async def run_agent(question: str) -> str:
    result = await agent.run(question)
    return result.output
 
if __name__ == "__main__":
    report = dataset.evaluate_sync(run_agent)
    report.print(include_input=True, include_output=True)

Each Case is one input you care about. evaluate_sync() runs the agent over every case, the LLMJudge grades each answer against the rubric, and report.print() lays it out as a table with a tick or a cross per case. The no_such_tool case is the one doing real work — it encodes the exact failure your users would hit, as a test you can run forever for free.

Bottom line: an eval is a scoring function over a golden set — and with model=judge_model, even the LLM judge runs free on your machine.

Step 4 — Run It, Then Break It on Purpose#

Run the suite:

python evals.py

Both cases should pass — the no_such_tool case passing because the agent refused to invent a temperature. But a suite that only ever passes is telling you nothing. The real test of an eval is watching it fail when it should. Loosen the agent’s instructions to allow guessing:

# swap the strict rule for a permissive one, then re-run
instructions="You are an ops support agent. Give your best answer to every question."

Re-run, and the no_such_tool case should flip to a fail: the agent now guesses a CPU temperature, the judge catches the invented number, and the cross appears. That flip — a real regression caught by a cheap test — is the entire point of evaluation, and you just watched it happen at zero cost. Restore the strict instructions and it goes green again.

Bottom line: if you can’t make your eval fail on demand, it isn’t protecting you — breaking the agent on purpose is how you prove the suite works.

Where This Goes Next#

You now have the eval skill, learned the cheap way. Everything from here is the same suite with the stakes turned up — and because you built it in the standard pydantic-evals shape, none of it is a rewrite:

• Trace what each run actually did. Add observability and richer scoring in AI Agent Observability & Evals (Part 6) — the hosted version of this exact suite.
• Gate every pull request. AI Agent Evals in CI turns the suite into a pytest assertion that blocks a merge when the score drops.
• Know what to measure. How to Evaluate an AI Agent covers the metrics — tool-call accuracy, trajectory, faithfulness — your rubrics should target.

The graduation move for real work: keep developing and iterating the suite locally for free, then point the judge (and sometimes the agent) at a stronger hosted model only for the runs you’ll actually trust.

Bottom line: local is where you learn and iterate; the same suite, aimed at a stronger judge and wired into CI, is where you ship.

Quick Recap#

Conclusion#

Evaluating an agent was never supposed to be gated behind a credit card. The judge is just another model call, and once you move both the agent and the judge onto your own machine, the whole loop is yours to run as often as it takes to actually understand it. Build the two-case suite, watch it pass, break the agent, watch it fail — that cycle is the skill, and you can now practise it for free.

Do the unglamorous part first: write down what “a good answer” means for two real questions your agent should handle. That single rubric is what turns “it seems fine” into a number you can trust — and now it costs nothing to check.

What’s the first agent failure you’d encode as a case — a hallucinated number, a wrong tool, a refusal that should’ve been an answer? Tell me in the comments.

Read next: How to Evaluate an AI Agent: Metrics & Frameworks — the metrics and scoring methods behind the rubric you just wrote.