# AI Agent Observability & Evals in Python (Part 6)
> Add AI agent observability and evals in Python — trace every agent run with Pydantic Logfire and score it against a test set using pydantic-evals.
*Source: https://www.infowok.com/build-agentic-ai-app-python-part-6/ · Sukhveer Kaur · Published June 18, 2026 · Updated July 6, 2026*
---
> **Series: Agentic AI in Python — Zero to Production**
>
> This is **Part 6** — adding AI agent observability and evals. The story so far:
>
> - **Part 1:** [A local LangGraph agent](/build-agentic-ai-app-python-part-1/) with tools and a checkpointer
> - **Part 4:** [Long-term memory](/build-agentic-ai-app-python-part-4/) across threads
> - **Part 5:** [Real tools via an MCP client](/build-agentic-ai-app-python-part-5/) — the agent can now act
>
> New here? You'll need the Part 5 agent that can call tools — this post puts a microscope and a scorecard on it.
By the end of Part 5 your agent could think, remember, and act. There's one thing it still can't do: tell you whether any of that is actually *good*. Right now you find out it broke the way everyone finds out — a user complains. This part fixes that with two tools that work together. **AI agent observability** shows you exactly what every run did. **Evals** score the agent against a fixed test set, so a bad change can't sneak past you.
I put these last on purpose. Adding capability feels like progress; measuring it feels like overhead. That holds right up until the day a prompt tweak quietly breaks tool-calling and you have no way to notice. By the end of this post you'll trace every run in a real dashboard and run a pass/fail suite you can wire into CI. Three short steps. First, what these two words actually mean.
- The tool-calling agent from [Part 5](/build-agentic-ai-app-python-part-5/) — this post instruments and scores it
- You've shipped at least Part 1's agent and can run it end to end
- An LLM API key (the eval judge is itself a model call and needs one)
- **Observability answers "what did my agent do?"; evals answer "is it any good?"** — you need both, and neither changes the agent, only instruments it.
- **Two lines of Logfire** (configure + instrument) turn every run, model call, and tool call into a trace; LangGraph emits the same OpenTelemetry spans.
- **Watch three numbers per run** — latency, cost (from token counts), and which tools were called — to turn "it feels slow" into a specific span you can fix.
- **Score the agent with pydantic-evals:** a `Dataset` of `Case`s graded by an `LLMJudge` rubric, run on every change and wired into CI so a regression can't merge.
## What We're Building
AI agent observability and evals answer two different questions, and you need both. **Observability** (seeing the internal state of a running system from the outside) answers *"what did my agent just do?"* — every model call, tool call, token, and millisecond. **Evals** (evaluations — scoring outputs against expected behaviour) answer *"is my agent any good?"* They check a set of cases you care about, not just the one you happened to test by hand.
The diagram shows the split. The same agent feeds two pipelines. One emits **OpenTelemetry** spans (timed records of each step) to a dashboard, so you can watch live traffic. The other runs the agent over test cases and scores the results. **You're not changing the agent — you're putting instruments around it.** The observability half is the "see it" lane; the evals half is the "score it" lane.
## What You Need
Everything here builds on the agent from Part 5 — you don't write any new agent logic, just wrap what you have.
- [ ] The Part 5 agent that can call MCP tools (LangGraph or Pydantic AI)
- [ ] `pip install -U logfire pydantic-evals` (both current in June 2026)
- [ ] A free [Pydantic Logfire](https://pydantic.dev/logfire) account for a write token, or any OpenTelemetry backend
- [ ] An API key for the model your judge will use (the evals step needs one)
If you'd rather not sign up for anything, Logfire is built on OpenTelemetry, so you can point the same spans at a self-hosted backend like Langfuse instead — the instrumentation code below doesn't change.
The `LLMJudge` below is itself a model call, so a growing suite costs real money. To follow the evals half of this post for free, point both the agent and the judge at a local model with [Ollama](/best-local-llm-for-your-laptop/): the [local-LLM eval walkthrough](/evaluate-ai-agent-local-llm-free/) shows the one `base_url` change that removes the API-key requirement entirely.
## Step 1 — Trace Every Run with Logfire
Here's the early win: two lines turn an invisible agent into one you can watch. [Pydantic Logfire](https://logfire.pydantic.dev/docs/ai-observability/) is an observability platform built on OpenTelemetry, and Pydantic AI has native support for it. You configure it once, at startup:
```python
# observability.py — turn tracing on once
import os
import logfire
logfire.configure(token=os.environ["LOGFIRE_TOKEN"])
logfire.instrument_pydantic_ai() # every agent run is now a trace
```
That's it. `logfire.configure()` sets up the exporter, and **`instrument_pydantic_ai()` makes every agent run, model call, and tool call show up as a span** — no per-call logging, no print statements. Run your Part 5 agent once and the dashboard fills with a tree you can expand: the run, the model's thinking, the `disk_usage` tool call, the result.
On the **LangGraph** side there's no separate API to learn. LangChain already emits OpenTelemetry traces, so set `LANGSMITH_OTEL_ENABLED=true` and Logfire (or any OTel backend) picks up the same spans. One dashboard, whichever framework you built on.
The token lives in an environment variable, never in the file — the same rule we used for the MCP server's auth in Part 5.
## Step 2 — AI Agent Observability in Practice
A trace is only useful if you know what to look at. This is where **AI agent observability** earns its keep: three numbers tell you most of what you need on any given run.
- **Latency** — how long the run took, and which step ate it. A slow agent is almost always one slow tool call, and the span tree points straight at it.
- **Cost** — Logfire reads token counts off each model call and totals them. You see the price per run, instead of guessing at the end of the month.
- **Tool calls** — every tool the model chose, with its arguments and result, so you can confirm it called `disk_usage` and not something it invented.
In my own runs, a healthy tool-backed answer round-trips in roughly 1–2 seconds and costs a fraction of a cent. When a run suddenly takes eight seconds, the trace shows the cause: a retry against an unreachable server, not a slow model. **The point of tracing is to turn "it feels slow" into a specific span you can fix.** That alone is worth the two lines of setup — but it still doesn't tell you whether the *answers* are right. For that, you need evals.
## Step 3 — Score the Agent with pydantic-evals
Observability shows you one run at a time. Evals run a whole batch and grade them, so you can change a prompt and instantly see if anything regressed (got worse than before). [Pydantic Evals](https://ai.pydantic.dev/evals/) models this as a `Dataset` of `Case` objects scored by evaluators:
```python
# evals.py — score the agent against a fixed test set
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge
dataset = Dataset(
cases=[
Case(name="disk_check",
inputs="How much disk is left on the ops box?"),
Case(name="no_such_tool",
inputs="What's the CPU temperature?"), # there is no tool for this
],
evaluators=[
LLMJudge(
rubric=("Answer uses a real tool result and never invents a number; "
"if no tool can answer, it says so plainly."),
),
],
)
async def run_agent(question: str) -> str:
result = await agent.run(question) # the Part 5 agent
return result.output
report = dataset.evaluate_sync(run_agent)
report.print(include_input=True, include_output=True)
```
Each `Case` is one input you care about. The [`LLMJudge`](https://ai.pydantic.dev/evals/evaluators/llm-judge/) evaluator uses a model to grade each answer against your plain-English rubric. That's ideal for qualities you can't check with `==`, like "didn't hallucinate a number." Then `evaluate_sync()` runs the agent over every case, and `report.print()` prints a table with a green tick per passed case.
**The second case is the one that matters.** It asks for data no tool provides, and a good agent says "I can't find that" instead of guessing. That's exactly the failure your users would hit, encoded as a test you run on every change.
The suite is meant to grow. Every real bug your agent ships becomes a new `Case` so it can never come back, and you can attach a different rubric to each case when one answer needs stricter grading than another. A handful of cases on day one is fine — what matters is that the set only ever gets bigger.
You can't improve what you can't see. Add **tracing before you tune prompts** — most "the agent is dumb" problems turn out to be a tool quietly returning junk, which only a trace reveals.
## Testing It + Common Errors
Run `python evals.py` and read the table: both cases should pass, with the CPU case passing *because* the agent refused to invent a temperature. Now prove the suite works by breaking the agent on purpose — loosen the system prompt to allow guessing, rerun, and watch the `no_such_tool` case flip to a fail. A suite that never fails isn't protecting you.
Three errors cost me the most time setting this up:
> **Common mistake:** calling `instrument_pydantic_ai()` *after* the agent already ran. Instrumentation only catches runs that happen after configuration, so put both lines at import time, before any agent call.
The other two are quieter. A **missing `LOGFIRE_TOKEN`** makes `configure()` fall back to local-only mode. Traces work in your terminal but never reach the dashboard, so check the token first if the web view stays empty. And an **LLMJudge with no API key** fails every case with an auth error that looks like a scoring failure. The judge is itself a model call, so it needs its own key.
## What to Build Next — and the Series Wrap
Your agent is now complete in the way that counts: it can act (Part 5), remember (Part 4), run as a service (Part 2), and — finally — be *measured*. The natural next move is to run the eval suite in CI so a pull request that drops the score can't merge. Because `evaluate_sync()` returns a report object, you can assert on its pass rate in a normal test and fail the build when it slips.
I'd build that gate before adding a single new tool. The whole series has pointed here: capability you can't measure is capability that quietly rots, and a ten-line eval suite is the cheapest insurance you'll ever write for an agent.
So, to close the series — drop your answer in the comments: now that you can see *and* score your agent, what's the first regression you'd want the suite to catch? Tell me, and I'll turn the best answers into a CI walkthrough.
**The full series — [Agentic AI in Python: Zero to Production](/agentic-ai-python-series/):**
1. [Part 1 — Tools, StateGraph & Memory](/build-agentic-ai-app-python-part-1/)
2. [Part 2 — FastAPI, Docker & Deploy](/build-agentic-ai-app-python-part-2/)
3. [Part 3 — Multi-Agent Systems](/build-agentic-ai-app-python-part-3/)
4. [Part 4 — AI Agent Memory](/build-agentic-ai-app-python-part-4/)
5. [Part 5 — MCP Client & Real Tools](/build-agentic-ai-app-python-part-5/)
6. **Part 6 — Observability & Evals** — you're here
Related: [Pydantic AI Tutorial — Type-Safe AI Agents in Python](/pydantic-ai-tutorial-type-safe-agents-python/)
- **Need the agent under test?** [Part 5](/build-agentic-ai-app-python-part-5/) gives it real tools to instrument.
- **Want type-safe agents?** The [Pydantic AI tutorial](/pydantic-ai-tutorial-type-safe-agents-python/) pairs naturally with pydantic-evals.
- **Newer to agents?** The [build-from-scratch series](/ai-agents-from-scratch-python-part-0/) explains every layer this series builds on.