Evaluating AI Systems: Testing Agents at Scale

Computational Analysis of Social Complexity

Fall 2026, Spencer Lyon

Prerequisites

• Pydantic AI Agents and Tools

• Python programming fundamentals

• Basic understanding of testing concepts

Outcomes

• Understand why systematic evaluation is critical for AI systems

• Identify when and what to evaluate in AI agents

• Implement deterministic and LLM-based evaluators

• Design evaluation datasets using code-first approaches

• Analyze and compare evaluation results across experiments

• Connect evaluation practices to production deployment concerns

References

• Pydantic AI Evals Documentation

• Pydantic Evals API Reference

• Testing practices from software engineering

Introduction: The AI Testing Problem¶

Why Testing AI is Different¶

• Traditional software testing: deterministic inputs → deterministic outputs

  • If add(2, 3) returns 5 once, it always will

  • Clear pass/fail criteria

• AI systems: same input → potentially different outputs

  • Ask an agent “What’s the capital of France?” twice, might get:

    • “The capital of France is Paris.”

    • “Paris is France’s capital city.”

    • “France’s capital is Paris, known for the Eiffel Tower.”

  • All correct, but different!

• Key challenge: How do we test something that’s non-deterministic?

Motivating Scenario: The Support Bot Problem¶

• You’ve built a customer support agent using Pydantic AI

• It handles questions about product returns, shipping, and account issues

• In development, it seems to work great on the examples you tried

• You deploy to production and...

  • Sometimes gives outdated return policy information

  • Occasionally “hallucinates” shipping partners that don’t exist

  • Has trouble with edge cases you didn’t think to test

• Question: How could systematic evaluation have caught these issues?

What We’ll Learn¶

• We’ll explore Pydantic Evals, a framework for testing AI systems

• Three main components:

  1. Datasets: Collections of test scenarios

  2. Evaluators: Scoring mechanisms to check outputs

  3. Experiments: Runs that combine datasets and evaluators

• Think of it like unit testing for AI:

  • Cases + Evaluators = individual unit tests

  • Datasets = test suites

  • Experiments = running your entire test suite

Setup¶

Install and Import Dependencies¶

We’ll need Pydantic AI and the evals module:

Configure API Keys¶

Make sure you have your Anthropic API key set:

Core Concepts: The Evaluation Framework¶

Structure: Cases, Datasets, and Experiments¶

Cases: Individual test scenarios

• Like a single unit test

• Contains:

  • inputs: Data you pass to the agent

  • expected_output: (Optional) What you expect back

  • metadata: (Optional) Context about this test

  • evaluators: (Optional) Case-specific checks

Datasets: Collections of cases

• Like a test suite

• Groups related test scenarios

• Can have dataset-level evaluators that apply to all cases

Experiments: Evaluation runs

• Like running pytest or julia test

• Executes your task function on all cases

• Applies evaluators to score outputs

• Generates reports with results

Example: Simple Support Bot Dataset¶

Let’s build intuition with a concrete example.

Suppose we want to test our support bot’s ability to identify the user’s intent:

Note the pattern:

• Each case tests one scenario

• We specify what we expect

• Cases are typed: Dataset[str, str] means string inputs → string outputs

Exercise 1: Design Your Own Cases¶

Scenario: You’re building a sentiment analysis agent for product reviews.

Task: Following the pattern above, create a Dataset[str, str] with 3-5 test cases for an agent that should classify product reviews as:

• “positive”

• “negative”

• “neutral”

Think about:

• What review texts represent typical positive/negative/neutral cases?

• What are edge cases? (e.g., mixed sentiments, sarcasm)

• Make sure each Case has:

  • A descriptive name

  • inputs (the review text)

  • expected_output (the sentiment label)

Evaluators: How to Score Outputs¶

Two Types of Evaluation¶

Deterministic Evaluators: Code-based checks

• Exact matches

• Type checking

• Format validation (email, phone number, URL)

• PII detection

• Regular expression matching

Non-Deterministic Evaluators: Subjective assessment

• LLM as judge

• Human evaluation

• Quality metrics (accuracy, relevance, helpfulness)

Built-in Evaluators¶

Pydantic Evals provides several ready-made evaluators:

1. Exact Matching¶

2. Type Checking¶

3. Membership/Contains¶

4. Performance Constraints¶

Example: Adding Evaluators to Dataset¶

LLM as Judge: When Correctness is Subjective¶

• Sometimes there’s no single “correct” answer

• Example: “Write a friendly response to this complaint”

  • Many valid responses exist

  • Hard to check with deterministic rules

• Solution: Use another LLM to evaluate

LLMJudge Evaluator:¶

How it works:

1. Your agent generates an output

2. LLMJudge sends that output + rubric to an LLM

3. LLM scores the output based on the rubric

4. Score is recorded in the evaluation report

Custom Evaluators: Domain-Specific Checks¶

You can create custom evaluators by subclassing Evaluator:

Key points:

• Implement evaluate method (can be sync or async)

• Access inputs/outputs through EvaluatorContext

• Return EvaluationReason with value and explanation

Exercise 2: Design Evaluators¶

For the sentiment analysis agent from Exercise 1:

1. What deterministic evaluators would you use?

   • Think about checking exact label matches, valid sentiment values

2. What subjective aspects might need LLM evaluation?

   • Example: When a review has mixed sentiment, is the chosen label reasonable?

   • What would your LLMJudge rubric say?

3. Design one custom evaluator for a domain-specific check

   • Example: “Output should be all lowercase” or “Response time should be fast”

Running Evaluations: From Code to Reports¶

The Evaluation Loop¶

Step 1: Define your task function¶

Step 2: Run evaluation¶

Step 3: Analyze results¶

Visualizing Results with Logfire¶

While the printed reports are useful, Pydantic Logfire provides a web UI for visualizing and analyzing evaluation results over time.

Why use Logfire?

• Interactive dashboards for evaluation metrics

• Trace exploration for debugging failed cases

• Track trends across multiple evaluation runs

• Team collaboration and sharing results

Let’s configure Logfire integration:

Note: With Logfire configured, all subsequent evaluations will automatically send their results to the Logfire web UI. You can then:

• View evaluation reports in an interactive dashboard

• Explore individual traces and spans

• Compare results across multiple runs

• Set up alerts for failing evaluations

Visit logfire.pydantic.dev to view your evaluation results.

Understanding Evaluation Results¶

Comparing Experiments: Tracking Improvements¶

Let’s modify our agent and compare results:

Exercise 3: Run Your First Evaluation¶

Using the sentiment analysis agent you designed in Exercise 1:

1. Create an agent that performs sentiment classification

2. Use your dataset from Exercise 1 (add more cases if needed)

3. Add 2-3 evaluators (mix of built-in and custom from Exercise 2)

4. Run the evaluation using await dataset.evaluate(task=your_function)

5. Interpret the results:

   • Which cases passed/failed?

   • What patterns do you notice?

   • What would you improve about the agent or the test cases?

Advanced Topics: Span-Based Evaluation and Dataset Generation¶

Span-Based Evaluation: Evaluating the Process, Not Just the Output¶

The Problem: Sometimes the final answer is correct, but the how matters

• Example: Math problem solving

  • Output: “42” ✓ Correct!

  • But did the agent:

    • Use the right formula?

    • Show its work?

    • Make calculation errors that happened to cancel out?

Spans: Execution traces from OpenTelemetry

• Capture what the agent did internally

• Tool calls made

• LLM requests and responses

• Intermediate reasoning steps

HasMatchingSpan Evaluator:¶

Why this matters:

• Catches “lucky guesses” where agent gets answer right for wrong reasons

• Validates agent is following intended reasoning process

• Useful for multi-step tasks where process correctness matters

Generating Datasets with LLMs¶

The Challenge: Creating comprehensive test datasets is tedious

• Need diverse inputs covering edge cases

• Need correct expected outputs

• Manual creation doesn’t scale

Solution: Use an LLM to generate test cases

How it works:

1. You specify the dataset type with input/output types (e.g., Dataset[str, str])

2. LLM generates diverse test scenarios based on your instructions

3. Returns properly structured Dataset object with generated cases

4. Can optionally save to file for version control (using path parameter)

Best practices:

• Review generated cases before using

• Mix generated and hand-crafted cases

• Regenerate periodically to expand coverage

• Use extra_instructions to guide the LLM toward specific edge cases

Evaluating RAG Systems: A Two-Stage Challenge¶

What is RAG?¶

• RAG = Retrieval-Augmented Generation

• Common pattern for AI agents that need to answer questions about documents/data

• Two stages:

  1. Retrieval: Find relevant documents/passages from knowledge base

  2. Generation: Use retrieved context to generate answer

Why RAG Evaluation is Different¶

• Traditional evaluation: just check the final answer

• RAG evaluation: need to check both stages

  • Is the retrieval finding the right documents?

  • Is the generation using those documents correctly?

• Failure can happen at either stage (or both!)

Example Failure Modes:

• ✓ Retrieval works, ✗ Generation fails: Found right docs, but hallucinated answer

• ✗ Retrieval fails, ✓ Generation works: Couldn’t find relevant docs, so generated plausible but wrong answer

• ✗ Both fail: Retrieved irrelevant docs and made up information

RAG Evaluation Metrics¶

Retrieval Metrics (Is the retrieval working?)

Generation Metrics (Is the generation working?)

Best Practices for RAG Evaluation¶

1. Evaluate Stages Independently

Create separate datasets for retrieval and generation to pinpoint where failures occur.

Why separate?

• Pinpoints where failures occur

• Can optimize retrieval and generation independently

• Clearer diagnosis: “Our retrieval is great but generation hallucinates” vs “Both need work”

Exercise: Design RAG Evaluation¶

Scenario: You’re building a RAG system that answers questions about a company’s internal documentation.

Tasks:

1. Identify failure modes: What are 3 ways this RAG system could fail?

2. Design retrieval tests: What cases would test if retrieval is working?

   • What queries should always retrieve specific documents?

   • What edge cases might break retrieval?

3. Design generation tests: Assuming perfect retrieval, how do you test generation?

   • What makes a “good” answer?

   • How do you detect hallucinations?

4. Create evaluation pipeline: Sketch code for evaluating both stages

   • What metrics would you track?

   • How would you report results?

Integration and Best Practices¶

When to Use Each Evaluation Type¶

Deterministic Evaluators when:

• Clear right/wrong answers exist

• Output format matters (structured data)

• Security/safety constraints (no PII leakage)

• Performance requirements (latency, cost)

LLM as Judge when:

• Multiple valid answers exist

• Quality is subjective (helpfulness, tone)

• Semantic equivalence matters (“Paris” vs “The capital of France is Paris”)

Span-Based Evaluation when:

• Process correctness matters, not just output

• Multi-step reasoning needs validation

• Tool usage patterns are important

• Debugging complex agent behaviors

Tips for Effective Evaluation¶

Start Small, Grow Gradually:

• Begin with 5-10 cases covering main scenarios

• Add cases as you discover failures

• Prioritize cases that would impact users most

Balance Coverage and Maintainability:

• Don’t try to test everything

• Focus on high-risk or high-value scenarios

• Remove redundant cases

Make Evaluators Specific and Clear:

• Good rubric: “Score 0-10 on factual accuracy. Check claims against provided context.”

• Bad rubric: “Score the quality of the response.”

Version Control Your Datasets:

• Store datasets as YAML/JSON in git

• Track changes over time

• Share across team

Automate Where Possible:

• Run evals in CI/CD pipeline

• Block deploys if pass rate drops

• Generate alerts for regressions

Exercise: Evaluation Strategy Design¶

Choose one scenario:

1. E-commerce support agent: Handles returns, shipping, account questions

2. Code review agent: Reviews pull requests, suggests improvements

3. Data analysis agent: Answers questions about datasets using pandas

For your chosen scenario:

1. Design an evaluation strategy:

   • What’s in your test dataset? (10+ cases)

   • What evaluators would you use?

   • How would you measure success?

2. Describe your development workflow:

   • When do you run evals?

   • What metrics do you track?

   • How do you decide when to deploy?

3. Plan for production:

   • How do you handle failures?

   • When do you update your dataset?

   • What triggers re-evaluation?

Connections to Course Themes¶

Game Theory and Evaluation¶

• Agent alignment: Evaluation as mechanism design

  • You design rubrics (rules) to incentivize desired behaviors

  • LLM as Judge is like a referee in a game

  • Pass/fail thresholds create strategic constraints

• Adversarial evaluation: Red team vs Blue team

  • Attackers try to make agent fail (jailbreaking, prompt injection)

  • Defenders build evals that catch these attacks

  • Nash equilibrium between robustness and capability

Network Effects in AI Systems¶

• Evaluation datasets as networks:

  • Cases can have dependencies (one builds on another)

  • Failures can cascade (if base functionality breaks, many cases fail)

  • Coverage metrics: are there “clusters” of untested scenarios?

• Agent-to-agent evaluation:

  • Multi-agent systems need coordinated evaluation

  • Agent A’s outputs become Agent B’s inputs

  • Network of evals reflects agent interaction topology

Emergence in Complex AI Systems¶

• Emergent behaviors in multi-step agents:

  • Simple agent + simple tools → complex behaviors

  • Can’t predict all outcomes from components

  • Evaluation discovers emergent capabilities (and failures)

• Evaluation as an ABM simulation:

  • Each test case is like running the simulation once

  • Aggregate results reveal patterns

  • Edge cases show boundary conditions of agent “behavior space”

Summary and Key Takeaways¶

What We Learned¶

1. Why Evals Matter

   • AI systems are non-deterministic

   • Systematic testing catches issues before production

   • Evals enable confident iteration and deployment

2. Core Framework: Pydantic Evals

   • Cases: individual test scenarios

   • Datasets: collections of cases

   • Evaluators: scoring mechanisms (deterministic, LLM, custom)

   • Experiments: runs that generate reports

3. Evaluation Strategies

   • Deterministic checks for clear criteria

   • LLM as Judge for subjective quality

   • Span-based evaluation for process correctness

   • Custom evaluators for domain-specific needs

   • RAG-specific metrics for retrieval + generation

4. Best Practices

   • Start small, iterate based on failures

   • Balance coverage with maintainability

   • Version control datasets and track metrics

   • Integrate into development and deployment workflows

Looking Forward¶

• Evaluations are “an emerging art/science”

• No single “right” approach exists

• Adapt techniques to your domain and constraints

• Key principle: Test systematically, deploy confidently

Final Exercise: Reflection¶

Think about an AI agent you might build:

1. What are the top 3 risks or failure modes?

2. How would you design evals to catch those?

3. What would “success” look like quantitatively?

Additional Resources¶

• Pydantic AI Evals Documentation

• Dataset API Reference

• Evaluators API Reference

• Pydantic Logfire for Visualization