Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Testing whether an AI agent finishes real tasks correctly, safely and every time: check the end state, score the tool calls and steps, track cost, and repeat each task to measure pass^k.
In short
Agent evaluation is how you test whether an AI agent finishes real tasks correctly, safely and every time, not just once. You run the agent on a fixed set of tasks in a safe copy of its world. Code checks the end result. You score the steps it took: the tools it used, the values it passed, its errors and its cost. Then you repeat each task several times. How often it succeeds on every try is a number called pass^k.
In short
Agent evaluation is how you test whether an AI agent finishes real tasks correctly, safely and every time, not just once. You run the agent on a fixed set of tasks in a safe copy of its world. Code checks the end result. You score the steps it took: the tools it used, the values it passed, its errors and its cost. Then you repeat each task several times. How often it succeeds on every try is a number called pass^k.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Agent Evaluation across a run of lessons, not one page. These 4 alone are about 275 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
One end-to-end score tells you something broke, not what. Split RAG and agent quality into retrieval, generation, tool, and trajectory evals that localize failures and score the path, not just the answer.
A model's recall falls as its window fills, and worst from the middle. Measure the fill level where your agent starts to rot, and re-run it per model version.
I took the features chapter's trained model and looked inside the saved file. Pickle made a 202,700-byte file. 95.9 percent of it was the raw bytes of the model's numpy arrays, and the rest named 16 classes from numpy and scikit-learn. Loading a pickle does not only read data: it imports those names and calls them. To show this safely, I built a harmless test file that writes one temp file and prints one line, and I read its call in the disassembly before loading it. Then I ran two scanners. By default, picklescan called 15 safe names in the real model suspicious and found no dangerous name in either file, and modelscan found no issue in either. Checked by name only, the obvious attack names, such as os system and builtins exec, are flagged on both lists (picklescan dangerous, modelscan CRITICAL). The reloaded model predicted identically to the bit.
Teams abandon error analysis in a text editor and record it as a conclusion about the method. Three behaviours fix it, one feature ruins the measurement, and the whole thing is a hundred lines of Python.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
An AI agent is a program that lets a large language model (LLM), the kind of model behind ChatGPT, decide what to do next. It calls tools, reads the results and decides again until the task is done. A tool is a function the agent may run, such as "find order" or "issue refund".
Evaluation means testing with a fixed set of examples and a score, so you can compare one version with another. Testing a chatbot is mostly about its answer. Testing an agent is harder, for three reasons.
First, the agent acts. It changes things in the world, such as a refund in a database. So you must check what changed, not only what it said. Second, it takes many steps, and a right answer can hide a bad path. Third, it is not consistent. LLMs give different outputs on different runs, so an agent that solves a task today may fail the same task tomorrow.
Agent evaluation answers three questions. Did it finish the task? Did it get there in a safe and sensible way? And will it do that every time?
My evaluating RAG and agents lesson splits an agent's quality into four layers. One total score tells you that something broke. Separate scores tell you where.
On top of these, track cost and speed for every task. The same lesson shows two agents that both give the right answer. One uses 3 tool calls. The other uses 41. A score that checks only correctness rates them the same, but the second one burns money and makes users wait.
There are two ways to grade an agent, and good test suites use both.
The first checks the end state. After the run, compare the world with the state it should be in: is there exactly one refund row, for the right order and amount? This is plain code, cheap and fair. It also accepts different correct paths. tau-bench, a benchmark (a fixed public test that many teams run so results can be compared) from the company Sierra, grades this way: it "compares the database state at the end of a conversation with the annotated goal state." The annotated goal state is the end state a person wrote down as correct.
The second checks the path, also called the trajectory: the list of steps the agent took. It catches what the end state hides. An agent that called a paid tool forty times, skipped an error and got lucky still leaves the right row behind. A path check flags it. The lesson's rubric, a scoring guide, checks three things: progress (did each step move toward the goal), efficiency (how close to the fewest steps), and recovery (were errors handled).
Some path checks are plain code, such as counting calls or checking that identity was verified before a refund. Others need a judge. Often the judge is another LLM, called an LLM-as-judge. Check that judge against answers that people marked by hand before you trust it. My judge failure modes lesson shows how judges go wrong.
Run one task once and you learn little, because the agent might have been lucky. So run each task several times. The tau-bench paper, by Shunyu Yao and others in 2024, made a number for this called pass^k, read "pass hat k". It is "the chance that all k i.i.d. task trials are successful, averaged across tasks." i.i.d. means each try is independent and run the same way.
Do not mix it up with pass@k, read "pass at k". That is the chance that at least one of k tries succeeds. pass@k goes up as k grows. pass^k goes down. My agent reliability math lesson works both out. An agent that succeeds half the time scores 99.6% on pass@8. An agent that succeeds 80% of the time scores only 17% on pass^8. A user who uses your agent every day feels pass^k, not pass@k.
Here is how to compute it. Run each task n times and count c successes. For one task, pass^k is the number of ways to choose k tries from the c successes, divided by the number of ways to choose k tries from all n. Average that over all tasks. The code below does it for 5 made-up tasks, each run 8 times.
pass_k.py
from math import comb
def pass_hat_k(n: int, c: int, k: int) -> float:
"""tau-bench's pass^k for one task: the chance that k tries, drawn from n tries with c successes, ALL succeed."""
return comb(c, k) / comb(n, k)
# 8 tries per task; how many of the 8 succeeded, for 5 tasks
successes = [8, 6, 4, 7, 2]
for k in (1, 2, 4, 8):
score = sum(pass_hat_k(8, c, k) for c in successes) / len(successes)
print(f"pass^{k} = {score:.2f}")Output, captured 2026-10-18
pass^1 = 0.68
pass^2 = 0.51
pass^4 = 0.35
pass^8 = 0.20Look at how fast it falls. One try looks like 68%. Ask for 8 out of 8, and only the one task that never failed counts, so the score drops to 20%. The real paper saw the same shape. Its table gives gpt-4o, a strong model, a pass^1 of 61.2 in retail and 35.2 in its airline tasks, and its abstract says pass^8 fell below 25% in retail.
Here is a plan that works for most agents, such as a customer support agent.
Offline tests, run before launch on saved tasks, are not the end. After launch, watch real results too: how many chats were solved without a person, how many were handed to a person, and what customers said. Anthropic's guide notes that for customer support, "success can be clearly measured through user-defined resolutions." My offline vs online evals lesson explains how the two kinds of test fit together.
LLM evaluation tests a model's answers. RAG evaluation tests a system that first searches documents and then answers, and splits the blame between the search and the answer. Agent evaluation includes both, and adds three things: tool calls, the path of steps, and the end state of the world the agent changed.
The full method, with labelled examples, judges you can trust and simulation at scale, is in my evaluating RAG and agents lesson. For the interview side, see the customer support agent interview guide and the AI agent guide.
tau-bench (Sierra)
A 2024 benchmark of agents talking with simulated users in retail and airline tasks. It grades the final database state and introduced pass^k.
Anthropic
Building effective agents (2024) recommends extensive testing in sandboxed environments and names customer support as a use case where success can be measured by resolutions.
LangSmith
LangChain Inc's tracing and evaluation product. It records each step of an agent run, which is the raw material for path checks.
OpenAI
A common choice for the user simulator or the judge model in an evaluation harness. The model under test and the judge should be checked separately.
Agent evaluation is testing whether an AI agent completes real tasks correctly, safely and consistently. You run it on a fixed task set in a sandbox, check the end state in code, score its tool calls and steps, track cost, and repeat each task to measure pass^k.
Collect real tasks with a known correct end state, run the agent in a sandbox that resets each time, and let an LLM play the user for multi-turn tasks. Check the end state in code, score tool calls and the path, log cost and time, and run each task several times to report pass^k.
Task success (the end state is correct), tool-call accuracy (right tool, valid arguments, errors handled), path quality (progress, efficiency, recovery), consistency (pass^k), and cost and latency per task. After launch, add resolution rate, handoff rate to people, and customer feedback.
pass@k is the chance that at least one of k tries succeeds, so it rises with k. pass^k is the chance that all k tries succeed, so it falls with k. pass@k measures what an agent can do with retries and a checker; pass^k measures whether users can rely on it every time.
Score each layer on its own: retrieval (did it find the right text), generation (is the answer true to that text), tool calls (right tool and arguments), and task success (correct end state). A single overall score shows that something broke but not where.
LLM Evaluation
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
RAG Evaluation
Measuring a retrieval-augmented generation system in two parts, whether retrieval found the right text and whether the answer used it, so a failure can be traced to one of them.
LLM-as-Judge
Using a language model to score another model's output. Fast and scalable, but it has real failure modes such as self-preference and position bias, so the judge itself must be validated against human labels.
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
Golden Dataset
A curated set of inputs with known-correct expected outputs, used to measure quality and catch regressions when you change a prompt, a model, or the retrieval stack.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
This page explains the idea. The "Evaluating RAG and Agents: One Score Detects, Components Localize" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Evaluating RAG and Agents: One Score Detects, Components Localize lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about Agent Evaluation in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (153 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
