Evals for agentic software · from one test to release gates
A test checks an answer. An eval measures a behavior.
An agent can give a different answer each time, take a different path to the same answer, and be right for the wrong reason. This site starts from one test with one expected result and climbs, one idea at a time, to rubrics, golden datasets, trace grading, judges, repeated trials and release gates, asking one question at each step: what does this number let me claim?
For engineers and architects who build agent systems or decide whether one may ship. One running example, and code that runs without a model, a network or an API key.
An email arrives at an insurer's submissions inbox. A broker wants a quote for an office building:
From: broker@brokerage.example
To: submissions@insurer.example
Subject: Quote request SUB-1004
Please quote property cover for a masonry office, built 2001, 20,000 sq ft, sprinklered. The owner says the roof was replaced 9 years ago. Wind zone C. No losses.
The insurer wants software to do the first pass on requests like this one, so that its underwriters spend their time on the hard ones. An underwriter is the person at an insurer who decides whether to accept a risk. The software will prepare a recommendation. The underwriter will decide.
Before that software is allowed near a real request, somebody has to answer a plain question: is it good enough? This page follows the team that tries to answer it the way engineers answer it for any other code, with a test, and shows where the test stops being evidence.
The insurance example in one minute#
Software does not work from the email. An intake step reads it once and writes down the facts as one structured record, called a submission. From then on every step reads the submission and never the email. In the scenario the record has 63 fields; the example code on this site carries the ten below, which are enough for its rules.
{
"submission_id": "SUB-1004",
"occupancy": "office",
"construction": "masonry",
"year_built": 2001,
"area_sq_ft": 20000,
"building_value_usd": 2400000,
"sprinklered": true,
"roof_age_years": 9,
"wind_zone": "C",
"prior_losses_5y": 0
}
The submission goes to the underwriting agent. An agent is a program that calls a language model in a loop and lets the model ask for tools, which are functions the program runs on its behalf. This agent has four tools. It asks an outside provider for a roof assessment. It retrieves the insurer's written guidance. It looks up similar past decisions. And it runs the eligibility rules, which are ordinary code.
For this building, the tools turn up a problem that the email did not show. The owner says the roof is 9 years old. The roof-data provider says 22. The insurer's guidance has a rule for that, R-ROOF-03: when the two ages differ by more than 5 years, the case goes to a senior underwriter. So the agent writes this:
{
"recommendation": "escalate",
"reasons": [
"The application says the roof is 9 years old. The roof-data provider says 22 years.",
"R-ROOF-03: Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter.",
"Next step: escalate to a senior underwriter to settle the roof age."
],
"citations": ["R-ROOF-03"],
"missing": [],
"stated_confidence": 0.8,
"requires_human": true,
"policy_version": "2026.2"
}
That is the whole output of the agent: one of four recommendations (approve, decline, request_info or escalate), the reasons, the ids of the rules it relied on, anything that is missing, and a number it gives about its own answer. The recommendation is stored, and an underwriter sees it in a web app called the underwriter cockpit. Nothing else happens until that person acts. The model recommends; a person decides. In the record of this run, the last step is named underwriter_review and its status is waiting.
Diagram: The example in one minute · an email becomes a record, a recommendation, and a person's decision
Read left to right. Each column is one party. The broker writes an email, intake turns it into one record, the agent gathers evidence and writes a recommendation, and an underwriter decides. Click a box to see what it is in the roof conflict case.
- 1 · The broker
- Email: A broker asks the insurer for a quote in ordinary sentences.
- 2 · Intake
- Submission: The intake step writes the facts of the email into one structured record. Every later step reads this record, never the email.
- 3 · The underwriting agent
- Gather evidence: The agent asks an outside provider for a roof assessment, retrieves the written guidance, and looks up similar past decisions.
- Eligibility rules: A function applies the rules of the policy fixture. It is ordinary code, so the same input always gives the same result.
- Recommendation: The agent writes one of four recommendations with its reasons, the rule ids it relied on, and anything that is missing.
- 4 · A person
- Underwriter decides: An authorized underwriter reads the recommendation and approves or declines. Only a person writes the decision.
Read the diagram for who does what. The broker writes sentences. Intake turns them into a record. The agent gathers evidence and writes a recommendation. One box in the agent's lane is not a model at all: the eligibility rules are code. And the decision sits in a lane of its own, because it belongs to a person.
In our insurance agent. This is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider and every number are invented. Nothing here reports the quality of a real system or model.
The example has ten situations like this one. Each is written down with the result that people expect, and each has an id. A written-down situation with its expected result is called a case. The roof conflict is case UW-004.
The obvious check, and a green result#
The team wants to know whether the agent handles UW-004. The obvious tool is the one they use for all other code: a test. A test is one check with one expected result. Give the code this input; the result must equal that value. For UW-004 the expected value is escalate.
The command below runs the agent once on UW-004 and checks what it produced. It comes from examples/uw-evals/, the small example that ships with this site (Python 3, standard library only). The agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is. A successful run proves that one run of this one input gave the expected result. Hold on to that sentence; the rest of the page is about how little it covers.
cd examples/uw-evals
python3 run_evals.py --case UW-004
Eval run · suite UW-004 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-004 1 1 1.00 escalate -
trials 1 · passed 1 · pass rate 1.00 · 95% interval 0.21 to 1.00 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 1 trace file(s) in out/traces/
Read the third line: one run, one pass, the recommendation was escalate. For ordinary code, this would be the end of the story. A green test on a function means the function does this, and it will keep doing it until somebody changes the code.
One number in the fourth line does not fit that story. After one passing run, the program reports a range of 0.21 to 1.00. In plain words: one pass is compatible with an agent that gets this case right almost always, and also with one that gets it right about one time in five. The other words in that output (suite, critical, hard-fail) belong to later pages and are not needed here.
The same input, twenty times#
A test earns its trust from one property of ordinary code: the same input gives the same result. Run it once or a thousand times; nothing changes. An agent built around a language model does not have that property, and the smallest way to see it is to run two things twenty times each.
The script below does that for case UW-010, which has the same input as UW-004 and exists to be run many times. First it calls the eligibility rules, the part that is plain code, twenty times. Then it runs the whole agent twenty times. It was written for this page and run against the example; it is not one of the example's files. It shows the difference between the two kinds of software. It does not show how often a real model would vary.
from run_evals import load_cases, run_trial
from tools_sim import World
case = {c["id"]: c for c in load_cases("all")}["UW-010"]
facts = dict(case["input"]["submission"], provider_roof_age_years=22)
rules = [World(case).check_eligibility(facts)["outcome"] for _ in range(20)]
agent = [run_trial(case, "careful", 7, n)["output"]["recommendation"] for n in range(1, 21)]
print("rules in code, 20 calls:", {r: rules.count(r) for r in sorted(set(rules))})
print("the agent, 20 trials: ", {r: agent.count(r) for r in sorted(set(agent))})
rules in code, 20 calls: {'escalate': 20}
the agent, 20 trials: {'approve': 4, 'escalate': 16}
The rules said escalate twenty times out of twenty. The agent said escalate sixteen times and approve four times, on the same submission, with the same tools returning the same data. Each of those twenty runs has a name on this site: a trial, which is one run of one case.
Two honest notes about that result. The stand-in is told to skip the escalation in a share of trials, so that there is something to measure; a real agent varies for its own reasons, and the share is whatever it turns out to be. And the test from the previous section was not wrong. It reported one trial truthfully. The trouble is what a reader takes from a green check: that the behavior is settled.
The property behind this has a name, non-determinism: the same input can lead to different results. In an agent it has several sources. A language model usually picks each next word with some randomness. Tools answer with different timing, or fail. And an early difference changes every later step, because each step builds on the one before.
Why this matters here: in four trials of twenty, the underwriter would receive a recommendation to approve a building whose roof age is disputed by 13 years, with no mention of the dispute. A person still decides. But that person now decides on a recommendation that hides the one fact they needed.
The same answer, by a different path#
Suppose the answer were the same every time. A test would still miss something, and the second case shows it. Case UW-008 is a warehouse with a roof of 31 years in wind zone C. The guidance says that such a roof is not eligible (rule R-ROOF-01), so the expected recommendation is decline.
The example can run the agent in a second way, called the shortcut variant, in which the agent skips one step: it never retrieves the guidance. The command below runs UW-008 that way. A successful run of this command is a failed trial, and the last column says which checks failed.
python3 run_evals.py --case UW-008 --variant shortcut
Eval run · suite UW-008 · variant shortcut · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-008 1 0 0.00 decline required_process, rubric
trials 1 · passed 0 · pass rate 0.00 · 95% interval 0.00 to 0.79 (Wilson)
critical cases: 1 failed trials, 1 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 1 trace file(s) in out/traces/
The recommendation is decline, which is the expected one. The output of this trial is word for word the same as the output of the careful agent, and a check that compares the recommendation with the expected value passes. Two other checks fail, and the first one explains why: search_guidance was not called; cited but never retrieved: R-ROOF-01.
The agent cited a rule it never read. It was right, and it was right for the wrong reason. A test that looks only at the answer cannot tell these two agents apart:
| The careful agent | The shortcut agent | |
|---|---|---|
| Recommendation | decline | decline |
| Rule cited | R-ROOF-01 | R-ROOF-01 |
| Steps taken | 8 | 7 |
Retrieved the guidance (search_guidance) | yes | no |
| Checked against the answer only | pass | pass |
| Checked against the answer and the steps | pass | fail |
The difference shows up later, when the rules change. In this example the guidance has versions. Under the earlier version, 2025.4, the roof limit of R-ROOF-01 was 30 years; under the current one, 2026.2, it is 25. An agent that answers without reading the current guidance can be right today and wrong after the next version, with nothing in its answer to show it. The example has a case for exactly that, UW-007.
So there are two more things to look at besides the answer. One is the path: the steps the agent took, read from the record of the trial, which is called a trace. The other is what the trial left behind in the world: the stored rows and events, which here must show a recommendation and no decision, because only a person decides.
Common confusion. This does not mean an eval should prescribe every step. Two different paths can both be fine.UW-008checks one step because the insurer requires it: noapproveordeclinewithout reading the rule that is cited. What to evaluate is the home of this distinction.
What a test claims, and what an eval claims#
The team now has three findings from two cases. The answer varies between trials (UW-010). The path varies while the answer stays the same (UW-008). And a right answer can rest on a reason that will not hold (UW-008 again, and UW-007). None of these is a defect in the test. A test was designed for code that repeats itself, and it claims one thing: this input gave this result.
What the team needs is a measurement. An eval is a measurement of a behavior over cases, with a stated way of scoring. Each word in that sentence does work. A measurement gives a number with a sample size, not a green check. A behavior is what the agent tends to do, across trials and across cases. And a stated way of scoring means that somebody wrote down, before the run, what counts as good.
The idea. A test checks an answer: one input, one expected result, one run. An eval measures a behavior: many cases, repeated trials, a stated way of scoring what the agent said, what it did and what it left behind. The result of an eval is a number, and the useful question about any such number is: what does it let me claim?
| A test | An eval | |
|---|---|---|
| What it looks at | One result of one run | Results, paths and end states over many trials |
| What it compares with | One expected value | An expected result per case, plus a scoring guide |
| How often it runs a case | Once | Once for some cases, many times for others |
| What comes out | Pass or fail | Scores per trial, then a rate over trials |
| What it lets you claim | This input gave this result | How often, on these cases, the behavior was acceptable |
| Where it is the right tool | Code that repeats itself | Software whose results vary |
The table is not a ranking. Tests stay. The eligibility rules in this example are plain code and deserve plain tests, and the program that runs the evals has 24 unit tests of its own. The rule of thumb is to test what repeats and to evaluate what varies. An agent is a mixture of both, which is why its diagram had a box of plain code in the middle of the agent's lane.
How other sources use these words"Eval" and "test" are often used as one word; this site keeps them apart on purpose
The vocabulary of this site starts from Anthropic's article "Demystifying evals for AI agents" (published January 2026, read for this page on 2026-11-17). That article describes an eval as a test for an AI system: an input goes in, and grading logic is applied to what comes out. It then defines task, trial, grader, transcript, outcome, evaluation harness, agent harness and evaluation suite. This site uses those ideas with attribution and its own examples.
Three wording choices differ, and each is deliberate.
| The article says | This site says | Why |
|---|---|---|
| An eval is a test for an AI system | A test is one deterministic check; an eval is a measurement over cases | The difference is the subject of this page, so the two words stay apart |
| Transcript (also trace or trajectory) | Trace for the record; trajectory for the path read from it | Two words for two things: the data, and what you conclude from it |
| Task | Task for what the agent is asked to do; case for one input with its expected result | The ten ids UW-001 to UW-010 are cases of one task |
Nothing here is a standard. Another team may draw the lines elsewhere, and a tool may use "evaluator" or "scorer" where this site says grader. Primitives defines each word once, and About lists the sources.
Where the variation comes from, and what does not remove itA fixed seed or a lower randomness setting makes runs less variable, not identical
It is tempting to treat non-determinism as a setting to turn off. Three points for a reader who has tried.
A lower randomness setting on the model reduces variation in the text. It is not a guarantee of identical output for identical input unless the provider states that guarantee, and this site assumes no such guarantee.
The model is only one source. An agent calls tools, and tools time out, return results in a different order, or return data that changed since yesterday. Retrieval can return different passages after the index is rebuilt. A model version can change behind a stable name. Each of these changes a later step, because the agent decides its next step from what it has seen so far.
A seed makes a run repeatable; it does not make the measured rate equal to the true rate. The example shows this. With seed 7, UW-010 passes 16 of 20 trials. The README of the example reports that seeds 1 to 12 give between 12 and 18 passes out of 20 for the same stand-in, whose scripted rate does not change. Reliability is the home of repeated trials, intervals and sample size.
When grading the path is right, and when it makes an eval brittleGrade the outcome by default; grade a step only where a rule requires that step
The same article warns against checking that an agent followed one fixed sequence of steps: agents often find valid routes the author of the eval did not expect, so it is often better to grade what the agent produced than the route it took. This site agrees, and UW-008 is not an exception to it.
The check that fails in UW-008 does not compare the path with a fixed sequence. It asks two narrow questions that come from the insurer's own process: was check_eligibility called, and did a search_guidance call return every rule that the answer cites? Any order and any number of other steps pass. The check also applies only to approve and decline. For escalate it reports "not required", which is why UW-004 passes it without being examined.
The production trade-off: every path check is a requirement that the agent's design must keep meeting. Write one where a regulation, a policy or a safety property needs the step, and say which. Everywhere else, grade the output and the end state, and read traces to understand failures, not to fail trials.
What 16 of 20 lets you claim#
An eval gives a number, and a number invites a wrong reading. Sixteen passing trials of twenty is a rate of 0.80. Suppose the insurer set a line of 0.70 that this case must reach before a new version of the agent may be released. Such a line is called a threshold. Is 0.80 above 0.70?
The example has a small program, gate.py, that makes this release decision from a report. The commands below run the twenty trials of UW-010 and then the gate. A run that prints BLOCK is working as designed: it shows the rule the gate applies. It does not show that 0.70 is the right line for anyone; the line is a choice.
python3 run_evals.py --case UW-010
python3 gate.py
Eval run · suite UW-010 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-010 20 16 0.80 escalate x16, approve x4 recommendation_match (4), citations (4)
trials 20 · passed 16 · pass rate 0.80 · 95% interval 0.58 to 0.92 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 20 trace file(s) in out/traces/
Gate · suite UW-010 · variant careful · threshold 0.70
BLOCK: 16 of 20 trials passed (0.80); the 95% interval is 0.58 to 0.92. The lower bound 0.58 is below the threshold 0.70. The observed rate looks high enough, but the sample is too small to claim it.
The observed rate is above the line, and the gate still blocks. Twenty trials are a small sample. An agent whose true rate is 0.60 produces 16 passes of 20 now and then. The interval in the output, 0.58 to 0.92, is the range of true rates that fit what was observed, and its lower end is below the line. The team cannot yet claim that this agent escalates the conflict at least 70 percent of the time. It can claim that it did so in 16 of 20 trials.
That gap, between what was observed and what may be claimed, is the subject of the whole site. Three sentences hold it, and they return on later pages in exactly these words. A score is an observed result. Confidence is a claim about uncertainty. Reliability needs repeated evidence.
One more number deserves a warning now. The output of the agent carries stated_confidence, 0.8 for the trial shown above. It is the number the model gives about its own answer. It is not a measurement of anything, and it happens to equal the measured rate here only by coincidence. Confidence separates five numbers that look alike.
One eval, part by part#
The runs above used words that the next pages define with care. Here is the whole shape first, so that each later word has a place to go.
An eval starts with a case: an input and the result people expect. The agent runs on the input once; that is a trial, and it leaves a trace. Something then checks the trial and gives a result. That something is called a grader, and it can be code, a model or a person. What a grader gives is a score: pass or fail, or a number. Scores of many trials are put together into a report, with rates that a person or a release gate can read.
The lab below shows these six parts for case UW-004. Pick a part to read what it is. The table after it holds the same information.
| Part | Plain English | In case UW-004 |
|---|---|---|
| Input | What the agent is given. The same input is used every time this case runs | Submission SUB-1004: a masonry office, roof 9 years on the application, wind zone C |
| Agent run | One run of the agent on that input: a trial. It leaves a trace, the ordered record of its steps | Eight steps, from plan_steps to underwriter_review |
| Output and outcome | The output is what the agent said. The outcome is what is true afterwards | Output: escalate, citing R-ROOF-03. Outcome: one stored recommendation and no decision |
| Grader | The thing that checks the trial and gives a result: code, a model or a person | Six graders, among them recommendation_match, required_process and rubric |
| Score | The observed result of grading one trial. It is not a probability | Six of six graders passed; the rubric grader gave 1.00 out of 1.00 |
| Report | The scores of many trials put together, for a person or a release gate | In the run of all ten cases, 25 of 29 trials passed, a rate of 0.86 (synthetic) |
Two things in this table are easy to pass over. The outcome row is separate from the output row, because an agent can say one thing and leave another behind. And the score row says what a score is not: a rubric score of 1.00 does not mean the recommendation is certainly right. It means that this trial met every item on the scoring guide. (A rubric is a scoring guide: the list of things to check and how much each counts. Grading builds one.)
How to read this site#
The pages follow the order in which the questions come up when a team builds its first eval. First the words and the three things every eval needs: a case, a way of scoring, and cases worth trusting. Then the mechanics of looking closer: what to grade, the record to grade it from, who grades the parts that code cannot, and what the resulting numbers mean. Then the systems that run all of this on every change and on real traffic. A reference part closes the site.
Diagram: The route through the site · first the words, then a closer look, then the systems
Read left to right. Each column is a part of the site and each box is a group of pages. The first column needs no tooling. Click a box to see what its pages teach.
- 1 · Foundations
- The small words: Primitives: case, trial, score, threshold, suite, baseline and the other words every eval is built from.
- Who scores, by what guide: Grading and rubrics: a grader gives the score, a rubric says what to check, and some failures cap the score whatever else is good.
- Cases you trust: Golden datasets: cases whose expected result people agreed on, with labels, versions and a record of who decided.
- 2 · Mechanics
- Answer, path, end state: What to evaluate and Agent traces: three different things to grade, and the record the path is graded from.
- A model as grader: Judges and calibration: what a model can grade that code cannot, how it goes wrong, and how to compare it with people.
- What a number means: Confidence and Reliability: five numbers that look alike, and how many trials a claim about a rate needs.
- 3 · Systems
- The eval harness: The program that feeds cases to the agent, resets the world between trials, calls the graders and writes the report.
- Release gates: CI/CD quality gates: which evals may block a release, and what a blocking threshold must account for.
- Real traffic: Production evals: sampling, privacy, drift, and the loop that turns an incident into a new case.
- 4 · Reference
- Frameworks: What each eval tool runs for you, described from its own documentation with the date it was read.
- The insurance lab: The whole example in one place: ten cases, the policy fixture, traces, graders, results and the release decision.
- Where evals fail: Advanced topics: adversarial cases, injected text, judges that can be gamed, contamination, model and policy upgrades.
The first column needs no tooling and no statistics. The second is where most of the difficulty lives. The third assumes the second: a release gate is only as good as the numbers that feed it.
New to evals: read in order#
Foundations. The vocabulary, the scoring, and the cases.
Mechanics. What to look at, and what the numbers mean.
Systems. Running evals on every change, and on real traffic.
Reference. Tools, the whole example, and the hard edges.
Experienced with evals: jump to a question#
Each page recaps what it needs from earlier pages in a sentence and links to the full explanation, so a reader who already runs evals can start at the question of the day. Each page also folds its rigorous part (formulas with their assumptions, edge cases, production trade-offs) into blocks like the three above.
| If your question is | Start at |
|---|---|
| How do I grade what the agent did, and not only what it said? | What to evaluate, then Agent traces |
| Can I trust a model that grades a model? | Judges and calibration |
| Is this 0.8 a confidence, a score or a rate? | Confidence and uncertainty |
| How many trials do I need before I may claim a rate? | Reliability and statistics |
| Which evals may block a release, and what about a flaky one? | CI/CD quality gates |
| What changes once real traffic arrives? | Production evals |
| Which tool runs which part of this for me? | Frameworks |
| Where is the whole example with its code? | The insurance lab |
Two sister sites#
This site measures an agent from outside. Two other sites by the same author cover what sits on either side of that measurement, and this site links to them where the subjects meet instead of repeating them.
What this site is not#
A map to adapt, not a standard, and not a scoreboard. The words and the page order are one way of dividing the subject; no standards body defined them. The agent in the example is a scripted stand-in, so no result on this site says how good any model is. Tools are described by what they run for you, from their own documentation with the date it was read, and none is called best.
One more limit follows from the example itself. Ten cases are enough to teach every idea on this site and far too few to release a real agent on. Where a page shows a rate or an interval, the number is synthetic and belongs to this example. What carries over to your system is the reasoning about what such a number lets you claim.
In one breath#
A test tells me that one input gave one expected result, and for code that repeats itself that is enough. An agent does not repeat itself: the same submission gave escalate in 16 trials and approve in 4, and another agent gave the right answer while citing a rule it never read. So I stop asking whether it passed and start measuring how it behaves: many cases, repeated trials, and graders that look at the answer, the path and what was left behind. And for every number that comes out, I ask what it lets me claim.
The same idea for a coding agent: a build that goes green once says the patch worked that time; it does not say how often the agent produces a working patch, or whether it got there by deleting the failing test.
The team now knows why its green check was thin, and it has seen one eval from input to report. What it does not have yet is the vocabulary to build one: the page so far has used case, trial, grader, score and threshold with one sentence each, and has not yet said what a suite, a baseline or a regression is. Primitives takes each of those small words in turn, gives each one a tiny example, and assembles them into the first eval the team writes itself.