All chapters
  1. Start hereA test checks an answer. An eval measures a behavior
  2. Foundations · 1Primitives: the small words every eval is built from
  3. Foundations · 2Grading and rubrics: who decides what good means
  4. Foundations · 3Golden datasets: the cases you trust
  5. Mechanics · 4What to evaluate: the answer, the path, and what was left behind
  6. Mechanics · 5Agent traces: the record you grade the path from
  7. Mechanics · 6Judges and calibration: when a model grades a model
  8. Mechanics · 7Confidence and uncertainty: five numbers that are not the same
  9. Mechanics · 8Reliability and statistics: one run proves little
  10. Systems · 9The eval harness: the program that runs the evals
  11. Systems · 10CI/CD quality gates: which evals may block a release
  12. Systems · 11Production evals: measuring what real traffic does
  13. Reference · 12Frameworks: what each eval tool runs for you
  14. Reference · 13The insurance lab: ten cases, end to end
  15. Reference · 14Advanced topics: where evals themselves fail
  16. ReferenceGlossary
  17. AboutAbout this site and its sources
Foundations · 1

Primitives: the small words every eval is built from

Input, expected result, assertion, score, threshold, task, trial, suite, baseline, regression: each word with one tiny example, then assembled into one eval.

Sixteen small words
Each one gets one plain sentence and one insurance example before it is used with another.
One eval, assembled
From one check on one case to a run of ten cases with a report you can read line by line.
What a result lets you say
Why "it passed" means nothing until you say what ran, how often, and against which line.

A quote request reaches the insurer: a masonry warehouse, roof 31 years old, wind zone C. The underwriting agent reads it and recommends decline to the underwriter. The model recommends; a person decides.

An engineer has changed the agent so that it answers in fewer steps. To check the change, the engineer runs the agent on the warehouse request, reads decline on the screen, and writes two words in the change description: "Tested, works."

A reviewer replies with four questions.

  1. You read decline and it looked right. Where is "right" written down, so that the next person does not have to look?
  2. Who says that decline is right for this warehouse?
  3. You ran one request, once. What about the request with no roof age, or the one where the roof provider does not answer? What about a second run of the same request?
  4. Is the agent as good as it was before your change?

"Tested, works" cannot answer any of them. The engineer was not careless. The sentence has no place to put the answers. Each answer needs a word, and the people who measure agents have settled on about sixteen of them.

This page introduces those words in the order the four questions force them. Each word gets one plain sentence and one small example from the warehouse request. At the end the engineer's change is measured again, and the result is not "works".

One note before any result appears: the agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is. The code is in examples/uw-evals/. It calls no model and needs no key.

1. One check, written down#

The reviewer's first question is about a check that lived only in the engineer's head. To write a check down, four things need names.

The first is the input: what the agent is given. Here it is one submission, the record of 63 fields that the intake step made from the broker's email. The example uses ten of the fields. For the warehouse, the two that matter are roof_age_years: 31 and wind_zone: "C".

The second is the output: what the agent gives back. Here it is one JSON record. This is the output of the agent for the warehouse, printed from a run of the example:

{
  "recommendation": "decline",
  "reasons": [
    "The application and the roof-data provider both give a roof age of 31 years.",
    "The building is in wind zone C.",
    "R-ROOF-01: Roof older than 25 years in wind zone C: not eligible.",
    "Next step: the underwriter reviews this recommendation and decides."
  ],
  "citations": ["R-ROOF-01"],
  "missing": [],
  "stated_confidence": 0.9,
  "requires_human": true,
  "policy_version": "2026.2"
}

R-ROOF-01 is a rule in the policy fixture, the small set of underwriting rules invented for this site. It is illustrative and is not insurance practice.

The third is the expected result: what the output should be, written down before the run. For the warehouse it says: the recommendation is decline, the citations include R-ROOF-01, nothing is reported missing, and requires_human is true. In the example it is the expected block of the file cases/UW-001.json.

The fourth is the assertion: one check that compares something observed with something expected and answers yes or no. The smallest useful one here is: the recommendation in the output equals the recommendation in the expected result.

The two answers of an assertion have names as well: pass and fail. Pass means the check held. Fail means it did not. Nothing in between exists at this level.

Plain English. An assertion is a sentence that can only be true or false: "the recommendation is decline". Written down, a program can repeat it tomorrow.

The diagram shows these five words as one picture. Read it left to right.

Diagram: One check on one case · five words

Read left to right. The input goes to the agent and an output comes back. The expected result was written down before the run. The assertion compares the two and answers pass or fail. Everything else on this page is built on this picture.

  1. 1 · Written before the run
    1. Input: What the agent is given: one submission, the record that the intake step made from the broker's email.
    2. Expected result: What the output should be, written down before the run by people who know the rules.
  2. 2 · Observed in the run
    1. The agent runs: The underwriting agent reads the input and does its work one time.
    2. Output: What the agent gives back: a record with a recommendation, reasons and the rules it relies on.
  3. 3 · The check
    1. Assertion: One check that compares something observed with something expected.
    2. Pass or fail: The answer of the assertion, with a sentence that says why.

Now the same thing in code. The function below is recommendation_match from graders.py, unchanged. It reads the expected recommendation from the case and the observed one from the run, and returns pass or fail with a sentence that says why.

def recommendation_match(case: dict[str, Any], trial: dict[str, Any]) -> Result:
    want, got = case["expected"]["recommendation"], trial["output"]["recommendation"]
    return _result("recommendation_match", got == want, f"expected {want}, got {got}")

The next script applies that one assertion to the warehouse. It shows that the four words are four separate values in the program. A successful run proves one thing: in this run, the recommendation matched. It does not prove that the reasons are sound, or that a second run would match. The script uses two functions of the example's runner and is not part of the example; it was run here from inside examples/uw-evals/.

from run_evals import load_cases, run_trial
from graders import recommendation_match

case = load_cases(case_id="UW-001")[0]
trial = run_trial(case, "careful", 7, 1)   # variant, seed, trial number

print("input   :", case["input"]["submission"]["roof_age_years"], "years, wind zone", case["input"]["submission"]["wind_zone"])
print("output  :", trial["output"]["recommendation"])
print("expected:", case["expected"]["recommendation"])
print("check   :", recommendation_match(case, trial))
input   : 31 years, wind zone C
output  : decline
expected: decline
check   : {'name': 'recommendation_match', 'passed': True, 'detail': 'expected decline, got decline'}

The first question is answered: "right" is written in a file, and a three-line function compares the output with it.

2. Who says that decline is right#

The assertion passed because the output matched the expected result. That moves the second question one step back: why is the expected result right?

The word for this is ground truth: the answer that is actually correct for an input. For an arithmetic problem the ground truth is a fact. For an underwriting question it is not a fact of nature. It is a judgment by people who know the rules. So on this site ground truth always comes with the people who decided it.

For the warehouse, the case file records who that was:

{
  "labelled_by": "senior underwriter",
  "role_is_synthetic": true,
  "date": "2026-08-03",
  "synthetic": true
}

A senior underwriter read the submission against version 2026.2 of the policy fixture and wrote decline and R-ROOF-01. The role and the date are invented, like everything in the example. The shape is the point: a role, a date and a rule version stand next to every expected result.

Common confusion. The expected result is what a file says. The ground truth is what is correct. They agree only as long as the people who labelled the case were right and the rules have not changed. When the roof limit moved from 30 years to 25 between two versions of the policy fixture, some expected results written under the old version stopped being true.

How people agree on labels, and how a set of labelled cases is kept true over time, is the subject of Golden datasets.

3. One check is not the whole answer#

The assertion compared one word, decline. An agent that declined every warehouse it ever saw would pass it. The expected result holds more than one word, so the example checks more.

Each check is done by a grader: a function, a model or a person that looks at a run and gives a result. That one sentence is enough for this page; Grading and rubrics is the home of the word. Six graders look at the warehouse run. This script prints what each one found. It proves that all six held in this run, and nothing more.

from run_evals import load_cases, run_trial

case = load_cases(case_id="UW-001")[0]
trial = run_trial(case, "careful", 7, 1)

for g in trial["graders"]:
    print(f"{g['name']:<22}{'pass' if g['passed'] else 'FAIL':<6}{g['detail']}")
print("trial passed:", trial["passed"])
recommendation_match  pass  expected decline, got decline
citations             pass  expected R-ROOF-01; got R-ROOF-01
missing_fields        pass  expected none; got none
required_process      pass  eligibility checked and cited rules retrieved
human_required        pass  recommendation stored; decision left to a person
rubric                pass  score 1.00 (threshold 0.7)
trial passed: True

Each line is a score: the observed result of grading one run. Five of the six scores have two possible values, pass and fail. The last one is a number. The rubric grader reads the written reasons against a scoring guide and gives a value between 0 and 1. Here it gave 1.00.

A number alone decides nothing. Somebody has to draw a line. That line is a threshold: the value a score must reach to count as a pass. In the example the scoring guide sets it at 0.70, so 1.00 passes and 0.65 would fail.

Then one rule turns six scores into one answer for the whole run. In the example the rule is the strictest one: the run passes only if every grader passes. It is one line of run_trial in run_evals.py, shown here without its indentation:

record["passed"] = all(g["passed"] for g in record["graders"])
Common confusion. A score of 1.00 from the rubric grader does not mean the recommendation is certainly right. It means the written reasons met every item of the scoring guide in this run. The output also carries stated_confidence: 0.9, which is a third thing: the agent's own claim about its answer. A score is an observed result. Confidence is a claim about uncertainty. Confidence and uncertainty keeps the two apart.

4. What ran, and how many times#

The third question was: one request, once? Answering it needs three words that people often use as if they were one.

A task is what the agent is asked to do, together with what counts as success. In this example there is one task: read a submission and write a recommendation, one of approve, decline, request_info or escalate, with reasons and with the rules it relies on, and leave the decision to a person.

A case is one input with its expected result and its labels. The warehouse is a case. Its id is UW-001. The task stays the same from case to case; the input and the expected result change. UW-003 is a submission with no roof age, and its expected recommendation is request_info.

A trial is one run of one case. The engineer's check at the top of the page was one trial of UW-001.

Why keep case and trial apart? Because the same case can give different results in different trials. UW-010 is a submission where the application says the roof is 9 years old and the provider says 22. The expected recommendation is escalate. The example runs this one case twenty times:

/opt/homebrew/bin/python3 run_evals.py --case UW-010
Eval run · suite UW-010 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-010       20    16   0.80  escalate x16, approve x4  recommendation_match (4), citations (4)
trials 20 · passed 16 · pass rate 0.80 · 95% interval 0.58 to 0.92 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 20 trace file(s) in out/traces/

One case, twenty trials, sixteen passes. In four trials the agent recommended approve. The stand-in is scripted to skip the escalation in a share of trials; a real agent varies for its own reasons, which Start here explains.

In our insurance agent. If the engineer had run one trial of UW-010 and seen escalate, "tested, works" would have been an honest report of that trial and a wrong report about the agent. One passing trial is one observation. Reliability needs repeated evidence.

5. From many scores to one number#

Twenty trials produced twenty pass-or-fail scores. Nobody wants to read twenty lines for every case, so the scores are combined.

A metric is a number computed over many scores. The most common one is the pass rate: passes divided by trials. For UW-010 that is 16 divided by 20, which is 0.80. In the example it is the function pass_rate in stats.py:

def pass_rate(results: list[bool]) -> float:
    """Share of trials that passed. 0.0 for an empty list."""
    return sum(results) / len(results) if results else 0.0

A metric needs a line too, and the word is the same: threshold. This is the second place a threshold appears, and the two must not be mixed. The first threshold sat on a score: does this one run's rubric value reach 0.70? This one sits on a metric: does the pass rate of this case reach the line that the team set for a release? In the example that line also happens to be 0.70. The two numbers are equal by coincidence and mean different things.

NumberWhat kindComputed overThe line it meets
Rubric value 1.00 for trial 1 of UW-001ScoreOne trialScore threshold 0.70 in the scoring guide
Pass rate 0.80 for UW-010MetricTwenty trialsRelease threshold 0.70 in the gate

So "it got 0.8" is not a sentence anyone can act on until it says whether 0.8 is one trial's score or a rate over many, and which line it was held against.

One caution. The pass rate 0.80 is what was observed in twenty trials, not the true rate of the agent. The output above prints a range next to it, 0.58 to 0.92, and the example's release gate compares the low end of that range with the threshold. Reliability and statistics explains the range, and CI/CD quality gates explains the rule.

6. Sets of cases, and surroundings that stay fixed#

The third question also asked about other kinds of request. The example has ten cases for ten situations. Running all of them on every small change can be too much, and running one is too few. So cases are grouped.

A suite is a named set of cases with one purpose. In the example a case names its suites in its own file; UW-001 belongs to smoke and regression. The smoke suite is the first three cases, a quick check that the agent still does the basic things:

/opt/homebrew/bin/python3 run_evals.py --suite smoke
Eval run · suite smoke · variant careful · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-001        1     1   1.00  decline                   -
UW-002        1     1   1.00  approve                   -
UW-003        1     1   1.00  request_info              -
trials 3 · passed 3 · pass rate 1.00 · 95% interval 0.44 to 1.00 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 3 trace file(s) in out/traces/

There is a hidden condition in all of this. The agent does not work from the submission alone. It asks a roof provider for the roof age, and it retrieves rules from the underwriting guidance. If the provider answered 31 years today and 12 tomorrow, the same case would have two different right answers, and no result could be compared with another.

A fixture is the prepared material that is put in place before a trial and is the same every time. In the example there are two. The policy fixture holds the rules, with a version and a date. The world block of each case holds what the stand-in tools return for that case. For UW-001 it says the provider reports a roof age of 31 years:

{
  "roof_assessment": {
    "status": "ok",
    "roof_age_years": 31,
    "source": "roof-data provider (stand-in)",
    "method": "aerial image, synthetic"
  }
}

The runner builds these surroundings fresh for every trial, so that a row written by one trial cannot change the next. With the fixture held still, a difference between two results has one fewer possible cause: it did not come from the provider or from the rules. The eval harness explains how the runner does this and what a stand-in tool cannot show.

7. Better or worse than before#

The fourth question was the one the engineer could least answer: is the agent as good as before the change? "As good as" needs something to compare with.

A baseline is an earlier result that is kept so that later results can be compared with it. Before the change, the regression suite, cases UW-001 to UW-009, gave this result:

trials 9 · passed 9 · pass rate 1.00 · 95% interval 0.70 to 1.00 (Wilson)

Nine of nine. That line is the baseline.

A regression is something that passed at the baseline and fails now. A team wants to hear about it before a release, because a change broke a behavior that used to hold.

Now the engineer's change can be measured the way the reviewer asked. In the example code the change is the shortcut variant of the stand-in agent, and the agent before the change is the careful variant. The command below runs the nine cases with the change and compares the result with the baseline, case by case. A successful comparison shows which cases changed from pass to fail on the same inputs and the same fixture. It does not show why.

/opt/homebrew/bin/python3 run_evals.py --suite regression --variant shortcut --compare careful
Eval run · suite regression · variant shortcut · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-001        1     0   0.00  decline                   required_process, rubric
UW-002        1     0   0.00  approve                   required_process
UW-003        1     0   0.00  request_info              rubric
UW-004        1     0   0.00  escalate                  rubric
UW-005        1     0   0.00  decline                   required_process, rubric, injection_ignored
UW-006        1     1   1.00  request_info              -
UW-007        1     0   0.00  decline                   required_process, rubric
UW-008        1     0   0.00  decline                   required_process, rubric
UW-009        1     0   0.00  approve                   required_process
trials 9 · passed 1 · pass rate 0.11 · 95% interval 0.02 to 0.44 (Wilson)
critical cases: 8 failed trials, 6 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 9 trace file(s) in out/traces/

Paired comparison on the same 9 trials: shortcut vs careful
pass rate 0.11 vs 1.00 · difference -0.89
only shortcut passes: 0 · only careful passes: 8 · both pass: 1 · both fail: 0
exact sign test on the 8 pairs that differ: p = 0.008

Read the first row. UW-001 is the warehouse. The recommendation is still decline, the word the engineer saw on the screen. The assertion from section 1 still passes. Two other graders fail: required_process and rubric.

The reason is in what the change did. To answer in fewer steps, the changed agent no longer looks the rule up in the guidance. It writes R-ROOF-01 from memory. The output is the same, word for word. The way the agent reached it is not, and the task requires the lookup. This is the subject of What to evaluate: a right answer reached without the required check.

The line "only careful passes: 8" is the count of regressions: eight cases passed at the baseline and fail with the change. The last line is a significance test on those eight pairs; Reliability and statistics explains what it does and does not support.

In our insurance agent. "Tested, works" becomes: one trial of UW-001 passed the recommendation check; on the regression suite the change passes 1 of 9 cases against a baseline of 9 of 9, with 8 regressions. Every word in that sentence was defined on this page, and a reviewer can check each number.

8. One eval, assembled#

Sixteen words are now on the table. Here they are together, in the order they were built. Every number is from the example and is synthetic.

WordOne plain sentenceIn the example
InputWhat the agent is givenThe submission of UW-001: roof 31 years, wind zone C
OutputWhat the agent gives backA JSON record with the recommendation decline
Expected resultWhat the output should be, written before the runThe expected block of cases/UW-001.json
Ground truthThe answer that is actually correct, and who decided itLabelled by a senior underwriter under version 2026.2 (synthetic)
AssertionOne check that is true or falseThe recommendation equals decline
Pass, failThe two answers of a checkrecommendation_match passed
ScoreThe observed result of grading one trialSix graders passed; rubric value 1.00
ThresholdThe line a score or a metric must reach0.70 for the rubric value; 0.70 for the release
TaskWhat the agent is asked to do, and what counts as successRecommend one of four actions, with reasons and rules; a person decides
CaseOne input with its expected result and labelsUW-001
TrialOne run of one caseUW-010 has 20 trials
MetricA number computed over many scoresPass rate of UW-010: 16 of 20, 0.80
SuiteA named set of cases with one purposesmoke: UW-001 to UW-003
FixturePrepared material that is the same for every trialThe policy fixture; the world block of a case
BaselineAn earlier result kept for comparisonregression before the change: 9 of 9
RegressionSomething that passed at the baseline and fails now8 of the 9 cases after the change

An eval is these parts working together: cases that share a task, trials of each case on a fixed fixture, graders that turn each trial into scores, metrics over the scores, and thresholds that turn the metrics into a statement someone can act on. The second diagram shows how the parts nest. A suite holds cases, a case is run as trials, a trial produces scores, and the scores of many trials become a metric that meets a threshold.

Diagram: One eval · how the sixteen words fit together

Read left to right. A suite holds cases that share one task. Each case is run as one or more trials on the same fixture. Graders turn each trial into scores. A metric is computed over the scores of many trials and is held against a threshold and a baseline. The numbers are from the example and are synthetic.

  1. 1 · What is run
    1. Task: What the agent is asked to do, with what counts as success.
    2. Suite: A named set of cases with one purpose.
    3. Case: One input with its expected result and its labels.
  2. 2 · Each run
    1. Fixture: Prepared material that is put in place before every trial and does not change.
    2. Trial: One run of one case. The same case can give different results in different trials.
  3. 3 · Each result
    1. Graders: Each grader checks one thing about the trial and gives a result with a reason.
    2. Scores: The observed results of grading one trial. A number becomes pass or fail at a score threshold.
  4. 4 · Many trials
    1. Metric: A number computed over the scores of many trials.
    2. Threshold: The line a metric must reach before a release goes ahead.
    3. Baseline and regression: The baseline is an earlier result kept for comparison. A regression is a case that passed then and fails now.

This is the whole example run once, all ten cases, with the agent before the change. It demonstrates every word of the table in one output. A successful run proves that the runner, the cases and the graders work together and gives one observation of each number. It says nothing about any real model.

/opt/homebrew/bin/python3 run_evals.py --suite all
Eval run · suite all · variant careful · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-001        1     1   1.00  decline                   -
UW-002        1     1   1.00  approve                   -
UW-003        1     1   1.00  request_info              -
UW-004        1     1   1.00  escalate                  -
UW-005        1     1   1.00  decline                   -
UW-006        1     1   1.00  request_info              -
UW-007        1     1   1.00  decline                   -
UW-008        1     1   1.00  decline                   -
UW-009        1     1   1.00  approve                   -
UW-010       20    16   0.80  escalate x16, approve x4  recommendation_match (4), citations (4)
trials 29 · passed 25 · pass rate 0.86 · 95% interval 0.69 to 0.95 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 29 trace file(s) in out/traces/

Every column is now readable. case is a case id. trials and pass count trials. rate is a metric. recommendation is a field of the output. failed graders names the scores that were a fail. The line under the table is the metric over all 29 trials. The line after it splits the cases into those that must pass in every trial and those judged by their pass rate; CI/CD quality gates explains the split.

The same eval, part by part#

The lab below walks through one trial of UW-001 in six parts. Pick a part to see what it is. The table after it holds the same content.

PartPlain definitionIn UW-001
1 · InputWhat the agent is given. The same input is used every time this case runsSubmission SUB-1001, a masonry warehouse, roof 31 years, wind zone C
2 · Agent runOne run of the agent on that input: a trial. It leaves an ordered record of its stepsEight steps, from plan_steps to underwriter_review
3 · Output and outcomeThe output is what the agent said. The outcome is what is true afterwards: the rows and events it left behindOutput: decline, citing R-ROOF-01. Outcome: one recommendations row, no decisions row yet
4 · GraderThe thing that checks the trial and gives a result. Code, a model or a personSix graders, from recommendation_match to rubric
5 · ScoreThe observed result of grading one trial. Not a probabilitySix of six graders passed; the rubric grader gave 1.00 out of 1.00
6 · ReportThe scores of many trials put together into metricsIn the whole run, 25 of 29 trials passed: a pass rate of 0.86

Part 3 holds one word this page has not needed: outcome. No decisions row exists after the trial, because only the underwriter writes one. What to evaluate covers grading what a trial left behind.

Check the words#

Five short questions on the words that are mixed up most often. The table after the lab holds the answers, so try the lab first.

QuestionRight answerReason
The agent runs case UW-004 once and the graders check it. What is that one run called?A trialA case is the input with its expected result. A trial is one run of it. UW-010 is one case with 20 trials
UW-004 says the expected recommendation is escalate. Who decided that?People who labelled the caseAn expected result is a decision by people, written down before the run. That is why every case records who labelled it
The rubric grader returns 0.75 for one trial. What is 0.75?A score: the observed result of grading one trialA score is an observed result. It is not a probability, and it is not confidence
16 of 20 trials passed, so the pass rate is 0.80. What is 0.80?A metric: a number computed over many scoresA metric is computed over many scores. The threshold is the line that the metric must reach
Last week all nine regression cases passed. After a prompt change, UW-007 fails. What is this called?A regressionThe baseline is the earlier result you compare with. A regression is something that passed at the baseline and fails now

Where the words are used loosely#

These definitions are the ones this site keeps. Other texts draw some lines elsewhere, and several of the words hide choices that matter in production. The blocks below hold that detail.

The same words in Anthropic's articleTask there is close to case here; the split is deliberate

The conceptual starting point of this site is Anthropic's article "Demystifying evals for AI agents". Its vocabulary and this site's agree on trial, grader, outcome, evaluation harness and agent harness. One word differs.

Word in the articleHow the article describes itWord on this site
Task (it adds: also called a problem or test case)"a single test with defined inputs and success criteria"Case, for the one input with its expected result. Task is kept for what the agent is asked to do across all cases
Trial"Each attempt at a task is a trial"Trial
Assertions, also called checksA grader can contain several of themAssertion
Evaluation suiteA collection of tasks designed to measure specific capabilities or behaviorsSuite
Transcript, also called trace or trajectoryThe complete record of a trialTrace for the record, trajectory for the path read from it

This site splits task from case for one reason. In the insurance example the instruction to the agent never changes, and ten inputs with ten expected results do. Two words keep "we changed the task" (the definition of success moved) apart from "we added a case" (one more input). When you read a tool's documentation, check which of the two its word "task" means.

An assertion is a decision about strictnessSubset or equality: two graders of the example choose differently

Two graders in graders.py compare a list in the output with a list in the expected result, and they do not compare in the same way.

def citations(case: dict[str, Any], trial: dict[str, Any]) -> Result:
    """Every expected rule id is cited. Extra citations are allowed here."""
    want, got = set(case["expected"]["citations"]), set(trial["output"]["citations"])
    return _result("citations", want <= got, f"expected {_ids(want)}; got {_ids(got)}")

citations uses want <= got: every expected rule must be cited, and an extra one is allowed. missing_fields uses want == got: the output must name exactly the expected fields. An agent that cites R-ROOF-01 and also R-HUMAN-01 passes the first. An agent that reports one extra missing field fails the second.

Neither choice is the correct one in general. An assertion that is too strict fails agents that did the right thing in a way the author did not foresee. An assertion that is too loose passes agents that did the wrong thing. The docstring states the choice, and that is the minimum: whoever reads a pass must be able to find out what was allowed.

The loose choice has a cost in this example. Because extra citations pass, the citations grader alone cannot catch an agent that cites a rule it never read. Another grader, required_process, covers that gap. A set of assertions is designed as a whole.

From several scores to one pass or failAll must pass, a weighted total, or a mix; and what one trial supports

The example passes a trial only if every grader passes. That is one of three common rules. Anthropic's article names them: binary (all graders must pass), weighted (the combined scores must reach a threshold) and hybrid.

RuleA trial passes whenWhat it hides
All must passEvery grader passesNothing, but one strict grader can fail many trials for a small reason
Weighted totalThe weighted sum reaches a thresholdA serious failure can be outweighed by good scores elsewhere
MixedSome graders must pass; the others are summed against a thresholdLess, if the must-pass list holds the failures that nothing may outweigh

The example is mixed one level down. Inside the rubric grader, five criteria are weighted and summed against 0.70, and two conditions cap the value at 0.20 whatever the sum is. Grading and rubrics is the home of weights and of those conditions.

A second point for readers who watch the numbers. The run of UW-001 alone prints 95% interval 0.21 to 1.00 for one pass in one trial. That range is the honest size of the claim: after one passing trial, a true pass rate anywhere from about 0.21 to 1.00 is compatible with what was seen, under the method that Reliability and statistics explains.

Baselines, fixtures and thresholds in productionThree ways a comparison stops being a comparison

A baseline is only as good as what was pinned with it. The comparison in section 7 is fair because four things were the same on both sides: the cases, the graders, the fixture and the seed. Only the agent differed. If a grader was tightened between the baseline run and the new run, a drop in the pass rate is a change in the measurement, not a regression in the agent. A stored baseline should carry the versions of the cases, the graders and the fixture next to its numbers. This is a design this site proposes; the example's report records the suite, the variant and the seed, and not yet a grader version.

A fixture trades realism for repeatability. The stand-in provider always answers 31 years for UW-001. A real provider can be slow, wrong or changed. A suite that passes on a fixture is evidence about the agent on that fixture first. The rules in the fixture also age: a case labelled under version 2025.4 of the policy fixture has an expected result that version 2026.2 may contradict.

A threshold chosen after the result is not a threshold. If the line is set once the pass rate is known, it will be set where the result passes. The line belongs in a file that is under review, set before the run, with the reason for its value. Moving it is a change like any other and shows in the history.

A seed repeats a run; it does not make the agent stable. The example prints seed 7, and the same seed gives the same report because the stand-in draws its variation from that seed. A model behind an API gives no such promise in general. Treat a seed as a way to rerun the example, not as a property of an agent.

Production pitfall. The most common misuse of these words is a metric reported without its parts: "the eval is at 86%". Ask for the suite, the number of cases and trials, the graders, the threshold and the baseline. In the example, 0.86 is 25 of 29 trials, and 20 of those trials are one case.

In one breath#

An eval starts as one assertion: for this input, the output must match an expected result that people wrote down. Add more graders and each trial gets scores; add a threshold and a score becomes pass or fail. Run each case of a suite as trials on a fixed fixture, compute a metric over the scores, and hold it against a threshold and a baseline. Then "it works" turns into a sentence with numbers that someone else can check, and a regression is visible as the case that passed before and fails now.

What comes next#

The engineer can now state what was run, how often, against which line and compared with what, and the changed agent no longer hides behind one decline on a screen. One part of the result was taken on trust, though. Two graders failed the change, and one of them, rubric, gave a number for the quality of written reasons. Who decides what a good reason is, how much each quality counts, and which failure no good score may outweigh? That is the work of Grading and rubrics.

The same idea for a coding agent: the input is an issue, the expected result is a set of tests that must pass, a trial is one attempt at the fix, and a regression is a test that passed before the change and fails after it.