An engineer at the insurer is writing the program that runs the evals. She starts with case UW-006. A broker asked for cover on a masonry office with a 12-year-old roof, and in this case the roof-data provider never answers. The underwriting agent is expected to try the provider twice, then recommend request_info and name roof_assessment as missing. The model recommends; a person decides.
She runs the case twice in a row, to see whether the result holds. The first time it passes. The second time it fails, with this reason: "the recommendations row is missing or repeated".
The agent did the same thing both times: two tries, the same controlled fallback, the same recommendation. What changed was the table it wrote into. The row from the first time was still there when the second one started, so the check that expects one row found two.
Nothing is wrong with the agent, and the report says it failed. The team would now search the agent for a fault that lives in the runner.
The earlier pages taught what to measure: cases, graders, traces, judges, repeated trials. Each assumed that some program feeds the cases to the agent, gives every trial a clean start, calls the graders and adds up the results. This page is about that program. Its faults are quiet. They crash nothing; they change the number.
1. Two programs are called a harness#
The word "harness" is used for two different programs, and they sit on opposite sides of the measurement.
The first is the software that makes a model into an agent. It sends the request to the model, runs the tool the model asked for, sends the result back, and repeats until the work is done. It also retries a failed tool and records what happened. This is the agent harness. It ships to production: when a broker's email arrives on a real day, this is the code that runs.
The second is the software that tests the first. It takes a case, starts the agent on the case's input, waits for it to finish, hands what happened to the graders, and writes the scores down. This is the eval harness. It runs on an engineer's laptop and in the build pipeline. On this site its short name is the runner.
Plain English. The agent harness is the car. The eval harness is the test track with its stopwatch. A fault in either one changes the lap time.
The two names and their meaning come from Anthropic's article "Demystifying evals for AI agents", which describes an evaluation harness as the software that runs an eval from start to finish, and an agent harness as the software around a model that lets it act as an agent. It adds a point that matters below: evaluating "an agent" means evaluating the model and its agent harness together. The eval harness stands outside both.
The agent harness has its own site, with the loop, the tools, state, limits and permissions each in a chapter: harness.sammuthu.com. This page needs only the seam between the two programs.
Diagram: Two harnesses and the world between them
Three lanes, three owners. The eval harness measures. The agent harness runs the model and its tools. The world is everything the agent touches; in an eval, the eval harness builds it new for each trial and discards it afterwards. Follow the arrows from the cases on the left to the report on the right.
- Eval harness · measures
- Load cases: Reads the case files and keeps the cases of the suite that was asked for.
- Reset the world: Builds a new starting point for every trial, so nothing is left over from the trial before.
- Grade: Hands the output, the trace and the end state of the trial to the graders the case names.
- Report: Adds up the scores of all trials and writes them down with the suite, the variant and the seed.
- Agent harness · runs the agent
- Model loop: Calls the model, runs the tool it asked for, sends the result back, and repeats until the work is done.
- Tool calls, retries, limits: Decides what happens when a tool is slow or fails. This behavior is part of what the eval measures.
- Trace: The ordered record of the trial, written by the agent side as it works.
- The world · built per trial
- Stand-in tools: Small programs with the names and shapes of the real tools. They answer from data stored in the case.
- Stored rows: What is true after the trial: the rows the agent and the underwriter left in the tables.
Ask of each box who owns it. The eval harness owns the cases, the reset, the grading and the report. The agent harness owns the loop, the tool calls and the trace. Between them sits the world: the tools the agent calls and the rows it leaves behind. In an eval, the eval harness builds that world and throws it away afterwards. In production, the world is real.
| Question | Agent harness | Eval harness |
|---|---|---|
| What is its job? | Run the model, its tools and its limits so the work gets done | Run the agent on known cases and score what happened |
| When does it run? | On every real submission | Before a release, and when someone asks whether a change helped |
| What does it produce? | A recommendation, stored rows, a trace | Scores and a report |
| In the example | agent_sim.py (a scripted stand-in for it) | run_evals.py |
When a trial fails, the first question is which column the fault is in. The opening story was a fault in the right-hand column that looked like one in the left.
2. What the runner does, step by step#
Before the code, the shape. A trial is one run of one case. Every trial has the same six steps, shown here as messages between four parties: the eval harness, the world it builds, the agent under test, and the graders.
Diagram: One trial of UW-006 · six steps, four parties
Time runs down the page. The eval harness is on the left and every message passes through it. The agent and the graders never exchange a message, and the agent never sees the expected result.
- Eval harness (the runner): 1 · Load the case. Reads the case file: the input, the world block, the expected result and the names of its graders.
- Eval harness (the runner) to World: 2 · Build a fresh world. A new world for this trial only: clock at zero, empty tables, tool answers taken from the case.
- Eval harness (the runner) to Agent (under test): 3 · Start the trial: the submission. The agent gets the input, a variant, a seed and a trial number. It does not get the expected result.
- Agent (under test) to World: Tool calls. The agent calls its tools as it would in production. Here the world answers in place of the real services.
- World to Agent (under test): Stand-in answers, or a timeout. Each answer is computed from the case. A failure the case asks for happens every time, in the same way.
- Agent (under test) to Eval harness (the runner): 4 · Output, trace, end state. The record of the trial: what the agent said, what it did, and what the tables hold afterwards.
- Eval harness (the runner) to Graders: 5 · The case and the record. Only now is the expected result used. The graders read; they change nothing.
- Graders to Eval harness (the runner): One result per grader. Each grader answers pass or fail with a reason. The trial passes only if all of them pass.
- Eval harness (the runner): 6 · Report and trace file. The score goes into the report with the scores of the other trials. The trace is stored in its own file.
The agent and the graders never exchange a message. Everything passes through the eval harness, which is what lets it keep the expected result away from the agent and one trial away from the next.
Now the code. The example's eval harness is examples/uw-evals/run_evals.py, 159 lines of standard-library Python. The agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is. No model is called. The excerpts below are copied from the files unchanged.
Step 1: load the cases#
A case is one input with its expected result and labels. A suite is a named set of cases with one purpose. The first function shows that selecting cases is plain data handling: a case says in its own suites field which suites it belongs to, and the runner only filters.
def load_cases(suite: str = "all", case_id: str | None = None) -> list[dict[str, Any]]:
"""Read cases/*.json. A suite is a named set of cases with one purpose."""
cases = [json.loads(p.read_text()) for p in sorted((HERE / "cases").glob("UW-*.json"))]
if case_id:
return [c for c in cases if c["id"] == case_id]
return [c for c in cases if suite == "all" or suite in c["suites"]]
The files are read in sorted order, so the cases always run in the same order. The runner does not know what smoke or regression mean; which suite runs on which change belongs to CI gates.
Step 2: reset the world#
A trial needs a starting point: what the tools will answer, what the tables contain, what time it is. The fixed data a trial starts from is called a fixture. Here it is the policy fixture plus the world block of the case. For UW-006 that block says one thing: the roof assessment has the status timeout.
The runner must rebuild that starting point for every trial. In the example it is one object, and the excerpt is the start of its class in tools_sim.py. It shows that a reset can be as small as a constructor, as long as every piece of changeable state lives inside the object.
class World:
"""Everything outside the agent for one trial: tool answers, the clock and the stored rows.
A new World is built for every trial. That is the reset between trials.
"""
def __init__(self, case: dict[str, Any]) -> None:
self.world: dict[str, Any] = case["world"]
self.version: str = case["policy_version"]
self.now_ms = 0
self.roof_attempts = 0
self.state: dict[str, list[dict[str, Any]]] = {"recommendations": [], "decisions": [], "audit_events": []}
A new World starts with the clock at zero, the count of provider attempts at zero, and three empty tables. Nothing from an earlier trial can be in there, because the object did not exist then.
Under the hood. In the example the lineworld = World(case)sits insiderun_agentinagent_sim.py, so the stand-in agent creates its own world. That keeps the example short. With a real agent, the eval harness builds the world and hands the agent only the addresses of its tools.
Step 3: run one trial#
This function is the center of the file. It shows the contract between the runner and the agent in one call: the runner gives a case, a variant, a seed and a trial number, and gets back a record of what the agent said, did and left behind.
def run_trial(case: dict[str, Any], variant: str, seed: int, trial: int) -> dict[str, Any]:
"""One trial: run the agent in a fresh world, then grade what it said, did and left behind."""
record = run_agent(case, variant=variant, seed=seed, trial=trial)
record["graders"] = grade_trial(case, record)
record["passed"] = all(g["passed"] for g in record["graders"])
record["hard_fails"] = [h for g in record["graders"] for h in g.get("hard_fails", [])]
spans = record["trace"]["spans"]
record["duration_ms"] = spans[0]["duration_ms"]
record["simulated_tokens"] = sum(s["attributes"].get("simulated_tokens", 0) for s in spans)
return record
The first line of the body is the trial. run_agent returns a dictionary with three things the graders need: output (what the agent said), trace (what it did) and end_state (the three tables as they are afterwards). Those are the three things to grade that What to evaluate separated. The runner also passes in seed and trial, so the agent does not pick its own randomness; section 8 explains why.
Step 4: collect the trace#
A trace is the ordered record of one trial: one entry, called a span, for each model call, retrieval, tool call and check. The agent side writes it: run_agent returns tracer.export() inside the record. The runner's part is to keep it, one file per trial, named so that a person can find the trial again. The function below shows that, and that the report and the traces are separate files, so the report stays small enough to read.
def write_outputs(report: dict[str, Any], case_results: list[dict[str, Any]], json_path: Path | None) -> None:
"""Write out/report.json and one trace file per trial."""
shutil.rmtree(OUT / "traces", ignore_errors=True)
(OUT / "traces").mkdir(parents=True)
for c in case_results:
for r in c["records"]:
trace = {"case_id": c["id"], "trial": r["trial"], "variant": r["variant"], "seed": r["seed"], **r["trace"]}
(OUT / "traces" / f"{c['id']}-{r['trial']:02d}.json").write_text(json.dumps(trace, indent=2) + "\n")
for path in {OUT / "report.json", json_path} - {None}:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(report, indent=2) + "\n")
Each trace file carries the case id, the trial number, the variant and the seed next to the spans: the four values needed to produce the same trial again. The structure of a trace is on Agent traces.
Step 5: grade#
A grader is the thing that gives a score: code, a model or a person. The runner contains no grading logic. This function from graders.py shows that the list of graders is part of the case, not part of the runner.
def grade_trial(case: dict[str, Any], trial: dict[str, Any]) -> list[Result]:
"""Run every grader the case asks for. A trial passes only if all of them pass."""
return [GRADERS[name](case, trial) for name in case["graders"]]
UW-006 asks for seven graders: six that every case uses, and retry_and_fallback, which counts the provider attempts in the trace. Back in run_trial, one line turns seven results into one: the trial passes only if every grader passes.
Step 6: report#
One trial gives a score. A report needs a metric, which is a number computed over many scores. The next function runs all trials of one case. It shows where "many trials" enters the runner: the number is a field of the case, 20 for UW-010 and 1 for the others, and the command line can override it.
def run_case(case: dict[str, Any], variant: str, seed: int, trials: int | None = None) -> dict[str, Any]:
"""All trials of one case, with the per-case numbers."""
n = trials or case.get("trials", 1)
records = [run_trial(case, variant, seed, t) for t in range(1, n + 1)]
passed = [r["passed"] for r in records]
low, high = stats.wilson_interval(sum(passed), n)
return {"id": case["id"], "title": case["title"], "tags": case["tags"], "critical": case["critical"],
"n": n, "passes": sum(passed), "pass_rate": round(stats.pass_rate(passed), 6),
"interval": [round(low, 6), round(high, 6)], "pass_any": any(passed), "pass_all": all(passed),
"records": records}
Why a pass rate from 20 trials needs an interval is the subject of Reliability and statistics. Two more functions, summarize and build_report, add the totals over all cases.
The six steps, run#
The command below runs all ten cases. A successful run proves that the six steps connect: every case loaded, every trial got a world, every grader was called. It proves nothing about a model.
cd examples/uw-evals
PY=/opt/homebrew/bin/python3
$PY run_evals.py --suite all
Eval run · suite all · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-001 1 1 1.00 decline -
UW-002 1 1 1.00 approve -
UW-003 1 1 1.00 request_info -
UW-004 1 1 1.00 escalate -
UW-005 1 1 1.00 decline -
UW-006 1 1 1.00 request_info -
UW-007 1 1 1.00 decline -
UW-008 1 1 1.00 decline -
UW-009 1 1 1.00 approve -
UW-010 20 16 0.80 escalate x16, approve x4 recommendation_match (4), citations (4)
trials 29 · passed 25 · pass rate 0.86 · 95% interval 0.69 to 0.95 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 29 trace file(s) in out/traces/
The first line matters as much as the last. It states the suite, the variant of the agent, the seed and what the agent was. A pass rate without those facts cannot be compared with another. Every number in this output is synthetic.
| Step | In the example | If the runner gets it wrong |
|---|---|---|
| 1 · Load cases | load_cases | Two runs measure different cases |
| 2 · Reset the world | World | A trial inherits state from the trial before |
| 3 · Run a trial | run_trial | The agent picks its own inputs or randomness |
| 4 · Collect the trace | write_outputs | A failure can be counted, not explained |
| 5 · Grade | grade_trial | Grading logic hides in the runner |
| 6 · Report | run_case, summarize, build_report | A number with no record of what produced it |
The rest of this page works through the right-hand column.
3. Why a leftover row failed the second trial#
The engineer's first runner made one mistake: it built the world once and used it for every trial.
The script below repeats that mistake on purpose. It was written for this page and run against the example. It runs UW-006 twice with a fresh world per trial, which is the correct design, then twice with one shared world, which is the mistake. A successful run proves that the same agent behavior gets two different scores, depending only on what the runner did between trials.
# A deliberate mistake, to show what the reset prevents: two trials share one World.
# Written for this page and run against examples/uw-evals; not part of that folder.
import agent_sim
from run_evals import load_cases, run_trial
from tools_sim import World
case = load_cases(case_id="UW-006")[0]
def two_trials(label: str) -> None:
for t in (1, 2):
r = run_trial(case, "careful", 7, t)
failed = [f"{g['name']} ({g['detail']})" for g in r["graders"] if not g["passed"]] or ["-"]
print(f"{label} · trial {t} · passed {r['passed']} · rows in recommendations "
f"{len(r['end_state']['recommendations'])} · failed: {'; '.join(failed)}")
two_trials("fresh world ")
shared = World(case)
agent_sim.World = lambda _case: shared # the mistake: every trial now gets the same World
two_trials("shared world")
fresh world · trial 1 · passed True · rows in recommendations 1 · failed: -
fresh world · trial 2 · passed True · rows in recommendations 1 · failed: -
shared world · trial 1 · passed True · rows in recommendations 1 · failed: -
shared world · trial 2 · passed False · rows in recommendations 2 · failed: human_required (the recommendations row is missing or repeated)
The fourth line is the opening story. The grader human_required expects exactly one row in recommendations. With a shared world it finds two. The grader is right to fail what it sees; the runner showed it the wrong thing.
The property that was missing has a name. Isolation means that nothing one trial does can change what another trial sees. The Anthropic article states the rule: each trial should start from a clean environment, because shared state can make failures depend on each other and can also raise scores.
The second direction is the dangerous one, because nobody investigates a pass. Suppose the provider's answer were cached in the world. With a shared world, a later trial of UW-006 would find a roof age in the cache, never meet the timeout, and pass a case about timeouts without handling one. A broken reset can make an agent look worse or better. Both are wrong numbers.
The three walls of isolationTrial from trial, agent from answer key, run from run: how the example does on each
| Wall | What must not cross it | In the example |
|---|---|---|
| Trial from trial | Rows, counters, caches, files, the clock | A new World per trial. Solid |
| Agent from answer key | The expected result, the labels, the list of graders | run_agent receives the whole case, including expected. The script reads only input, world, policy_version and agent_behaviour. A real agent must be handed the input alone |
| Run from run | Output files, ports, shared accounts | write_outputs deletes out/traces before it writes. Two runs of the harness in the same folder at the same time would overwrite each other |
The second wall is the one a real model tests hardest. A scripted agent reads only the fields its author chose. A model reads everything it is given, so an expected result that sits in the same directory, the same prompt or the same tool answer as the input is part of the input.
Common confusion. A failed trial and a broken trial are different results. A failed trial means the agent ran and a grader said no. A broken trial means the measurement itself went wrong: the world was dirty, the machine ran out of memory, the harness crashed. Section 6 comes back to this.
4. Stand-in tools or the real ones#
The world the runner builds contains tools. The agent calls get_roof_assessment, and something has to answer.
The runner can connect the agent to the real provider. Or it can connect it to a small program with the same name, inputs and outputs, which answers from data inside the case. That program is a stand-in tool. In the example every tool is one: World.get_roof_assessment reads the case's world block and either returns the roof record stored there or raises a timeout.
UW-006 shows why stand-ins exist. The case needs the provider to stay silent for two attempts, and no one can ask a real provider to fail on request. A stand-in fails whenever the case says so, every time, in the same way.
So a stand-in buys control: the same answer every time, failures on demand, no cost per call, no side effects in someone else's system. What it gives up is the truth about the provider. It answers the way the team believed the provider answered on the day the stand-in was written. If the provider later renames a field, the eval keeps passing against a world that no longer exists.
The practical answer is to use more than one. Most trials run against stand-ins, because most questions are about the agent. A few scheduled checks run against the real interface and ask one narrow question: does the real tool still answer in the shape the stand-in assumes?
| Choice | What it gives | What it cannot tell you |
|---|---|---|
| Stand-in tool | Repeatable trials, staged failures, no cost, no side effects | Whether the real provider still behaves this way |
| The provider's test environment, if it has one | The real interface without real customers | Rare failures, real latency |
| The real tool | The truth today | The same truth tomorrow; it costs, it varies, and its writes must be made harmless |
Production pitfall. A result against a stand-in is evidence about the agent on the assumption that the stand-in is faithful. Write that assumption next to the number: "25 of 29 trials passed against stand-in tools".
5. Replay: pay for a trial once#
A trial of a real agent costs model calls and minutes. When a grader changes, a team does not want to pay for every trial again. The answer is to store what happened and use the stored copy. This is replay: using a stored record of a trial in place of running it live.
The most useful kind is replay for grading. The runner stores the whole record of a trial (output, trace, end state) and runs the graders on it again later. The agent does not run. This is safe whenever only the graders changed, because graders only read.
The script below does this for UW-008 under the shortcut variant, the trial in which the agent writes the right recommendation without retrieving the rule it cites. It was written for this page and run against the example. A successful run proves that a record read back from disk gets the same six grader results as the live trial, with no agent and no tools involved.
# Replay for grading: store a trial once, grade it again later without running the agent.
# Written for this page and run against examples/uw-evals; not part of that folder.
import json
from pathlib import Path
from agent_sim import run_agent
from graders import grade_trial
from run_evals import load_cases
case = load_cases(case_id="UW-008")[0]
stored = Path("out/UW-008-shortcut.json")
stored.write_text(json.dumps(run_agent(case, variant="shortcut"))) # the expensive part, done once
record = json.loads(stored.read_text()) # later: no agent, no tools
for g in grade_trial(case, record):
print(f"{g['name']:<22}{'pass' if g['passed'] else 'FAIL':<6}{g['detail']}")
recommendation_match pass expected decline, got decline
citations pass expected R-ROOF-01; got R-ROOF-01
missing_fields pass expected none; got none
required_process FAIL search_guidance was not called; cited but never retrieved: R-ROOF-01
human_required pass recommendation stored; decision left to a person
rubric FAIL hard-fail (cites_unretrieved_rule): 1.00 capped at 0.20
These are the results the README of the example reports for the live trial: required_process and rubric fail, the output graders pass. If the team now tightens required_process, it can grade every stored trial again in seconds and see which old passes turn into failures.
The script had to store the full record. The example's own out/traces/ files hold the spans but not the end state, so they are enough to grade the trajectory again and not the end state. A runner built for replay stores all three parts.
The other replay: stored tool answers, and where it stops workingStored answers fit only the questions the first trial asked
Replay of tool answers stores what each tool returned during one trial and feeds those answers to a new trial. It removes the cost and the variation of the tools.
Its limit is strict: the stored answers fit only the calls the first trial made. If the changed agent searches with a different query, or calls the provider a third time, there is no stored answer for that call. The replay must then fail loudly. Answering with the nearest stored answer would grade the agent on evidence it never asked for.
A stand-in tool does not have this limit. A replay stores answers. A stand-in stores data and computes the answer for whatever is asked: World.search_guidance ranks the rules of the policy fixture for any query text. That is why the example can run the shortcut variant, which makes different calls than careful, on the same cases.
| Replay of tool answers | Stand-in tool | |
|---|---|---|
| Stores | The answers one trial received | The data the answers are computed from |
| When the agent makes a new call | No answer exists; the trial must be marked incomplete | Computes an answer |
| Effort | Low: record and play back | Higher: someone writes and maintains the program |
| Best for | Looking at one old trial again, exactly as it happened | Evaluating a changed agent |
6. Two clocks, and a third kind of result#
UW-006 is a case about a timeout, and it is tempting to think the timeout is the runner's concern. It is not. An eval has two clocks, and they belong to the two harnesses.
The first clock belongs to the agent side. The stand-in provider gives up after 2,000 ms, and the agent waits 500 ms and tries once more. Those limits are part of the behavior under test. The trace of UW-006 records them: one get_roof_assessment span that starts at 918 ms and ends in an error after 2,000 ms, a second at 3,418 ms with the same result, and a whole trial of 7,474 ms. The times come from a simulated clock. The grader retry_and_fallback reads these spans. The runner does not interfere.
The second clock belongs to the eval harness: how long may one trial take before the runner stops waiting? A real agent can loop and a tool can hang. Without a limit per trial, one stuck trial blocks the whole run.
The example's runner has no such clock. run_trial contains no time limit and no try; with a scripted agent it does not need them, and if run_agent raised an exception the whole run would stop without a report. A runner for a real agent needs both, and one more thing that is easy to miss: a third kind of result.
| Result of a trial | Meaning | Counts against the agent? |
|---|---|---|
| Pass | The agent ran and every grader passed | No |
| Fail | The agent ran and a grader did not pass | Yes |
| Error | The trial could not be completed or graded: harness timeout, crash, a grader that threw | No. It counts against the run |
A harness timeout is the hard one to classify: an agent that never stops is an agent fault, and a machine that was too slow is not. The runner cannot always tell which happened. It can report errors apart from failures, with the trace as far as it got, and refuse to state a pass rate when errors are more than a small share of the trials. The Anthropic article makes the underlying point: a failure caused by the infrastructure says nothing about the agent, and several trials that fail for the same infrastructure reason are not independent evidence.
7. Trials side by side#
The example runs its 29 trials one after another. With a real model each trial takes seconds to minutes, most of it spent waiting for a reply, so running trials at the same time is the obvious way to shorten a run.
Whether that is safe depends on isolation. Two trials running at once share whatever the runner lets them share. If each trial has its own world, its own randomness and its own place in the results, the order in which trials finish cannot matter.
The script below checks that for the example. It was written for this page and run against the example. It runs the 29 trials on eight threads, runs them again one after another, and compares every record. A successful run proves that the results do not depend on scheduling. It does not prove a speed-up: the scripted agent finishes too quickly for threads to help.
# Trials in parallel. Written for this page and run against examples/uw-evals; not part of that folder.
from concurrent.futures import ThreadPoolExecutor
from run_evals import DEFAULT_SEED, load_cases, run_case, run_trial
cases = load_cases("all")
jobs = [(c, t) for c in cases for t in range(1, c.get("trials", 1) + 1)]
with ThreadPoolExecutor(max_workers=8) as pool:
records = list(pool.map(lambda job: run_trial(job[0], "careful", DEFAULT_SEED, job[1]), jobs))
parallel = {(r["case_id"], r["trial"]): r for r in records}
sequential = {(r["case_id"], r["trial"]): r
for c in cases for r in run_case(c, "careful", DEFAULT_SEED)["records"]}
print(f"trials {len(parallel)} · passed {sum(r['passed'] for r in parallel.values())}")
print("same records as the sequential run:", parallel == sequential)
trials 29 · passed 25
same records as the sequential run: True
Three properties of the example made this work, and each is a requirement for any runner that goes parallel.
- Each trial builds its own
World, so no rows, counters or clock are shared. - Each trial gets its own random stream.
run_agentcreates it from the seed, the case id and the trial number, so trial 14 behaves the same whether it runs first or last. - Each record carries its case id and trial number, so results are matched by identity, not by the order in which they arrive.
What parallel trials break in a real systemRate limits, shared accounts, and failures that stop being independent
The example has no outside dependency, so the list above is complete for it. A runner for a real agent meets four more problems.
Rate limits become trial results. Forty trials at once can exceed what the model provider or a tool provider allows. The agent then sees rejected calls it would not see in production at normal load. If its retries cope, the trial is slower. If they do not, the trial fails for a reason that has nothing to do with the case. The runner needs a ceiling on how many trials run at once, and it needs to record a rejected call so that a person reading the trace can tell this failure from a real one.
Shared resources leak state sideways. Section 3 dealt with state that leaks forward in time, from one trial to the next. Parallel trials can also leak sideways: two trials writing to the same test database, the same sandbox account or the same temporary folder. The fix is the same idea with a stricter rule: every trial gets resources that are named after the trial, and the names never repeat.
Failures stop being independent. The statistics on Reliability and statistics assume that each trial is an independent look at the agent. If ten parallel trials fail because one machine ran out of memory, that is one event counted ten times. An interval computed from those trials is too narrow. The runner should report errors by cause, so that one cause appearing in many trials at the same moment is visible.
Latency numbers change meaning. A trial's duration measured under heavy parallel load includes waiting that the runner caused. If latency is one of the things being evaluated, measure it in a separate run at a realistic load.
8. Same inputs, same report#
Section 2 left a question open: why does the runner pass a seed and a trial number to the agent?
A seed is a starting number for a random generator: the same seed gives the same sequence of "random" choices. The stand-in agent uses randomness to imitate an agent that varies; in UW-010 it is scripted to skip the escalation in about one trial in five. If that randomness were left free, every run of the harness would give a different report, and no one could tell a change in the agent from a change in luck.
So the runner fixes it. With the same cases, code and seed, the report must be the same, byte for byte. This property is reproducibility: a run of the harness can be repeated and gives the same result. The commands below check it. A successful run proves that nothing in the runner depends on the time of day.
$PY run_evals.py --suite all --json a.json > /dev/null
cp -R out/traces traces-a
$PY run_evals.py --suite all --json b.json > /dev/null
cmp a.json b.json && echo "reports identical"
diff -r traces-a out/traces && echo "29 trace files identical"
reports identical
29 trace files identical
The trace files are identical too, including every duration, because time comes from the simulated clock in World.
Now change the seed and nothing else.
$PY run_evals.py --suite all --seed 8
UW-010 20 15 0.75 escalate x15, approve x5 recommendation_match (5), citations (5)
trials 29 · passed 24 · pass rate 0.83 · 95% interval 0.65 to 0.92 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 15 of 20 trials passed
wrote out/report.json and 29 trace file(s) in out/traces/
Only the last four lines of the output are shown; the nine single-trial cases pass as before. UW-010 now passes 15 of 20 trials where seed 7 gave 16. The agent did not change. A seed makes a run repeatable. It does not make the measured rate the true rate, and Reliability and statistics explains what to do about that.
Common confusion. Reproducible does not mean deterministic agent. With a real model the runner cannot fix what the model writes. What it can do is fix everything else: the cases, the fixture, the tool answers, the graders. Then the variation that remains in the report is the agent's own, and that is the thing the eval set out to measure.
What a report must say about itself#
Reproducing a run later needs more than a seed. It needs to know which version of everything produced the number. Here is the meta block of the example's report, copied from out/report.json.
{
"suite": "all",
"variant": "careful",
"seed": 7,
"agent": "scripted stand-in (no model called)",
"synthetic": true
}
It names the suite, the agent variant and the seed. It does not name the version of the graders, the rubric or the cases. For a teaching example in one folder under version control, the commit covers that. For a real runner it is a gap with a specific cost. Suppose the pass rate rises from one week to the next. If someone loosened a grader in between, the agent did not get better; the measuring stick got shorter. Without grader versions in the report, nobody can tell the two explanations apart afterwards.
What a report must record, and what to do when a grader changesEight things to write down; grade stored trials with both versions before trusting a trend
| The report should record | Why | In the example's report |
|---|---|---|
| Suite and the ids of the cases that ran | Two pass rates over different cases are not comparable | Yes: meta.suite and the cases list |
| Version of the case set | A relabelled case changes what "pass" means | No |
| Agent version: model id, prompt version, agent harness version | This is the thing under test | Partly: variant |
| Grader and rubric versions | A changed grader changes scores of an unchanged agent | No |
| Judge model and judge prompt version, if a judge is used | A judge is a model and changes when either is changed | No. The judge is a scripted stand-in |
| Fixture versions | The expected result depends on the policy version | Per case: policy_version |
| Seed, trials per case | To repeat the run | Yes: meta.seed, n per case |
| Stand-in or real tools | It limits what the number can claim | Yes, for the agent: meta.agent |
A grader change is a change to the instrument. The discipline is the same as for any instrument: measure the same thing with the old and the new version, and look at the difference before using the new one alone.
Replay for grading (section 5) makes this cheap. Take the stored trials of the last accepted run. Grade them with the old grader version and with the new one. The agent's behavior is identical in both, since it is the same stored record, so every difference in score is caused by the grader. Three outcomes are possible.
| What the two gradings show | Reading | Action |
|---|---|---|
| No trial changes its result | The change did not affect these trials | Adopt the new version; keep the trend line |
| Some passes become failures | The new grader is stricter, or it has a fault | Read each changed trial. If the new failures are right, the old pass rate was too high: restate the baseline with the new grader |
| Some failures become passes | The new grader is looser, or the old one had a fault | Read each changed trial with more suspicion. A looser grader is the change most likely to hide a regression |
Two rules follow. First, a trend line must not cross a grader change without a mark: restate the baseline under the new version, or start a new line. Second, a pull request that changes the agent and a grader together cannot be evaluated by one run, because the run cannot say which change moved the number. Split it, or grade the old agent's stored trials with the new grader first.
The same reasoning applies to a judge. A new judge model or a new judge prompt is a new grader version, and its agreement with human labels has to be measured again. Judges and calibration covers how.
This is a design this site proposes. The example does not implement grader versions.
9. What a run costs#
A run of the example costs nothing: no model is called. A run against a real agent has a price in tokens and minutes, and the runner decides that price, because it decides how many trials happen.
To make the growth visible, the stand-in agent attaches an invented token count to its two model steps, so every trial "uses" 2,160 simulated tokens. The 29 trials of the full run come to 62,640.
Cost has three multipliers: the number of cases, the trials per case, and what one trial uses. The second is the one that surprises people. The release gate of the example (gate.py, explained on the next page) answers BLOCK on the full run, because 16 passes in 20 trials of UW-010 leave the interval too wide. More trials narrow it. The command runs the same case 80 times and then asks the gate.
$PY run_evals.py --case UW-010 --trials 80 && $PY gate.py
Eval run · suite UW-010 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-010 80 65 0.81 escalate x65, approve x15 recommendation_match (15), citations (15)
trials 80 · passed 65 · pass rate 0.81 · 95% interval 0.71 to 0.88 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 65 of 80 trials passed
wrote out/report.json and 80 trace file(s) in out/traces/
Gate · suite UW-010 · variant careful · threshold 0.70
PASS: 65 of 80 trials passed (0.81); the 95% interval is 0.71 to 0.88. The lower bound 0.71 reaches the threshold 0.70, and no critical case failed.
The observed pass rate hardly moved, from 0.80 to 0.81. The decision changed from BLOCK to PASS, because the lower bound of the interval rose from 0.58 to 0.71. The table puts the price next to that gain. Tokens and time are read from summary in out/report.json. Every value is from the example at seed 7, and synthetic.
Trials of UW-010 | Passed | 95% interval | Simulated tokens | Simulated agent time |
|---|---|---|---|---|
| 20 | 16 | 0.58 to 0.92 | 43,200 | 63 s |
| 80 | 65 | 0.71 to 0.88 | 172,800 | 251 s |
| 320 | 257 | 0.76 to 0.84 | 691,200 | 1,003 s |
Each row costs four times the row above it and makes the interval about half as wide. That ratio is why a team cannot run everything, many times, on every change.
Two costs are not in the table. A judge, when it is a real model, is one more model call for every trial it grades. And people cost the most: someone has to read the failed trials. The Anthropic article insists on reading the records of many trials, because that is the only way to find out whether the graders themselves are right. A runner that puts the trace of a failed trial one click away saves that person's time.
Spending a fixed budget of trialsWhere extra trials buy information and where they buy nothing
Given a fixed number of trials to spend, three rules decide where they go. These are design recommendations, not measurements.
Trials go to cases whose result varies. In the example, UW-001 to UW-009 are critical cases that the stand-in passes every time, and UW-010 is the measured case. One more trial of UW-010 narrows an interval. One more trial of a case that has passed every time so far gives much less, though not nothing: a single pass does not show that a case always passes, and a critical case that fails once in fifty trials is found only by running it more than once. A reasonable split is one or a few trials for critical cases on every change, and a deeper, less frequent run that repeats them.
Cheap graders run first. A trial that fails recommendation_match in code does not need a model judge to grade its prose for the purpose of pass or fail. Skipping the judge on trials that already failed a code grader saves judge calls. The trade-off is that the rubric score for those trials is missing from the report, which matters if someone tracks the rubric score as a metric on its own.
Stored trials are free to grade again. Any question that can be answered by grading stored records again (section 5) should be, before a single new trial is paid for.
One caution about the formulas behind sample-size planning: pass@k = 1 − (1 − p)^k and pass^k = p^k hold only if the trials are independent and each one succeeds with the same probability p. Parallel trials that fail together for an infrastructure reason break the first condition, which is one more reason to count errors apart from failures.
In one breath#
An eval harness is the program that measures an agent from outside: it loads cases, builds a clean world for every trial, runs the agent, keeps the trace, calls the graders and writes a report. It is a different program from the agent harness, which is the runtime that lets the model act. A score is an observed result, and the runner is the instrument that observes it, so every fault in the runner becomes a fault in the number. The three properties to demand are isolation between trials, a report that says exactly what produced it, and the same result when the same inputs are run again.
The same idea for a coding agent: the world is a fresh copy of a repository in a clean container per trial, the stand-in tools are a package index and a test database that answer the same way every time, and the end state the graders read is whether the test suite of that copy passes.
What comes next#
You can now read a runner and say what each part is for, tell an agent fault from a runner fault, and name what a report must record before its number can be compared with another. The example's runner is deliberately small: it has no trial timeout, no error result and no grader versions, and this page said where each would go.
One problem is left open. A narrower interval costs trials, and trials cost tokens and minutes, so no team can run every case eighty times on every change. Someone has to decide which suites run when, which results may stop a release, and what to do with a case that passes on Monday and fails on Tuesday. That is the subject of CI/CD quality gates.