All chapters
  1. Start hereA test checks an answer. An eval measures a behavior
  2. Foundations · 1Primitives: the small words every eval is built from
  3. Foundations · 2Grading and rubrics: who decides what good means
  4. Foundations · 3Golden datasets: the cases you trust
  5. Mechanics · 4What to evaluate: the answer, the path, and what was left behind
  6. Mechanics · 5Agent traces: the record you grade the path from
  7. Mechanics · 6Judges and calibration: when a model grades a model
  8. Mechanics · 7Confidence and uncertainty: five numbers that are not the same
  9. Mechanics · 8Reliability and statistics: one run proves little
  10. Systems · 9The eval harness: the program that runs the evals
  11. Systems · 10CI/CD quality gates: which evals may block a release
  12. Systems · 11Production evals: measuring what real traffic does
  13. Reference · 12Frameworks: what each eval tool runs for you
  14. Reference · 13The insurance lab: ten cases, end to end
  15. Reference · 14Advanced topics: where evals themselves fail
  16. ReferenceGlossary
  17. AboutAbout this site and its sources
Mechanics · 8

Reliability and statistics: one run proves little

The same input gives different runs. Repeated trials, pass@k and pass^k, sample size, confidence intervals and paired comparison with a baseline.

After this page you can say what one passing trial proves and what it does not. You can choose between two questions that sound alike: "does it succeed at least once?" and "does it succeed every time?". And you can tell how many trials a number needs before it may decide a release.

One pass
A passing trial is one observation. It shows the agent can do it, not how often.
Two questions
At least once, or every time. The same pass rate gives two very different answers.
How sure
A rate from 20 trials is a range, not a point. The range decides what you may claim.
Better or luck
Two variants are compared pair by pair on the same trials, not by two separate rates.

The case that passed, and then did not#

The insurer's regression suite holds case UW-004. A broker's submission says the roof is 9 years old. The roof-data provider says 22. Rule R-ROOF-03 of the policy fixture (illustrative) says: when the two ages differ by more than 5 years, escalate to a senior underwriter. So the expected result is the recommendation escalate, citing R-ROOF-03 and naming both sources. The model recommends; a person decides.

The eval harness runs UW-004 once. The trial passes. The report line reads 1 trial, 1 pass, rate 1.00, and someone writes in the release note that conflict handling works.

Case UW-010 is the same submission, run 20 times. This is what the harness prints. A successful run of this command proves only that the counting works; the agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is.

cd examples/uw-evals
PY=/opt/homebrew/bin/python3
$PY run_evals.py --case UW-010
Eval run · suite UW-010 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-010       20    16   0.80  escalate x16, approve x4  recommendation_match (4), citations (4)
trials 20 · passed 16 · pass rate 0.80 · 95% interval 0.58 to 0.92 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 20 trace file(s) in out/traces/

Sixteen trials escalate. Four recommend approve for a building whose roof age nobody has settled. Nothing changed between the trials: same submission, same tools, same policy fixture. The release note was written from a trial that happened to be one of the sixteen.

One more detail from the report. The four failing trials carry a stated confidence of 0.9, and the sixteen passing ones carry 0.8. The number the agent gives about itself is highest where it is wrong. Confidence explained why that number cannot stand in for a measurement. This page builds the measurement.

Why the same input gives different runs#

An ordinary function returns the same output for the same input. An agent often does not. The word for this is non-determinism: the same input can lead to different runs. Start here introduced it; here is what causes it in an agent like ours.

The model picks each next word with some randomness, so the wording differs, and sometimes the decision differs with it. The roof-data provider answers fast in one trial and times out in the next. A retrieval returns the same passages in another order. Any of these can move a trial onto another path.

Not all variation is a problem. In the 20 trials of UW-010 the wording of the reasons changes from trial to trial (the report labels three wordings A, B and C), and the graders pass all three, because they check the recommendation, the citation and the content, not the exact sentence. The variation that matters is the one that changes the result: approve where escalate was expected.

In our insurance agent. The stand-in is scripted to drop the provider's answer in a share of trials (agent_behaviour.non_escalation_rate is 0.2 in cases/UW-010.json). Without the provider's 22 years it sees no conflict and recommends approve. A real agent needs no script for this: it has its own ways to skip a step now and then.

Count the trials: the pass rate#

If one trial cannot answer "does it work?", the plain fix is to run the case many times and count. Each run of one case is a trial. The share of trials that pass is the pass rate. For UW-010 that is 16 of 20, or 0.80.

The pass rate has a second name when you think about a single trial. If you run the case once, the chance that this one trial passes is about 0.80. That is pass@1, read "pass at one": the chance of success on one trial.

Whether 0.80 is good depends on a question the number does not answer: how is the agent used? There are two very different uses, and each has its own metric.

At least once, or every time#

The first use: several tries, one is kept. Suppose a senior underwriter could ask the agent three times and keep the answer that survives a check. Then the agent is good enough if at least one of the three trials passes. The metric for this is pass@k: the chance that at least one of k trials passes.

The second use: every run counts. In the real workflow nobody asks three times. Each submission gets one recommendation, and each broker sees only their own. If three submissions with a roof-age conflict arrive this week, the insurer needs all three handled correctly. The metric for this is pass^k, read "pass to the power k": the chance that all k trials pass.

The two names, and the advice to pick by product, come from Anthropic's article Demystifying evals for AI agents: pass@k where one success is enough, pass^k where users expect the same right behavior every time. The examples and numbers here are our own.

With a pass rate of 0.80 for one trial and three trials, simple arithmetic gives both. All three pass: 0.80 × 0.80 × 0.80 = 0.51. At least one passes: that is everything except "all three fail", and all three fail with 0.20 × 0.20 × 0.20 = 0.008, so 0.99.

Under the hood. In general, with pass rate p for one trial: pass@k = 1 − (1 − p)^k and pass^k = p^k. Both formulas hold only if the trials are independent (one trial's result does not affect another's) and each trial succeeds with the same probability p. When trials fail together, the formulas are wrong.

The same 0.80 is now 0.99 or 0.51, depending on the question. Here are both functions from stats.py, unchanged, followed by a short script that applies them to the 20 recorded trials. The script shows that the formulas and the direct count can be computed from one report. It does not show that the formulas fit a real agent; that depends on the assumption above.

def pass_at_k(p: float, k: int) -> float:
    """pass@k by formula: the chance that at least one of k trials passes."""
    return 1 - (1 - p) ** k


def pass_all_k(p: float, k: int) -> float:
    """pass^k by formula: the chance that all k trials pass."""
    return p ** k
import json
import stats

report = json.load(open("out/report.json"))   # written by: run_evals.py --case UW-010
results = [t["passed"] for t in report["cases"][0]["trials"]]
print("".join("P" if r else "F" for r in results))

p = stats.pass_rate(results)
print(f"pass rate {stats.two(p)} ({sum(results)} of {len(results)})")
for k in (1, 3, 5, 10):
    print(f"k = {k:<2}  pass@k {stats.two(stats.pass_at_k(p, k))}   pass^k {stats.two(stats.pass_all_k(p, k))}")
print("measured, groups of 3:",
      "pass@3", stats.two(stats.pass_at_k_empirical(results, 3)),
      "  pass^3", stats.two(stats.pass_all_k_empirical(results, 3)))
PFPPPPPPPFPFPPPPPPFP
pass rate 0.80 (16 of 20)
k = 1   pass@k 0.80   pass^k 0.80
k = 3   pass@k 0.99   pass^k 0.51
k = 5   pass@k 1.00   pass^k 0.33
k = 10  pass@k 1.00   pass^k 0.11
measured, groups of 3: pass@3 1.00   pass^3 0.67

At k = 1 the two metrics are the same number, the pass rate. As k grows they move apart fast: by k = 10, "at least one" is certain to two decimals and "all ten" is down to 0.11. The last line uses no formula. It cuts the trials into groups of three (six full groups; the last two trials are left over) and counts. All six groups hold a pass, and four of the six hold only passes.

The measured pass^3 is 0.67 and the formula says 0.51. That is not a contradiction. Six groups are a very small sample, and the next section shows how little six of anything can pin down.

The lab below lets you move p and k and watch the two curves separate. The table holds the same numbers for p = 0.80.

k (trials)pass@k: at least one passespass^k: all pass
10.800.80
30.990.51
51.000.33
101.000.11

Formulas at p = 0.80, assuming independent trials with the same p. For 16 passes in 20 trials the observed rate is 0.80 and the 95% interval (Wilson) runs from 0.58 to 0.92. All numbers are synthetic.

Common confusion. A high pass@k is not reliability. An agent that fails four times in five still reaches pass@10 = 0.89 by the formula. pass@k is the right metric only when something trustworthy picks the good result: a test suite, a second check, a person. In our workflow each submission gets one recommendation, so pass^k is the honest one.
When the two formulas are wrongTrials that fail together, and cases that differ

The formulas need two things: independent trials and one shared p. Each can fail in practice.

Trials that are not independent. If the roof-data provider is down for a minute, every trial in that minute fails together. If trials share state (a cache, a database row left by an earlier trial), one trial changes the next. The Anthropic article names this risk: leftover shared state can cause correlated failures that come from the infrastructure, not from the agent. When failures cluster, "all k pass" is more likely than p^k says and "at least one passes" is less likely than the formula says, because a bad moment takes several trials at once. This is why the harness in the example builds a new World for every trial; The eval harness covers that.

Cases that do not share one p. Suppose one case passes every time and another passes 6 times in 10. Their average pass rate is 0.80, and the formula gives pass^3 = 0.51. The true value is the average of the two cases' own results: 1.00 for the first and 0.6 × 0.6 × 0.6 = 0.22 for the second, so 0.61. (This is an illustrative calculation, not a run.) The rule: compute pass^k for each case from that case's own trials, then average. Never raise an averaged pass rate to the power k.

Prefer the direct count when you have the trials. pass_all_k_empirical in stats.py assumes nothing about independence within a group: it looks at what happened. Its cost is sample size. Twenty trials give six groups of three, and only two groups of ten.

How far to trust 16 of 20#

So far the page has treated 0.80 as the agent's pass rate. It is not. It is what 20 trials showed. Run another 20 and the count will differ. This movement of a number when nothing changed is called variance.

The example can show it, because the runner takes a --seed (more on seeds below) and each seed gives a fresh set of 20 trials from the same stand-in:

for s in 1 2 3 4 5 6; do $PY run_evals.py --case UW-010 --seed $s | sed -n 3p; done
UW-010       20    17   0.85  escalate x17, approve x3  recommendation_match (3), citations (3)
UW-010       20    13   0.65  escalate x13, approve x7  recommendation_match (7), citations (7)
UW-010       20    17   0.85  escalate x17, approve x3  recommendation_match (3), citations (3)
UW-010       20    16   0.80  escalate x16, approve x4  recommendation_match (4), citations (4)
UW-010       20    12   0.60  escalate x12, approve x8  recommendation_match (8), citations (8)
UW-010       20    17   0.85  escalate x17, approve x3  recommendation_match (3), citations (3)

The same agent scores 0.85, 0.65, 0.85, 0.80, 0.60 and 0.85. Seeds 1 to 12 give between 12 and 18 passes out of 20. A team that ran seed 5 on Monday and seed 6 on Tuesday would report a jump from 0.60 to 0.85 and credit Tuesday's prompt change, which did nothing.

Here the example has a luxury no real project has: the stand-in's true pass rate is known, because it is written in the case file. It is 0.80. With a real agent nobody knows the true rate. All you have is the count, so the useful question becomes: which true rates could have produced 16 of 20?

Plain English. A confidence interval is that answer: a range of true pass rates that fit what you counted. For 16 of 20 the range runs from 0.58 to 0.92. A true rate of 0.60 could easily have produced 16 of 20. So could 0.90.

The example computes the range with a method called the Wilson score interval, in wilson_interval of stats.py. The "95%" belongs to the method: used on many samples, it gives a range that contains the true rate about 95 times in 100. The method assumes that the trials are independent and share one true pass rate, the same two assumptions as before.

The practical reading is short. Do not hold the observed 0.80 against a threshold. Hold the low end of the interval against it. The gate of the example does that, with a threshold of 0.70:

$PY gate.py
Gate · suite UW-010 · variant careful · threshold 0.70
BLOCK: 16 of 20 trials passed (0.80); the 95% interval is 0.58 to 0.92. The lower bound 0.58 is below the threshold 0.70. The observed rate looks high enough, but the sample is too small to claim it.

The gate does not say the agent is below 0.70. It says 20 trials cannot show that it is above. CI gates builds the full blocking rule on this idea.

The diagram puts the steps in order: what each one adds, and what you may say after it.

Diagram: From one passing trial to a claim you can defend

Read left to right. Each step adds evidence and changes what you may say. One passing trial lets you say that it can work. Many trials give a rate. The interval says how far that rate can be trusted. Only then is there something to hold against a threshold. The numbers in the examples are synthetic.

  1. 1 · One trial
    1. One trial passes: A single run of one case passed. It shows the agent can get this case right, not how often it does.
  2. 2 · Many trials
    1. Repeat the same case: The harness runs the same case again and again, each trial from a fresh state, so one trial cannot change the next.
    2. Count the passes: Passes divided by trials. It is a measurement of this sample, and a new sample gives a different number.
  3. 3 · How sure
    1. Put an interval around the rate: The range of true pass rates that fit the count. Few trials give a wide range.
    2. Pick the question the product asks: If one good result out of k is enough, read pass@k. If every one of k must be right, read pass^k.
  4. 4 · The claim
    1. Hold the low end against the line: A claim that the rate reaches a threshold holds only if the low end of the interval reaches it.

The interval also has a consequence for the last section. If the true rate can be anywhere from 0.58 to 0.92, then pass^3 can be anywhere from 0.20 to 0.78 by the formula. A pass^k computed from a small sample inherits all the uncertainty of the rate, raised to the power k.

The Wilson interval, and why not the textbook oneThe formula, its assumptions, and what 95% means

This is wilson_interval from stats.py, unchanged. z = 1.96 gives about 95%.

def wilson_interval(passes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """Wilson score interval for a pass rate (z = 1.96 gives about 95%). (0, 1) when n is 0."""
    if n == 0:
        return (0.0, 1.0)
    p = passes / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return (max(0.0, centre - half), min(1.0, centre + half))

Why not the simpler formula. The interval most people learn first is the observed rate plus or minus 1.96 standard errors, where the standard error is the square root of p × (1 − p) / n. It breaks exactly where evals live. With 9 passes in 9 trials the observed p is 1, the standard error is 0, and the interval has zero width: it claims certainty from nine trials. Near 0 or 1 it can also run past the ends of the scale. The Wilson interval, published by E. B. Wilson in 1927, has neither problem: it moves its centre toward 0.5 by an amount that shrinks as n grows. For 9 of 9 it gives 0.70 to 1.00.

What the 95% means. It describes the method over many uses. It is not a 95% chance that the true rate lies inside this one range, and it is not the agent's stated confidence.

Assumptions. Independent trials, one true pass rate, pass or fail outcomes. If trials fail in clusters, the real uncertainty is larger than the interval shows.

A stricter alternative. The exact (Clopper-Pearson) interval never covers less often than its nominal level, and it is usually wider for that reason. A gate that prefers to err toward blocking can use it. The example does not implement it.

A shortcut for zero failures. If all n trials pass, a rough 95% lower bound for the true rate is 1 − 3/n (the rule of three). Twenty passes in twenty still leave room for a true rate near 0.85; the Wilson interval for 20 of 20 is 0.84 to 1.00.

How many trials are enough#

The interval gets narrower as trials are added. The next script asks how fast. It applies wilson_interval to the same observed rate, 0.80, at eight sample sizes. Sample size means the number of trials behind a number. The counts are illustrative; the function is the one from the example.

import stats

for passes, n in [(1, 1), (4, 5), (16, 20), (40, 50), (80, 100), (160, 200), (400, 500), (800, 1000)]:
    low, high = stats.wilson_interval(passes, n)
    print(f"{passes:>4} of {n:<5} rate {stats.two(passes / n)}   95% interval {stats.two(low)} to {stats.two(high)}   width {stats.two(high - low)}")
   1 of 1     rate 1.00   95% interval 0.21 to 1.00   width 0.79
   4 of 5     rate 0.80   95% interval 0.38 to 0.96   width 0.59
  16 of 20    rate 0.80   95% interval 0.58 to 0.92   width 0.34
  40 of 50    rate 0.80   95% interval 0.67 to 0.89   width 0.22
  80 of 100   rate 0.80   95% interval 0.71 to 0.87   width 0.16
 160 of 200   rate 0.80   95% interval 0.74 to 0.85   width 0.11
 400 of 500   rate 0.80   95% interval 0.76 to 0.83   width 0.07
 800 of 1000  rate 0.80   95% interval 0.77 to 0.82   width 0.05

The first row is the release note from the opening. One trial, one pass: the range runs from 0.21 to 1.00. That is what "it passed" was worth.

The rest of the table shows a hard rule. To cut the width in half you need about four times the trials: 0.34 at 20, 0.16 at 100, 0.07 at 500. Precision gets expensive quickly, and every trial of a real agent costs model usage and time.

So the number of trials should follow from the claim, decided before the run. For the gate above, the claim is "the low end reaches 0.70". If the agent keeps passing 8 in 10, that takes 85 trials (68 of 85 gives 0.70 to 0.87). If it passed 19 in 20, twenty trials would be enough (0.76 to 0.99). An agent far above the line needs few trials to show it. An agent near the line needs many.

What you want to claimWhat it takes
"It can do this case"One passing trial
"It does this case most of the time"Tens of trials; read the low end of the interval
"Its rate is above a threshold that sits close to the observed rate"Hundreds of trials
"It never does X"No number of passing trials proves it. Zero failures in n trials still leaves a rate near 3/n possible

The last row matters for a critical case such as UW-005, where a retrieved passage tries to inject an instruction. Trials can make a failure rate look small. They cannot make it zero. For the things that must never happen, add a control in code that does not depend on the model, and use the eval to watch it.

Planning the number of trials before the runA rough formula, and three ways to waste trials

A rough planning rule for the half-width h of a 95% interval: n ≈ p × (1 − p) × (1.96 / h)². It uses the normal approximation and assumes independent trials with one true rate, so treat the result as a starting point and check it with wilson_interval. At p = 0.80: h = 0.10 needs about 62 trials, h = 0.05 about 246, h = 0.02 about 1,537. These are calculations, not runs.

Trials of one case are not cases. Twenty trials of UW-010 narrow the estimate for that one submission. They say nothing about a conflict of 6 years, or about a different occupancy. Spending 200 trials on one case buys a precise number about a narrow question. Trials tell you how stable the agent is on a case; cases tell you how wide the behavior holds. A fixed budget has to be split between the two.

Do not pool unlike things. The full run of the example reports 25 of 29 trials passed, 0.86, interval 0.69 to 0.95. Twenty of those 29 trials are one case, so the trials are not independent draws from one population, and the interval is narrower than the evidence supports. The gate ignores the pooled number for that reason and reads the measured case and the critical cases separately.

Do not stop when it looks good. Running "until the lower bound clears the line" and stopping at the first moment it does is a way to pass by luck. Fix the number of trials first.

What a seed fixes, and what it does not#

The runner has printed seed 7 on every report. A seed is a starting number for a random number generator: the same seed produces the same sequence of "random" choices. The stand-in agent draws its luck from a generator built like this, in run_agent of agent_sim.py:

rng = random.Random(f"{seed}:{case['id']}:{trial}")

Seed, case and trial number together decide whether this trial drops the provider's answer and which wording it uses. So --seed 7 gives 16 of 20 today, tomorrow and on a colleague's machine. Run it twice and compare; the two reports are the same.

That buys three things. A report can be reproduced. A failing trial can be replayed: trial 2 with seed 7 fails every time, so it can be debugged. And two variants can be given the same trials, which the comparison below depends on. It is easy to expect more.

A seed doesA seed does not
Make the randomness that your own program draws repeatChange the true pass rate. The stand-in is at 0.80 under every seed
Let you replay one failing trialMake the measured rate right. Seed 5 says 0.60, seed 10 says 0.90
Give two variants the same trialsAdd evidence when repeated. Seed 7 run ten times is one sample of 20, copied ten times
Make a report reproducibleControl what happens outside your program

The last row is where real agents differ from the stand-in. Here the seed controls everything, because all the randomness lives in one Python generator. A real agent calls a model over a network. Some model APIs accept a seed setting; what it promises is for that provider's documentation to say, and it covers at most the sampling inside the model call. It does not cover a tool that times out, a retrieval index that was rebuilt, or a provider that changed the model behind the same name. Such a setting makes a model less variable. Do not call an agent deterministic unless the provider guarantees it and you have verified it.

Production pitfall. A fixed seed can hide the problem you are measuring. If every nightly run uses seed 7, the dashboard shows 0.80 every night and looks stable. The stability belongs to the seed. Keep a fixed seed for reproducing a result, and vary the seed (or run more trials) when you want to learn the rate.

Where the failures sit: slices#

One pass rate over a whole suite can hide where the failures are. A slice is a part of the cases that share a label, such as all cases about a roof-age conflict; Golden datasets explains how cases get their labels. The report of the example already counts by label. This script prints those counts from the full run with an interval for each. It shows where the evidence is thin; it proves nothing about slices that have no cases.

import json
import stats

summary = json.load(open("out/report.json"))["summary"]   # written by: run_evals.py --suite all
for tag, s in summary["by_tag"].items():
    low, high = stats.wilson_interval(s["passes"], s["n"])
    print(f"{tag:<15} {s['passes']:>2} of {s['n']:<3} 95% interval {stats.two(low)} to {stats.two(high)}")
clean            1 of 1   95% interval 0.21 to 1.00
conflict        17 of 21  95% interval 0.60 to 0.92
human-override   1 of 1   95% interval 0.21 to 1.00
injection        1 of 1   95% interval 0.21 to 1.00
missing-data     1 of 1   95% interval 0.21 to 1.00
policy-version   1 of 1   95% interval 0.21 to 1.00
process          1 of 1   95% interval 0.21 to 1.00
reliability     16 of 20  95% interval 0.58 to 0.92
roof            24 of 28  95% interval 0.69 to 0.94
tool-failure     1 of 1   95% interval 0.21 to 1.00

The overall number of that run is 25 of 29, a pass rate of 0.86. The slices say more. All four failures sit in conflict. And seven slices rest on a single trial each, with an interval from 0.21 to 1.00: the suite has said almost nothing about injection or tool failure yet. Read each slice with its own count.

Is the new variant better, or was it luck#

Now the most common use of all these numbers: someone changes the agent and wants to know whether it got better. The existing version is the baseline, and the changed one is the candidate.

The obvious approach is to run each and compare the two pass rates. Suppose the baseline scores 16 of 20 and the candidate 18 of 20. Ten points better? The intervals are 0.58 to 0.92 and 0.70 to 0.97. They overlap almost entirely. And the seed experiment showed the same agent moving between 0.60 and 0.90 by luck alone.

There is a sharper tool, and it starts with a change in how the trials are run. Give both variants the same trials: same cases, same seed. Then look at each pair of results. This is a paired comparison.

The reason it works: in most pairs the two variants do the same thing. Both pass the easy trial; both fail the unlucky one. Those pairs say nothing about which variant is better. All the information is in the pairs that differ. If the two variants were equally good, a differing pair would go either way like a coin flip. So the question becomes: could a fair coin have produced this split?

Diagram: A paired comparison · two variants on the same trials

Read left to right. Both variants run the same cases with the same seed, so trial 7 of one faces what trial 7 of the other faced. The pairs in which both pass or both fail say nothing about which variant is better. Only the pairs that differ are counted.

  1. 1 · The same trials
    1. Same case, same seed, same trial number: Both variants get identical inputs and identical scripted luck, so a difference comes from the variant.
  2. 2 · Two variants
    1. Baseline: The variant you compare against. Its results are the reference.
    2. Candidate: The variant you want to judge. It runs the same trials as the baseline.
  3. 3 · Pair by pair
    1. Pairs that agree: These pairs carry no information about which variant is better. They are set aside.
    2. Pairs that differ: The evidence. Count how many go each way.
  4. 4 · The test
    1. Could a fair coin give this split?: If the variants were equally good, each differing pair would go either way with chance one half. The test gives the chance of a split at least this uneven.

The runner does this with --compare. The example has two variants of the stand-in: careful, and shortcut, which skips the guidance search. The command runs the 20 trials of UW-010 under both and compares them pair by pair. A successful run shows the mechanics of the comparison on a difference that is large by construction.

$PY run_evals.py --case UW-010 --variant careful --compare shortcut
Eval run · suite UW-010 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-010       20    16   0.80  escalate x16, approve x4  recommendation_match (4), citations (4)
trials 20 · passed 16 · pass rate 0.80 · 95% interval 0.58 to 0.92 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 20 trace file(s) in out/traces/

Paired comparison on the same 20 trials: careful vs shortcut
pass rate 0.80 vs 0.00 · difference +0.80
only careful passes: 16 · only shortcut passes: 0 · both pass: 0 · both fail: 4
exact sign test on the 16 pairs that differ: p = 0.000

Read the third line from the bottom. In 4 pairs both variants fail. These are the same four trials (2, 10, 12 and 19) in which the shared seed makes the stand-in drop the provider's answer, under either variant. That is pairing at work: bad luck that hits both sides cancels out. In the other 16 pairs, careful passes and shortcut fails. No pair goes the other way.

The last line is the coin question. The chance that a fair coin gives 16 to 0, or something as lopsided, is 2 × 0.5^16, about 0.00003. The runner prints three decimals, so it shows 0.000; read it as "below 0.001". This chance is called a p-value, and when it is small, people call the difference statistically significant: chance is a poor explanation for it.

Now the comparison from the paragraph above, 18 of 20 against 16 of 20, with paired_comparison from stats.py. The two lists are illustrative input. The run shows what the test says when few pairs differ.

import stats

# Illustrative input: 20 paired trials, variant B against baseline A.
a = [True] * 15 + [True, False, False, False, False]    # A passes 16
b = [True] * 15 + [False, True, True, True, False]      # B passes 18
c = stats.paired_comparison(b, a)
print(f"B {stats.two(c['rate_a'])} vs A {stats.two(c['rate_b'])} · difference {c['difference']:+.2f}")
print(f"only B passes: {c['only_a']} · only A passes: {c['only_b']} · both pass: {c['both_pass']} · both fail: {c['both_fail']}")
print(f"exact sign test on the {c['only_a'] + c['only_b']} pairs that differ: p = {c['p_value']:.3f}")
B 0.90 vs A 0.80 · difference +0.10
only B passes: 3 · only A passes: 1 · both pass: 15 · both fail: 1
exact sign test on the 4 pairs that differ: p = 0.625

Four pairs differ, three to one. A fair coin does at least that well more than half the time. The ten-point gain is an observation. This run does not show that B is better, and it does not show that B is the same; it shows that 20 paired trials cannot tell.

Comparison in the examplePairs that differSplitp-valueReading
careful vs shortcut, 20 trials of UW-0101616 to 0about 0.00003Chance does not explain it
shortcut vs careful, regression suite, 9 trials80 to 80.008Chance does not explain it
Illustrative B vs A, 20 trials43 to 10.625Shows nothing either way
Common confusion. Significant does not mean important, and not significant does not mean equal. A p-value says how easily chance could produce the split. It says nothing about the size of the difference, and nothing about whether a failure matters. One hard-fail on a critical case blocks a release with no test at all.
The sign test in detail, and where it stopsThe computation, minimum pairs, many comparisons, one case

The computation. paired_comparison in stats.py counts the pairs only one side passes, takes m as the number of pairs that differ, and sums the chances of the observed split and every more lopsided one under a fair coin, doubled for both directions (capped at 1). This is the exact two-sided sign test, also called the exact McNemar test. It assumes the pairs are independent of each other and that pairing is real: the same case and conditions on both sides.

A floor on the evidence. With all differing pairs in one direction the p-value is 2 × 0.5^m: 0.125 for 4 pairs, 0.0625 for 5, 0.031 for 6. So a comparison with fewer than six differing pairs cannot go below the usual line of 0.05, whatever the split. If two variants rarely disagree, you need many trials to collect six disagreements. That line of 0.05 is a convention, to be set before the run.

Many comparisons. Try twenty prompt variants against the baseline and, at a line of 0.05, about one will look significant by chance alone. Decide in advance which comparison counts, or tighten the line when you test many.

Twenty trials of one case are one case. The first row of the table says careful beats shortcut on this submission. It does not say so for other submissions. The second row, with nine different cases, speaks more broadly with fewer trials. For a claim about the agent, pair across many cases; for a claim about stability on one case, pair across trials.

What the stand-in cannot show. The shortcut variant fails by script, in every trial, so "chance" hardly applies to it. With a real model both variants vary, the pairs disagree in both directions, and the test earns its place.

An observation is still worth acting on. "The candidate failed UW-008 in all three trials and the baseline passed all three" is not a significance result. It is a strong signal on a critical case and a reason to open the trace. Set such triggers in advance, next to the statistical rule.

Edge cases and trade-offs#

A flaky grader looks like a flaky agent. If a model is the grader, the grader varies too, and a pass rate mixes two sources of variation. Grade one recorded trial several times to see the grader's own variation before blaming the agent. Judges covers how a judge is checked against people.

A rate of 0 or 1 deserves suspicion. Zero of 20 is more often a broken case or a broken grader than a hopeless agent, a point the Anthropic article also makes. Twenty of 20 on a case that should be hard may mean the grader checks too little.

Critical and measured cases need different rules. For UW-001 to UW-009 the example asks for a pass in every trial and treats any failure as a reason to stop and look. For UW-010 it measures a rate and reads the low end. Mixing the two into one average hides both. How many trials each kind gets on which change is a budget question that CI gates works out.

In one breath#

A passing trial is one observation, so we run a case many times and count: 16 of 20 is a pass rate of 0.80. That rate answers two different questions, "at least one of k" (pass@k) and "all k" (pass^k), and for an agent where every submission counts we read pass^k, with the formulas trusted only when trials are independent with one shared rate. The 0.80 itself is a sample, so we report the interval, 0.58 to 0.92, and hold its low end against the threshold. To compare two variants we give both the same trials and count only the pairs that differ. A seed makes all of this repeatable; it does not make any of it true.

A score is an observed result. Confidence is a claim about uncertainty. Reliability needs repeated evidence.

The same idea for a coding agent: one green test run of a generated patch is one trial, pass@k fits when the test suite picks the working patch out of k, and pass^k fits when every merged change has to be right.

What comes next#

You can now turn a set of trials into a claim with a stated width, and tell a real difference from a lucky run. Every one of those claims leaned on a quiet promise: that each trial started clean, that the tools behaved the same way for both variants, that a seed reached everything it was supposed to reach. None of that is statistics. It is the work of the program that runs the trials. The eval harness opens that program and shows what it must guarantee for these numbers to mean anything.