All chapters
  1. Start hereA test checks an answer. An eval measures a behavior
  2. Foundations · 1Primitives: the small words every eval is built from
  3. Foundations · 2Grading and rubrics: who decides what good means
  4. Foundations · 3Golden datasets: the cases you trust
  5. Mechanics · 4What to evaluate: the answer, the path, and what was left behind
  6. Mechanics · 5Agent traces: the record you grade the path from
  7. Mechanics · 6Judges and calibration: when a model grades a model
  8. Mechanics · 7Confidence and uncertainty: five numbers that are not the same
  9. Mechanics · 8Reliability and statistics: one run proves little
  10. Systems · 9The eval harness: the program that runs the evals
  11. Systems · 10CI/CD quality gates: which evals may block a release
  12. Systems · 11Production evals: measuring what real traffic does
  13. Reference · 12Frameworks: what each eval tool runs for you
  14. Reference · 13The insurance lab: ten cases, end to end
  15. Reference · 14Advanced topics: where evals themselves fail
  16. ReferenceGlossary
  17. AboutAbout this site and its sources
About

About this site and its sources

What this site is, what it is not, where its ideas come from, and how the facts were checked.

This is a knowledge site about measuring software that contains a language model. It starts from one test with one expected result and climbs to rubrics, golden datasets, traces, judges, repeated trials and release gates. At each step it asks one question: what does this number let me claim? It is written for engineers and architects who build agent systems and have to decide whether a change is safe to release.

One example runs through every page. An insurer receives a broker's request for a quote. An underwriting agent reads it, checks the roof, looks up the guidance and writes a recommendation. A human underwriter approves or declines. The model recommends; a person decides.

What it is and is not#

It isIt is not
A ladder: one idea per step, each word in plain English with a tiny example before it is usedA standard, or the only way to organize evaluation work
One running example, ten cases UW-001 to UW-010, followed through every chapterA record of any real insurer. The cases, the rules, the fields and the provider are invented
A demonstration of method: how a case, a grader, a trace, a trial and a gate fit togetherA measurement of any model or product. No page reports how good a real system is
Code that was run: a small eval harness in standard-library Python, with its real outputA framework to install. It is short so that it can be read whole, and it leaves things out
Tools described from their own documentation, by what each one runs for youA ranking or a buying guide. No tool is called best, and no price is quoted
Statistics with their assumptions stated next to each formulaA statistics course. The pages cover what a release decision needs and link to the rest

What is original here and what is not#

Most of the ideas on this site were published by other people first. The first source is the article "Demystifying evals for AI agents" by Anthropic. This site uses that article's ideas with attribution. It does not use its prose, its examples or its figures.

PartStatus
The words trial, grader, outcome, suite, evaluation harness and agent harnessFrom the Anthropic article, used here with the same meaning
Task and caseThe article's task is one test with its inputs and its success criteria. This site calls that a case, and keeps task for what the agent is asked to do
TraceThe article calls the full record of a trial a transcript. This site says trace and gives the record a structure of spans, following OpenTelemetry
Code, model and human gradersThe three kinds named in the Anthropic article
pass@k and pass^kDescribed in the Anthropic article. The formulas on this site hold only under the assumptions written next to them
Capability and regression suitesFrom the Anthropic article. The smoke suite and the rules for selecting a suite are common practice in continuous integration
Confidence intervals, paired comparison, sample sizeStandard statistics. The reading list names where this site read them
Calibration and expected calibration errorPublished research on calibration. See the reading list
The five numbers that look alikeAn arrangement proposed on this site, to keep five different quantities apart
The gate rule in the exampleA design proposed on this site: the lower end of the interval must reach the threshold, and a critical case may never fail
The ten cases, the policy fixture, the rubric, the human labelsWritten for this site. All synthetic
The example code and its testsWritten for this site, in standard-library Python
The interactive labs and the diagramsWritten for this site

Where a page takes an idea from a source, the page says so. Where this site proposes something, it calls it a design.

How the facts were checked#

Kind of claimHow it was checked
A named toolRead on the tool's own documentation on the day of writing and logged with the page address and the date. Where the documentation is silent, the page says "the documentation does not say". Statements about tools live on Frameworks
An idea from a sourceThe source was opened and read on the day of writing. It is listed below
The exampleRun. Every command shown from examples/uw-evals/ was executed, and the output on the page is the output it printed. The folder has 24 unit tests
The labsThey read data that a script generates from the example with a fixed seed. Their arithmetic is a second implementation of the Python functions, and a test fails if the two differ
FormulasEach one appears with the assumptions it needs. The formulas for pass@k and pass^k assume independent trials with the same chance of success
NumbersSynthetic and illustrative unless a source is linked. A number from the example is the result of a script, never a measurement of a model
Other codeA block that was not run says how far it was checked, for example "documented shape; not run here"
Links and structureA build step checks every internal link, every diagram and every glossary reference before the site is published

Documentation changes. Treat every statement about a tool as true on the day it was read, and read the tool's page again before you build on it.

What the example does not show#

The insurance example is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider and every number are invented. Nothing here reports the quality of a real system or model.

The agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is. The list below says exactly what stands in for what.

In the exampleWhat it really is
The underwriting agentA script with two variants, careful and shortcut. No model is called. There is no network and no key
The judgeA script with keyword rules that stands in for a model used as a grader
The human labelsSix synthetic labels, written for the page on judges
The toolsStand-ins that return what the case file tells them to return
The policy fixtureFive illustrative rules. They are not insurance practice
Times in a traceValues from a simulated clock
Variation between trialsProduced by a fixed seed, so that every reader sees the same run

The run that the pages quote ends with 25 of 29 trials passed. A script and a seed produced that count. It teaches how to read a pass rate and its interval. It says nothing about how often a model would pass these cases. The whole example is on The insurance lab.

Sources and reading list#

Each link below was opened on the day this page was written. The dates in the table are publication dates given by the source.

Start here#

SourceWhy read it
Demystifying evals for AI agents, Anthropic, January 2026The conceptual starting point of this site: the structure of an evaluation, the kinds of grader, and a path from no evals to a working suite

Statistics#

SourceWhy read it
A statistical approach to model evaluations, Anthropic, November 2024A short summary of why an eval score is an estimate that needs an interval, and why two versions are compared pair by pair
Adding Error Bars to Evals, Evan Miller, 2024The paper behind that summary, with the formulas. Depth for Reliability
Binomial proportion confidence interval, WikipediaThe Wilson score interval, which the example uses for a pass rate from few trials. The page credits it to E. B. Wilson, 1927

Calibration#

SourceWhy read it
On Calibration of Modern Neural Networks, Guo, Pleiss, Sun and Weinberger, 2017A study of how far a model's stated probability can sit from how often it is right. Depth for Confidence

Traces#

SourceWhy read it
OpenTelemetry semantic conventions for generative AIThe shared names for spans of model calls and agent steps. The older location on opentelemetry.io now says the conventions moved to this repository. Depth for Traces

Tool documentation#

These are the entry pages of each tool's own documentation. What each tool runs for you is described on Frameworks, and only there.

Sister sites#

Three sites share one voice and one way of working. Each takes a different question, and none repeats a chapter of another.

SiteThe question it answers
Harness EngineeringWhat does the software around a model call have to do? The agent harness that this site evaluates is explained there
Diff to DeployHow does one code change travel through a delivery pipeline that uses coding agents? The pipeline that runs the gates of CI/CD quality gates is explained there
Eval Engineering, this siteHow do you measure what an agent does, and what may you claim from the number?

Who wrote it#

Sam Muthu, an engineer and architect who builds agent systems and holds Anthropic's Claude Certified Architect credentials (Foundations and Professional). More of his work is at sammuthu.com. Two sister publications sit next to this one: Harness Engineering covers the runtime around a model call, and Diff to Deploy follows one code change through a delivery pipeline that uses coding agents.

The site is a working version. If a claim is wrong or out of date, the fix is to read the source again and correct the sentence. Every term the pages use is collected in the Glossary.