This is a knowledge site about measuring software that contains a language model. It starts from one test with one expected result and climbs to rubrics, golden datasets, traces, judges, repeated trials and release gates. At each step it asks one question: what does this number let me claim? It is written for engineers and architects who build agent systems and have to decide whether a change is safe to release.
One example runs through every page. An insurer receives a broker's request for a quote. An underwriting agent reads it, checks the roof, looks up the guidance and writes a recommendation. A human underwriter approves or declines. The model recommends; a person decides.
What it is and is not#
| It is | It is not |
|---|---|
| A ladder: one idea per step, each word in plain English with a tiny example before it is used | A standard, or the only way to organize evaluation work |
One running example, ten cases UW-001 to UW-010, followed through every chapter | A record of any real insurer. The cases, the rules, the fields and the provider are invented |
| A demonstration of method: how a case, a grader, a trace, a trial and a gate fit together | A measurement of any model or product. No page reports how good a real system is |
| Code that was run: a small eval harness in standard-library Python, with its real output | A framework to install. It is short so that it can be read whole, and it leaves things out |
| Tools described from their own documentation, by what each one runs for you | A ranking or a buying guide. No tool is called best, and no price is quoted |
| Statistics with their assumptions stated next to each formula | A statistics course. The pages cover what a release decision needs and link to the rest |
What is original here and what is not#
Most of the ideas on this site were published by other people first. The first source is the article "Demystifying evals for AI agents" by Anthropic. This site uses that article's ideas with attribution. It does not use its prose, its examples or its figures.
| Part | Status |
|---|---|
| The words trial, grader, outcome, suite, evaluation harness and agent harness | From the Anthropic article, used here with the same meaning |
| Task and case | The article's task is one test with its inputs and its success criteria. This site calls that a case, and keeps task for what the agent is asked to do |
| Trace | The article calls the full record of a trial a transcript. This site says trace and gives the record a structure of spans, following OpenTelemetry |
| Code, model and human graders | The three kinds named in the Anthropic article |
| pass@k and pass^k | Described in the Anthropic article. The formulas on this site hold only under the assumptions written next to them |
| Capability and regression suites | From the Anthropic article. The smoke suite and the rules for selecting a suite are common practice in continuous integration |
| Confidence intervals, paired comparison, sample size | Standard statistics. The reading list names where this site read them |
| Calibration and expected calibration error | Published research on calibration. See the reading list |
| The five numbers that look alike | An arrangement proposed on this site, to keep five different quantities apart |
| The gate rule in the example | A design proposed on this site: the lower end of the interval must reach the threshold, and a critical case may never fail |
| The ten cases, the policy fixture, the rubric, the human labels | Written for this site. All synthetic |
| The example code and its tests | Written for this site, in standard-library Python |
| The interactive labs and the diagrams | Written for this site |
Where a page takes an idea from a source, the page says so. Where this site proposes something, it calls it a design.
How the facts were checked#
| Kind of claim | How it was checked |
|---|---|
| A named tool | Read on the tool's own documentation on the day of writing and logged with the page address and the date. Where the documentation is silent, the page says "the documentation does not say". Statements about tools live on Frameworks |
| An idea from a source | The source was opened and read on the day of writing. It is listed below |
| The example | Run. Every command shown from examples/uw-evals/ was executed, and the output on the page is the output it printed. The folder has 24 unit tests |
| The labs | They read data that a script generates from the example with a fixed seed. Their arithmetic is a second implementation of the Python functions, and a test fails if the two differ |
| Formulas | Each one appears with the assumptions it needs. The formulas for pass@k and pass^k assume independent trials with the same chance of success |
| Numbers | Synthetic and illustrative unless a source is linked. A number from the example is the result of a script, never a measurement of a model |
| Other code | A block that was not run says how far it was checked, for example "documented shape; not run here" |
| Links and structure | A build step checks every internal link, every diagram and every glossary reference before the site is published |
Documentation changes. Treat every statement about a tool as true on the day it was read, and read the tool's page again before you build on it.
What the example does not show#
The insurance example is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider and every number are invented. Nothing here reports the quality of a real system or model.
The agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is. The list below says exactly what stands in for what.
| In the example | What it really is |
|---|---|
| The underwriting agent | A script with two variants, careful and shortcut. No model is called. There is no network and no key |
| The judge | A script with keyword rules that stands in for a model used as a grader |
| The human labels | Six synthetic labels, written for the page on judges |
| The tools | Stand-ins that return what the case file tells them to return |
| The policy fixture | Five illustrative rules. They are not insurance practice |
| Times in a trace | Values from a simulated clock |
| Variation between trials | Produced by a fixed seed, so that every reader sees the same run |
The run that the pages quote ends with 25 of 29 trials passed. A script and a seed produced that count. It teaches how to read a pass rate and its interval. It says nothing about how often a model would pass these cases. The whole example is on The insurance lab.
Sources and reading list#
Each link below was opened on the day this page was written. The dates in the table are publication dates given by the source.
Start here#
| Source | Why read it |
|---|---|
| Demystifying evals for AI agents, Anthropic, January 2026 | The conceptual starting point of this site: the structure of an evaluation, the kinds of grader, and a path from no evals to a working suite |
Statistics#
| Source | Why read it |
|---|---|
| A statistical approach to model evaluations, Anthropic, November 2024 | A short summary of why an eval score is an estimate that needs an interval, and why two versions are compared pair by pair |
| Adding Error Bars to Evals, Evan Miller, 2024 | The paper behind that summary, with the formulas. Depth for Reliability |
| Binomial proportion confidence interval, Wikipedia | The Wilson score interval, which the example uses for a pass rate from few trials. The page credits it to E. B. Wilson, 1927 |
Calibration#
| Source | Why read it |
|---|---|
| On Calibration of Modern Neural Networks, Guo, Pleiss, Sun and Weinberger, 2017 | A study of how far a model's stated probability can sit from how often it is right. Depth for Confidence |
Traces#
| Source | Why read it |
|---|---|
| OpenTelemetry semantic conventions for generative AI | The shared names for spans of model calls and agent steps. The older location on opentelemetry.io now says the conventions moved to this repository. Depth for Traces |
Tool documentation#
These are the entry pages of each tool's own documentation. What each tool runs for you is described on Frameworks, and only there.
| Tool | Documentation |
|---|---|
| DeepEval | deepeval.com/docs/getting-started |
| Promptfoo | promptfoo.dev/docs/intro |
| LangSmith | docs.langchain.com/langsmith/evaluation |
| Phoenix | arize.com/docs/phoenix |
| Ragas | docs.ragas.io |
| OpenAI eval tooling | developers.openai.com/api/docs/guides/evals |
Sister sites#
Three sites share one voice and one way of working. Each takes a different question, and none repeats a chapter of another.
| Site | The question it answers |
|---|---|
| Harness Engineering | What does the software around a model call have to do? The agent harness that this site evaluates is explained there |
| Diff to Deploy | How does one code change travel through a delivery pipeline that uses coding agents? The pipeline that runs the gates of CI/CD quality gates is explained there |
| Eval Engineering, this site | How do you measure what an agent does, and what may you claim from the number? |
Who wrote it#
Sam Muthu, an engineer and architect who builds agent systems and holds Anthropic's Claude Certified Architect credentials (Foundations and Professional). More of his work is at sammuthu.com. Two sister publications sit next to this one: Harness Engineering covers the runtime around a model call, and Diff to Deploy follows one code change through a delivery pipeline that uses coding agents.
The site is a working version. If a claim is wrong or out of date, the fix is to read the source again and correct the sentence. Every term the pages use is collected in the Glossary.