The release went out on a quiet day. The regression suite had passed, the gate had printed PASS, and nothing in the report asked for attention.
A week later a broker's quote request arrives for a masonry office: built 2005, 18,000 sq ft, sprinklered, roof 14 years old, wind zone B, no losses. The underwriting agent reads the submission, checks the roof with the outside provider, retrieves the guidance, runs the eligibility rules and recommends approve. The model recommends; a person decides. Ninety minutes later the underwriter opens the underwriter cockpit and declines. The reason field says: "The broker's loss run shows an open water damage claim that is not on the application."
Nothing crashed. No grader failed, because no grader was running. The agent followed every required step, and on the input it was given, approve was the answer the rules produce. And still a person with one more document said no. This page is about that gap: what happens after the gate, on inputs nobody chose, where no expected result exists.
A suite holds the situations somebody thought of. Production holds the rest. The work is to notice what the suite missed, agree on what should have happened, and put it in the suite so that it is never missed twice.
1. The suite passed, and it was telling the truth#
Start with what was known on release day. A case is one input with its expected result. A suite is a named set of cases with one purpose. The regression suite of the example holds nine cases, UW-001 to UW-009, and its purpose is to show that the agent still handles situations it handled before. CI gates explained the gate that reads its report.
The two commands below run that suite and then the gate. A successful run proves that the agent, on these nine stored inputs, did what people had agreed it should do. It does not prove anything about an input that is not among the nine. The agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is.
cd examples/uw-evals
PY=/opt/homebrew/bin/python3
$PY run_evals.py --suite regression && $PY gate.py
Eval run · suite regression · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-001 1 1 1.00 decline -
UW-002 1 1 1.00 approve -
UW-003 1 1 1.00 request_info -
UW-004 1 1 1.00 escalate -
UW-005 1 1 1.00 decline -
UW-006 1 1 1.00 request_info -
UW-007 1 1 1.00 decline -
UW-008 1 1 1.00 decline -
UW-009 1 1 1.00 approve -
trials 9 · passed 9 · pass rate 1.00 · 95% interval 0.70 to 1.00 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 9 trace file(s) in out/traces/
Gate · suite regression · variant careful · threshold 0.70
PASS: Every critical case passed. There are no measured cases in this suite.
Every line is true. Nine situations, nine passes.
The office with the open water claim was not one of the nine on the day of that release. No one had imagined a submission that says "no losses" while a separate document says otherwise. The last row of the table, UW-009, is in the suite today because of what happened next. The rest of this page follows that case from the underwriter's click to the case file.
2. Two places to measure, and what each one lacks#
There are two places to measure an agent, and they fail in opposite ways.
The first is the one this site has used so far. You store cases in advance, run the agent on them away from real traffic, and compare what happened with the expected result. That is an offline eval. Its strength is the expected result: a grader (the thing that gives a score) can say right or wrong. Its weakness is the word "stored". Someone had to think of each case.
The second place is real traffic. You measure what the agent did on requests that real brokers sent. That is an online eval. Its strength is that nobody chose the inputs, so it contains the situations nobody thought of. Its weakness is the mirror image: there is no expected result. When the agent recommended approve for the office, nothing in the system knew whether that was right.
Anthropic's article on agent evals describes the same trade: automated evals are repeatable and touch no user, yet can give false comfort when they do not match real usage; production monitoring shows real behavior, yet it is reactive, noisy, and has no ground truth to grade against. Its conclusion is to combine layers, because no single one catches everything (sources).
| Offline eval | Online eval | |
|---|---|---|
| Input | A stored case, chosen by people | A real submission, chosen by nobody |
| Expected result | Written in advance | None |
| Can say right or wrong | Yes, by comparing | Only for rules that hold on every run, or after a person looks |
| When it runs | Before release, on every change | After release, on live or recent traffic |
| What it cannot see | A situation that is not in the set | Whether an unreviewed answer was correct |
| In the example | run_evals.py --suite regression | The underwriter's decision on SUB-1009 |
Online evals find what is missing. Offline evals keep it found. The path between them is the loop in section 5.
Common confusion. Running the golden dataset against the production deployment is not an online eval. The inputs are still stored cases with expected results. It is an offline eval in a different environment, useful for checking the deployment, silent about real traffic.
3. What can be measured when nobody wrote the expected result#
Without an expected result, most of the graders from the earlier pages have nothing to compare with. recommendation_match needs the expected recommendation. Three kinds of measurement still work.
Rules that hold for every run. Some checks need no expected result because they are true of every correct run, whatever the input. Before any approve or decline, the agent must call check_eligibility once and search_guidance at least once, and the result must include the cited rule. A cited rule must be one that was retrieved. The agent must never write a decisions row. These are code checks on the trace (the ordered record of one run) and on the stored rows. They are cheap, so they can run on every recommendation. In the example they are the graders required_process and human_required, described on What to evaluate.
What the person did next. The underwriter approves, declines, asks for more, or decides against the recommendation. A decision against the recommendation is an override. It is the natural signal in this system, because the design already puts a person after every recommendation.
A closer look at a sample. A person, or a judge (a model used as a grader) checked against people as Judges describes, reads a selection of real runs and scores them with the rubric.
Here is the override on the office, as the example stores it. This is the decisions row and the second of the two audit events from the end state of UW-009, printed from a run.
{
"decisions": [
{
"submission_id": "SUB-1009",
"decided_by": "underwriter",
"decision": "decline",
"override": true,
"reason": "The broker's loss run shows an open water damage claim that is not on the application."
}
],
"audit_event": {
"event": "decision_recorded",
"submission_id": "SUB-1009",
"actor_role": "underwriter",
"actor_type": "human",
"override": true,
"at_ms": 3368
}
}
Three fields make this row usable as a signal. override is computed, not typed: the stand-in for the cockpit sets it by comparing the person's decision with the recommendation. reason is required. And the audit event names a role and the actor type human. An override with no reason says only that two parties disagreed.
Now the important caution. It is tempting to read the share of overridden recommendations as the agent's error rate. It is not, in either direction.
| What happened | Agent was right for its input | Agent was wrong for its input |
|---|---|---|
| Underwriter overrode | The person knew something the agent was not given, or the person erred | A caught error |
| Underwriter agreed | The ordinary good case | A missed error: nobody noticed |
The office is in the top-left cell. The agent was right for the submission it received, and the underwriter was right about the building. The bottom-right cell is the dangerous one, and overrides cannot see it at all. A person who sees a confident recommendation tends to accept it, so a wrong recommendation can pass through quietly. That is why the third kind of measurement, a reviewed sample that includes the agreed cases, is not optional.
Production pitfall. The recommendation influences the decision that is later used to judge the recommendation. If underwriters accept most of what they are shown, a high agreement rate measures their trust at least as much as the agent's quality.
Signals in fullEach online signal, who produces it, what it supports and what it does not
| Signal | Who produces it | What it supports | What it does not support |
|---|---|---|---|
| Required process followed | Code, on every trace | The steps were taken in this run | That the recommendation was right |
| Cited rule was retrieved | Code, on every trace | The citation is grounded in a retrieval | That the rule was the right one to apply |
No decisions row by the agent | Code, on the stored rows | The person-decides boundary held | Anything about quality |
| Tool errors, retries, fallbacks | Code, on every trace | The outside provider's health, as in UW-006 | That a fallback was the best response |
| Override with reason | The underwriter | A disagreement worth reading | An agent error, until adjudicated |
| Request for more information | The underwriter | The input may have been incomplete | A fault of the agent |
| Stated confidence | The model | A claim to compare with outcomes later | A probability of being right; see Confidence |
| Rubric score on a sample | A judge or a person | Quality of the written reasons on those runs | A rate for all traffic, unless the sample was random |
| Later outcome, such as a claim | The world, months later | The slowest and most real evidence | A verdict on one recommendation: one loss does not show the decision was wrong |
A score is an observed result. Confidence is a claim about uncertainty. Reliability needs repeated evidence. An online signal is an observation about real runs; turning it into a claim about quality takes adjudication and enough runs.
Later outcomes need one more caution. A declined building has no outcome to observe at all, so outcome data is evidence about the rules more than about a single run.
4. Nobody can read everything, so choose what to read#
Suppose the insurer handles 1,200 submissions in a week (an illustrative number). The code checks can run on all 1,200. Human review cannot. Someone has to pick which runs get read. Picking is called sampling, and how you pick decides what you are allowed to conclude.
There are two ways to pick, and they answer different questions.
A random sample gives every recommendation the same chance of being read. Its value is that a rate measured on it estimates the rate on all traffic. If reviewers read 200 randomly chosen recommendations and judge 6 of them wrong, the estimate for the week is 3 in 100, with an interval around it.
A targeted sample picks by a rule: every override, every run with a tool error, every recommendation with low stated confidence (the number the model gives about its own answer). It finds problems much faster, because it looks where problems are likely. But its rate means nothing for the whole. If half of the reviewed overrides turn out to be agent errors, that says nothing about the share of errors among the 1,100 runs nobody overrode.
The snippet below uses the interval function of the example, stats.py:wilson_interval, on two illustrative review counts. It shows how wide the estimate stays when the sample is small. The counts are invented; the arithmetic is the real function, run in examples/uw-evals.
from stats import wilson_interval
wrong, reviewed = 6, 200 # illustrative counts
low, high = wilson_interval(wrong, reviewed)
print(f"judged wrong: {wrong} of {reviewed} = {wrong / reviewed:.2f} · 95% interval {low:.3f} to {high:.3f}")
wrong, reviewed = 1, 20
low, high = wilson_interval(wrong, reviewed)
print(f"judged wrong: {wrong} of {reviewed} = {wrong / reviewed:.2f} · 95% interval {low:.3f} to {high:.3f}")
judged wrong: 6 of 200 = 0.03 · 95% interval 0.014 to 0.064
judged wrong: 1 of 20 = 0.05 · 95% interval 0.009 to 0.236
Twenty reviewed runs with one error are compatible with an error rate under 1 in 100 and with one near 1 in 4. Reliability is the home of intervals and sample size; this page only applies them.
| Layer | What is looked at | Share of traffic | What it may claim |
|---|---|---|---|
| Code checks | Process, citations, the person-decides boundary, tool errors | All of it | A count of rule violations on real traffic |
| Random review | Recommendation and reasons, scored with the rubric | A fixed random share | An estimated rate for all traffic, with an interval |
| Targeted review | Every override, every error, every must-never-happen event | Chosen by rule | Leads to investigate; no rate for all traffic |
Keep the last two rows apart in every report. A mixed number, part random and part targeted, estimates nothing.
Sampling in practiceSlices, fixed budgets, review capacity and what an interval assumes
Rare slices need their own sample. A slice is a group of cases that share a label, as Golden datasets explains. If one submission in fifty is a wind zone C building with a roof near the 25-year limit, a random sample of 200 holds about four of them. To say anything about that slice, sample within it on purpose (a stratified sample) and report it as its own rate. To combine slice rates into one overall rate, weight each by its share of real traffic, not by its share of the sample.
The interval assumes independent, comparable runs. The Wilson interval treats each reviewed run as one independent draw with the same chance of being wrong. Real traffic bends that: one broker may send forty similar buildings in a day, and one reviewer may be stricter than another. Treat the interval as a lower limit on your uncertainty.
Review capacity is the real budget. If two people can adjudicate 40 runs a week, the design starts from 40, not from the traffic. Spend part on the random sample so the rate stays measurable, and the rest on targeted review. A queue that grows faster than people can read it is a signal nobody is reading.
A judge can widen the sample, with the limits of a judge. It can score many more runs than people can, and its scores inherit its false passes and false fails. In the example, the scripted stand-in judge agrees with the human label on 4 of 6 answers and gives 2 false passes: an illustration, not a measurement.
5. From one override to a case every release must pass#
What turns that row into something that protects the next release? Seven steps, in a fixed order, because each removes a different risk: privacy first, then a single person's opinion, then forgetting.
Diagram: From a production signal to a golden case, and back to production
Read left to right. After the last box the loop starts again at the first, because monitoring is where the next signal comes from. A signal from real traffic becomes a cleaned trace, people agree on what should have happened, the agreed result becomes a case that every later release must pass, and monitoring watches for the next signal. The example follows the override in case UW-009.
- 1 · Notice
- Production signal: Something in real traffic says a recommendation may have been wrong: here, a person decided against it.
- 2 · Prepare
- Cleaned trace: The trace of that one request with names, addresses, contact details and real ids removed, and every field a rule reads kept exact.
- Human adjudication: Two or more qualified people read the cleaned trace and agree on what the agent should have done with the input it had.
- 3 · Keep
- Golden case: The input, the agreed expected result, labels, the policy version and who labelled it, written as one case file.
- Regression suite: The case joins the set of cases that every later version of the agent must still pass.
- 4 · Release
- Candidate release: A changed agent runs the suite, and the release gate reads the report and prints pass or block.
- Monitoring: After release, the signal that started the loop is counted on real traffic, so the next surprise is noticed.
The path is a feedback loop: something noticed on real traffic becomes a stored case, and the same thing is checked before every later release. Anthropic's article recommends the same source for cases: real failures, the bug tracker, the support queue. The walk below follows the office.
Step 1. The signal picks a run#
The override on SUB-1009 puts one run into the targeted review queue. Nothing is concluded yet. The signal's only job is to say: read this one.
Step 2. The trace is cleaned before anyone outside the case team reads it#
A production trace holds things a test file must never hold: the broker's email, the insured business, a street address, a building value, the provider's report, the name of the underwriter who decided. A golden dataset is copied to laptops, read in code review and kept for years. So the trace is cleaned first. Traces is the home of redaction, the removal of sensitive values from a trace; this section is about one question: what goes and what stays when a trace becomes a case.
One rule decides it. Keep exactly what a rule or a grader reads. Remove what identifies a person, a business or a place. Replace what is only needed to tie records together.
| In the production trace | In the case | Why |
|---|---|---|
| The broker's original email, with signature, phone number and attachments | Removed. A short synthetic email is written from the cleaned fields | The agent never read the email, only the submission. The email is where most personal data lives |
| Insured business name, contact person, street address, postal code | Removed | No rule reads them. They identify the customer |
| Broker's name, email address and brokerage | Replaced by a role: broker@brokerage.example | The role matters to the story; the identity does not |
| Real submission id and policy numbers | Replaced by a new id, SUB-1009. The mapping stays in the access-controlled production system | Lets an authorized person find the original; reveals nothing in the case file |
roof_age_years, wind_zone, occupancy, sprinklered, area_sq_ft | Kept exact | The eligibility rules read them. Changing 14 to "10 to 20" could change the outcome |
building_value_usd, year_built | Rounded, unless a rule reads the exact value | Exact values help re-identify a building; a rounded value serves the case equally well |
| Provider's roof report: image links, coordinates, report id | Removed. The age, the status and the method are kept | Coordinates are an address. The agent used only the age and the status |
| Underwriter's name or employee id | Replaced by the role: decided_by: underwriter | The audit grader checks that a human role decided, not who |
| The underwriter's free-text reason | Rewritten by a person to the fact that matters | Free text can hold a claim number or a claimant's name. No pattern match is trusted with it |
| Model inputs and outputs stored in spans | Scanned and cleaned like the email, or dropped | A span can quote any of the fields above |
| Dates and times | Made relative, or shifted | An exact time plus a region can point at one transaction |
Two rows carry most of the risk: free text, because a person can type anything into a reason field, and the spans of model steps, because they copy input text into places a field-by-field cleaner does not look.
The cleaning has to be checked like any other change, with one test: run the agent on the cleaned case and compare with the original trace. The recommendation, the cited rules and the trajectory (the sequence of steps) must match. If they do not, the cleaning removed something the behavior depended on, and the case no longer represents the incident.
In our insurance agent. Case UW-009 is what this step produces: ten submission fields, a four-line synthetic email, a role in place of each person, and one sentence of reason. Every case in the example was invented from the start, so nothing was cleaned in practice; the file shows the shape a cleaned case should have.
Privacy limitsWhy removing names is not anonymity, and what else a team must decide
Removing direct identifiers does not make a record anonymous. A combination of ordinary fields can identify one building: a very large warehouse of an unusual construction type, built in a known year, in a small region. If a case is that distinctive, change the fields that no rule reads until it is not, and run the behavior check again.
Often the right answer is a new synthetic case. The adjudicators write down what made the incident hard: here, a loss on a separate document that the submission says does not exist. Then a case is written from scratch with those decision-relevant facts and invented everything else. The production trace is evidence for the case, never its content.
Clean before storage where you can. Cleaning when a case is made protects the dataset, not the trace store. Decide which fields are never recorded at all, how long raw traces live, and who may open one.
Keep the link, guard the link. The mapping from SUB-1009 to the real submission is what lets an auditor confirm the case. It is also the one table that undoes the cleaning. It belongs in the production system, with its access rules, not in the eval repository.
Sending traces to a judge is a data transfer. Clean before the judge sees the text, or run the judge under the same data agreement as the agent itself.
This is engineering practice, not legal advice. Which fields count as personal data, and how long anything may be kept, depends on the law that applies and on the contracts with brokers and customers. The people who own privacy in the company decide that; this table is what an engineer brings to the meeting.
Step 3. People agree on what should have happened#
One underwriter disagreed with the agent. That is one opinion against one output. Before it becomes an expected result, two or more qualified people read the cleaned trace and agree. That is adjudication, and Judges describes how disagreement between graders is settled.
The question they answer is narrow: given the input the agent had, what should it have done? There are four possible findings, and they lead to different work.
| Finding | What it means | What follows |
|---|---|---|
| The agent was wrong for its input | It had the facts and misused them | A new case that fails today, and a fix |
| The input was incomplete | The fact that mattered never reached the agent | A change upstream, and a case for the boundary |
| The person erred | The recommendation was right and the override was not | Feedback to the reviewer; possibly a case that protects the right answer |
| The guidance is unclear | Two experts read the same rule two ways | A question for the policy owners; no case until it is answered |
For the office, the finding is the second one. All eight graders of UW-009 pass, including no_invented_blocker: the agent did not make up a reason to decline. The open claim was on the broker's loss run, a document the intake step did not turn into a field. So adjudication produces two outputs. One is a request to the intake team. The other is the case below, which pins down what must stay true at the boundary between agent and person.
Step 4. The agreed result is written as a golden case#
A golden case is a case whose expected result people agreed on. Here is the part of cases/UW-009.json that holds the agreement and where it came from.
{
"expected": {
"recommendation": "approve",
"citations": [],
"missing": [],
"requires_human": true,
"process": ["check_eligibility", "search_guidance"],
"end_state": {
"recommendations": 1,
"decisions": 1,
"decision": "decline",
"override": true,
"audit_actor_role": "underwriter"
}
},
"provenance": {
"labelled_by": "compliance analyst",
"role_is_synthetic": true,
"date": "2026-08-18",
"synthetic": true
}
}
The case expects approve, because for this input that is right. It does not ask the agent to guess at documents it cannot see. What it adds is the end_state block: after the person acts, there must be one decisions row with the decline, override true and a reason, and the audit event must name a human role. The case protects the record of the disagreement. If a later change lost the reason, or logged the agent as the decider, the loop would have nothing to start from.
Step 5. The case joins the regression suite#
The grader that reads the end state is graders.py:end_state_audit. It demonstrates a check on what is stored after a run, apart from anything the agent said. A pass proves that the row and the event have the agreed shape; it proves nothing about whether the underwriter's decision was wise.
def end_state_audit(case: dict[str, Any], trial: dict[str, Any]) -> Result:
"""After the underwriter acts: a decisions row with the override and its reason, and an audit event that names the person's role."""
want, state, problems = case["expected"]["end_state"], trial["end_state"], []
rows = state["decisions"]
if len(rows) != want["decisions"]:
problems.append(f"{len(rows)} decisions rows, expected {want['decisions']}")
for row in rows:
if row["decision"] != want["decision"] or row["override"] != want["override"] or not row["reason"]:
problems.append("the decisions row does not hold the decision, the override flag and a reason")
events = [e for e in state["audit_events"] if e["event"] == "decision_recorded"]
if not events or any(e.get("actor_role") != want["audit_actor_role"] or e.get("actor_type") != "human" for e in events):
problems.append("the audit event does not name the underwriter's role as the actor")
return _result("end_state_audit", not problems, "; ".join(problems) or "override, reason and human actor recorded")
Running the one case shows the result. A trial is one run of one case.
$PY run_evals.py --case UW-009
Eval run · suite UW-009 · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-009 1 1 1.00 approve -
trials 1 · passed 1 · pass rate 1.00 · 95% interval 0.21 to 1.00 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 1 trace file(s) in out/traces/
One trial, one pass, and an interval from 0.21 to 1.00: a single passing trial says the check can pass, not that it reliably does.
This case passes on the day it is written, because the finding was "input incomplete". A case from the first finding, "the agent was wrong", is different. It fails on the day it is written; that is why it exists. Such a case cannot go straight into a suite whose failures block releases, or it blocks everything until the fix lands. Anthropic's article describes the usual route: a new, hard case starts in a suite that measures what the agent cannot yet do, and when the agent passes it steadily, the case graduates into the regression suite. CI gates explains the suites.
Steps 6 and 7. The next candidate must pass it, and monitoring keeps watching#
From now on every candidate release runs UW-009 with the other eight. The last row of the output at the top of this page is a production incident that can no longer be forgotten.
The loop closes where it began. After the release, overrides are counted again, by reason. If the intake team's change works, overrides with the reason "fact on a document that was not in the submission" should become rarer. If they do not, the signal picks another run, and the loop turns again.
Under the hood. A case made from an incident should record where it came from: the date, the role of the people who agreed, and a pointer to the incident that only authorized people can follow. The provenance block of the case file is the place. A case nobody can trace back is a case nobody dares to change when the policy changes.
6. Drift: the suite stands still while the world moves#
The loop handles surprises that arrive one at a time. A slower problem arrives without any single surprise.
Suppose a new broker joins, and within two months the share of wind zone C submissions doubles (illustrative). No rule changed. The agent did not change. The regression suite passes exactly as before. But the suite was built for the old mix of buildings, and it now speaks for a smaller part of what really arrives. That slow movement is drift: the inputs, the rules, the model, a tool or a grader change over time, and an old measurement stops describing the present.
The example holds one case of it. In UW-007, a roof is 28 years old in zone C, and a similar building was approved last year. That approval was correct under policy fixture version 2025.4, where the limit was 30 years. Under the current version 2026.2 the limit is 25, and the expected result is decline. The past decision did not become wrong. The world it was right in ended on the effective date.
| What moves | How it shows on real traffic | What it does to the suite | In the example |
|---|---|---|---|
| Inputs | The mix of slices shifts; new kinds of submissions appear | Coverage shrinks without any failure | More zone C buildings than the suite was built for |
| Policy | A new version takes effect | Expected results of old cases go stale | UW-007: version 2025.4 against 2026.2 |
| Model | A new model version, or a changed prompt | Behavior shifts on cases near a boundary | Wording varies in UW-010; a share of trials do not escalate |
| Tools | The provider changes its method or its failure pattern | Stand-in tools no longer behave like the real ones | UW-006 assumes a timeout after 2000 ms |
| Graders | A judge model or a rubric is updated | Scores change with no change in the agent | A reason to version graders, as The eval harness says |
Drift is noticed by comparing a signal with itself over time, slice by slice: this month's override rate in zone C against last quarter's, with the interval of each. It is repaired through the same loop, by sampling the part of traffic that grew and making cases from it. The drift of a dataset itself, and how to version one, is on Golden datasets.
Common confusion. Drift is not a regression. In a regression, the agent gets worse on the same cases. In drift, the cases stop standing for what really arrives. A regression fails the suite; drift leaves it green.
7. Trying a candidate on real traffic: shadow, then canary#
So far the direction was from production back to the suite. The other direction matters as much. A candidate has passed the offline gate. That is a claim about stored cases. How does a team learn how it behaves on real submissions without betting every decision on it?
The cautious answer has two stages, and each exposes a little more.
Diagram: Four stages before a candidate serves everyone
Read left to right. Each stage answers a different question and exposes a little more. Passing one stage is not a claim about the next. At any stage after the first, a worse signal sends the candidate back.
- 1 · Offline
- Offline gate: The candidate runs the suites on stored cases. No real submission is involved.
- 2 · Shadow
- Shadow test: The candidate runs on a copy of real submissions. Its recommendation is stored for comparison and shown to no one.
- 3 · Canary
- Canary: A small share of real submissions gets the candidate's recommendation in the underwriter cockpit, and the signals are watched.
- Roll back: If a watched signal gets worse or a must-never-happen check fires, all traffic returns to the current version.
- 4 · Everyone
- Full release: The candidate becomes the current version for all submissions. Sampling, review and alerts keep running.
In a shadow test, the candidate runs on a copy of real submissions while the current version keeps serving. The candidate's recommendation is stored and shown to no one. The underwriter sees only the current agent's recommendation. Because both versions saw the same submissions, they can be compared pair by pair: where do they differ, and on those, which one matches what the underwriter decided?
In a canary, a small share of real submissions goes to the candidate for real. Its recommendation appears in the underwriter cockpit, and the signals from section 3 are watched. If they get worse, traffic goes back to the previous version. A canary shows what a shadow test cannot: how people respond to the new recommendations.
| Shadow test | Canary | |
|---|---|---|
| Who sees the candidate's output | Nobody | The underwriters on a small share of submissions |
| Comparison | Pair by pair, same submissions | Between two groups of different submissions |
| Signals available | Differences from the current version; match with the person's decision on the other version's recommendation | Overrides, review scores, process checks on the candidate's own runs |
| Main risk | Side effects, and the cost of a second run of every shadowed request | A wrong recommendation reaches a person |
| What passing means | No alarming differences on this traffic | No worse signals on this share, in this period |
Each stage is a separate claim. "Passed the gate", "quiet in shadow", "quiet as canary" and "safe for everyone" are four statements, and only the first three were observed. How a release moves through branches and environments is the subject of the sister site diff-to-deploy.sammuthu.com.
A person decides every submission here, which lowers the risk of a canary without removing it: a weaker candidate can pass because underwriters quietly correct it.
Shadow and canary, the hard partsSide effects, noise in paired comparison, small canaries and rollback
A shadow run must not act. The candidate calls the same tools as the current agent. Reads are safe if the provider allows the doubled load. Writes are not: record_recommendation must go to a shadow table, and nothing may send a message or create a task for a person. The safest design gives the shadow run its own tool set, where each writing tool is replaced by a recorder.
Two versions differ even when nothing changed. The same agent, run twice on one submission, does not always give the same recommendation: in the example, UW-010 escalates in 16 of 20 trials. So some differences between candidate and current version are plain variation. Measure that first, by shadowing the current version against itself. The baseline is context, not a threshold: three differences that all turn a decline into an approve matter however many differences are usual. The paired method itself is on Reliability.
A shadow label is biased toward the current version. The underwriter's decision was made while looking at the current agent's recommendation. Agreement with that decision favors the version the person saw. Send the pairs that differ to adjudication without showing which version said what.
A small canary sees only common failures. If a failure happens once in 500 submissions, a canary of 60 will most likely see none, and silence will look like safety. State in advance which signals the canary can detect at its size, and which only the offline suite and the code checks can guard.
Decide the stop rule before starting. Two kinds. A must-never-happen event, such as a decisions row written by the agent, stops the canary at once, on one occurrence. A rate signal, such as the override rate, is compared with the current version's rate using intervals, at a time fixed in advance. Looking every hour and stopping at the first bad hour reads noise as a finding.
Rollback is a feature to test. A canary is only as safe as the way back. Runs in progress, recommendations already shown and rows already written by the candidate all need a defined fate.
8. Alerts, review and the audit record#
Three practices hold the page together in daily work.
Alerts. An alert is a rule, set in advance, that says when a person must look. The two kinds from the canary apply to all traffic. Some events must never happen: the agent writes a decision, a recommendation is shown without a required check, a retrieved passage marked untrusted is cited as a rule. One occurrence is enough, and no rate is involved. Other signals are rates that move: overrides, fallbacks, review scores. For these, an alert compares a window with an earlier one and includes the interval, because a rate over 30 submissions swings widely with no cause at all. An alert of the second kind is a reason to investigate, not a verdict.
Human review. The review queue is where the random sample, the overrides and the alerts meet. It works only if reviewers score the same way and the queue is sized to the people who read it.
Audit. In a regulated business, someone will later ask of one decision: who decided, on what recommendation, under which rules? The record has to answer without anyone's memory. For the office it is the two audit events in the end state: the recommendation recorded by the actor underwriting-agent of type agent, and the decision recorded by the role underwriter of type human, with override true. Add the policy version (2026.2), the version of the agent and the trace id (tr-UW-009-01 in the example), and a later reader can reconstruct the run. The feedback loop starts from the same record, which is why UW-009 guards it with a grader.
| Practice | The question it answers | In the example |
|---|---|---|
| Alert on a must-never-happen event | Did a boundary break? | Any decisions row with decided_by set to the agent |
| Alert on a rate | Has a signal moved more than its usual variation? | Override rate in a slice, with its interval |
| Review queue | What do people conclude from the runs that were picked? | Random sample plus every override |
| Audit record | Who decided what, under which version? | The two audit events of SUB-1009 |
Plain English. Monitoring tells you something moved. Review tells you whether it matters. The audit record lets someone check both of you later.
In one breath#
Offline evals check the situations we thought of, and production is where the others show up, without an expected result attached. So on real traffic we run the checks that are true of every run, we treat an underwriter's override as a lead and not a verdict, and we review a random sample so that we can state a rate with an interval. When something turns up, we clean the trace, have qualified people agree on what should have happened, and write that as a case the next release must pass. A new version meets real submissions in stages, first in shadow where nobody sees it, then as a canary on a small share, and each stage is its own claim.
The same idea for a coding agent: a pull request that a reviewer rewrote by hand is the override, and the cleaned repository state plus the agreed fix becomes the next regression case.
What comes next#
The office is now case UW-009, the intake team has a request, and the next candidate cannot lose the record of a disagreement without failing a grader. What this page described is a good deal of machinery: storing traces, running checks on live runs, drawing samples, keeping review queues, comparing versions. The example has none of it; run_evals.py runs stored cases and stops. Frameworks reads the documentation of each tool and says which parts it runs for you and which parts stay with you.