All chapters
  1. Start hereA test checks an answer. An eval measures a behavior
  2. Foundations · 1Primitives: the small words every eval is built from
  3. Foundations · 2Grading and rubrics: who decides what good means
  4. Foundations · 3Golden datasets: the cases you trust
  5. Mechanics · 4What to evaluate: the answer, the path, and what was left behind
  6. Mechanics · 5Agent traces: the record you grade the path from
  7. Mechanics · 6Judges and calibration: when a model grades a model
  8. Mechanics · 7Confidence and uncertainty: five numbers that are not the same
  9. Mechanics · 8Reliability and statistics: one run proves little
  10. Systems · 9The eval harness: the program that runs the evals
  11. Systems · 10CI/CD quality gates: which evals may block a release
  12. Systems · 11Production evals: measuring what real traffic does
  13. Reference · 12Frameworks: what each eval tool runs for you
  14. Reference · 13The insurance lab: ten cases, end to end
  15. Reference · 14Advanced topics: where evals themselves fail
  16. ReferenceGlossary
  17. AboutAbout this site and its sources
Foundations · 2

Grading and rubrics: who decides what good means

A grader gives the score and a rubric says what to check. Code, model and human graders, weighted scores, and the failures that no good prose may outweigh.

What you will understand
Four words
A grader gives the score. A rubric tells it what to check. A metric adds scores up. A judge is one kind of grader.
Three kinds of grader
Code, a model, or a person. Each one fits a different kind of question.
A precise rubric
Why "rate the quality" starts arguments, and what a rubric must say so that two graders agree.
Weights and hard-fails
How several small scores become one number, and which failures no good prose may outweigh.

The underwriting agent has finished case UW-004. A broker asked for a quote on a masonry office. The application says the roof was replaced 9 years ago. The roof-data provider says the roof is 22 years old. The policy fixture, the illustrative rule set this site uses, has a rule for that: R-ROOF-03 says that when the two ages differ by more than 5 years, a senior underwriter must settle it. So the expected result is a recommendation to escalate. The model recommends; a person decides.

Case UW-010 runs this same input 20 times, and the wording of the answer varies from trial to trial. Here are the written reasons from two of those trials. Both recommend escalate. Both cite R-ROOF-03.

Trial 6
  The application says the roof is 9 years old. The roof-data provider says 22 years.
  R-ROOF-03: Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter.
  Next step: escalate to a senior underwriter to settle the roof age.

Trial 3
  Two sources disagree on the roof age (9 and 22 years), so R-ROOF-03 applies.
  Please review.

Every check from Primitives passes on both: the recommendation and the cited rule are the expected ones.

Now sit in the senior underwriter's chair. Trial 6 tells her which source gave which age and what she is asked to do. Trial 3 tells her that two sources disagree and asks her to "review". Which source said 22? Review what? She has to open the submission and work it out again, which is the work the agent was supposed to save her.

One answer is better than the other, and the checks we have cannot see it. Something has to read the reasons and say how good they are. Who or what does the reading? Against which standard? And how does "how good" become a number that a release decision can use? This page answers those three questions in that order.

One note before the first result: the agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is.

1. Four words that get mixed up#

Start with the thing that produces a score. A grader is whatever gives the score: a piece of code, a model, or a person. The check that compares the recommendation with the expected one is a grader. So is the senior underwriter when she reads trial 3.

A grader needs to know what "good" means. A rubric is a scoring guide: the list of things we check and how much each is worth. One line of a rubric for the written reasons could be: "Do the reasons say where each fact came from?" Trial 6 does. Trial 3 does not.

These are different kinds of thing. The grader is the reader. The rubric is the page the reader holds. A rubric cannot score anything by itself, and a grader without a rubric scores by its own taste, which changes from one grader to the next.

Two more words sit close by. A metric is a number computed over many scores. In this example, 16 of the 20 trials of UW-010 pass, so the pass rate is 0.80. No single trial has a pass rate; Primitives is the home of that word. A judge is a model used as a grader. It is a kind of grader, not a fourth thing next to the others, and it has its own page: Judges and calibration.

The diagram puts the four words in the order in which they act on one trial.

Diagram: Four words on one trial · the rubric guides, the grader scores, the metric adds up

Read left to right. One trial produces an answer. A grader reads it, and a rubric tells that grader what to check. The result is one score. A metric appears only when the scores of many trials are put together. A judge is not a separate box: it is one of the three kinds of grader.

  1. 1 · What is graded
    1. One trial: One run of one case. Here the grader reads the reasons the agent wrote for its recommendation.
  2. 2 · The guide
    1. Rubric: The scoring guide: the list of things to check and how much each one counts. It scores nothing by itself.
  3. 3 · Who scores
    1. Code grader: A function that checks something that is true or false. It needs no rubric.
    2. Model grader: A model that reads the answer with the rubric and returns a score per criterion.
    3. Human grader: A person with the knowledge to judge the answer, using the same rubric.
  4. 4 · The result
    1. Score: The observed result of grading one trial: pass or fail, a level, or a number.
    2. Metric: A number computed from the scores of many trials. No single trial has one.
Common confusion. "The rubric gave it 0.75." A rubric gives nothing. A grader gave 0.75 while using the rubric. The difference matters on the day a score looks wrong, because there are two places to look: the guide, or the reader of the guide.

2. Three kinds of grader, and the question each one can answer#

Look at three questions someone could ask about trial 3.

Is the recommendation escalate? That is a fact about one field. Do the reasons say where each fact came from? That needs reading: "two sources disagree" mentions sources without naming one, and something has to decide whether that counts. Is escalation the right call for this building? That needs underwriting knowledge, and somebody with authority has to have decided it. In this example the case file records the expected result as labelled by a senior underwriter (a synthetic role).

Three kinds of question lead to three kinds of grader.

A code grader is a function that checks a fact. This is the whole grader for the first question, from graders.py:

def recommendation_match(case: dict[str, Any], trial: dict[str, Any]) -> Result:
    want, got = case["expected"]["recommendation"], trial["output"]["recommendation"]
    return _result("recommendation_match", got == want, f"expected {want}, got {got}")

It is fast, it costs almost nothing, and it returns the same result every time it sees the same trial. When it fails, the reason is one line: expected escalate, got approve. Its limit is that it reads characters, not meaning. It cannot tell trial 6 from trial 3.

A model grader reads text the way a person does and returns a score, for as many trials as needed. It can also be wrong, it can score the same text differently on different calls, and nobody should trust it before its scores have been compared with a person's. In this example its place is taken by judge_sim.py, which scores with keyword rules so that the example runs without a model, a key or a network.

A human grader is a person with the knowledge to judge the answer. For questions of expertise there is no other source, and a person's labels are what the other two graders are checked against. People are slow and expensive, and two people do not always agree. In this example, human_labels.json holds six synthetic labels by a senior underwriter.

This split, and the strengths and weaknesses in the table, follow Anthropic's article "Demystifying evals for AI agents" (sources are listed on About). The last column is this site's.

GraderGood forWeak atIn this example
CodeFacts: a field, a citation, a call in the trace, a stored rowMeaning; a valid answer in a form nobody expectedNine functions in graders.py
Model (a judge)Reading free text against a rubric, at volumeGiving the same score every time; being right without checks against peoplejudge_sim.py, a scripted stand-in
HumanExpertise; deciding what "right" is; checking the other twoSpeed, cost, volumeSix synthetic labels in human_labels.json

The standard that follows is short. A fact that code can check for certain must be checked by code. A model or a person gets only what needs reading or expertise. UW-004 follows it: the case lists six graders, five of them are code, and one, rubric, reads the reasons.

3. Three shapes a score can take#

Graders do not all return the same kind of result, and the shape decides what arithmetic is allowed later.

A binary score has two values: pass or fail. recommendation_match is binary. "Almost escalate" does not exist.

An ordinal score is one of a few ordered levels. The rubric in this example uses three: 0 for missing, 1 for partly, 2 for fully. Take the question "Do the reasons say where each fact came from?" Trial 6 names the application and the provider next to their ages: 2. An answer that named only one of them would be a 1. The levels have an order, so 2 is better than 1. They have no measured distance: nothing says that the step from 0 to 1 is as large as the step from 1 to 2.

A continuous score can take any value in a range. The rubric total later on this page is one: 0.75 on a range from 0 to 1.

ShapeValuesExample hereUse it when
Binarypass, failrecommendation_match on UW-004The thing is a fact, or a rule that must hold
Ordinala few named levels in order0, 1 or 2 per rubric criterion"Partly" is a real state and worth knowing
Continuousany value in a rangeThe weighted total, 0 to 1The number is computed or measured, not asked for

The last cell is a design rule of this site. A grader asked to say "0.73" for clarity has no way to tell 0.73 from 0.71. Ask a grader for a level it can defend, and compute the fine-grained number from several such levels.

4. The same rubric, written vaguely and written precisely#

Here is the first rubric a team usually writes for the written reasons. It is shown as a mistake.

Rate the quality of the reasons from 1 to 5.
Consider accuracy, completeness and clarity.

Grade trial 3 with it. One reader notices that both ages are correct, the rule is right and the answer is short. Under "accuracy" and "clarity" a 4 is defensible. A second reader notices that the answer never says which source gave which age, and that "please review" names no person and no step. Under "completeness" a 2 is defensible too. This is an illustration, not a measurement, but nothing in the rubric says which reader is wrong. The eval now measures the reader.

The vague version has three defects. It asks several questions and takes one number, so each reader weighs the three words in private. It uses words with no observable meaning: complete compared with what? And its levels have no names, so nobody can say what separates a 3 from a 4. Anthropic's article makes the same point about model graders: a vague rubric gives inconsistent judgments, and each dimension needs a clear, structured rubric.

The repair is to split the one question into several, make each one answerable by pointing at something in the answer or the trace, and name the levels. This is the rubric the example uses: examples/uw-evals/rubric.json, shown whole.

{
  "id": "reasons-rubric",
  "version": "1",
  "applies_to": "The written reasons of one recommendation.",
  "note": "Illustrative. A rubric score says how well the reasons meet this guide. It is not the probability that the recommendation is correct.",
  "scale": {"0": "missing", "1": "partly", "2": "fully"},
  "max_score": 2,
  "pass_threshold": 0.7,
  "hard_fail_cap": 0.2,
  "criteria": [
    {"id": "grounded", "label": "Grounded", "weight": 3, "question": "Is every reason backed by a tool result or a retrieved rule?"},
    {"id": "cites_rule", "label": "Cites the rule", "weight": 2, "question": "Do the reasons name the rule that decides, or say plainly that no rule blocks?"},
    {"id": "names_sources", "label": "Names its sources", "weight": 2, "question": "Do the reasons say where each fact came from (the application, the provider)?"},
    {"id": "actionable", "label": "Next step", "weight": 2, "question": "Does the underwriter know what to do next, and who does it?"},
    {"id": "clear", "label": "Clear", "weight": 1, "question": "Are the sentences short and plain?"}
  ],
  "hard_fails": [
    {"id": "claims_final_decision", "label": "Claims a final decision", "text": "The reasons say the agent approved, declined or bound the submission. Only a person decides."},
    {"id": "cites_unretrieved_rule", "label": "Cites a rule it did not retrieve", "text": "A cited rule was not returned by any search_guidance call in this trial."}
  ]
}

Leave the weights and the hard-fails for sections 5 and 6, and read the five questions under criteria. A criterion is one item of a rubric: one question with its own score. Now put the two versions side by side.

PropertyVague versionPrecise version (rubric.json)
What is graded"The reasons"The written reasons of one recommendation. The recommendation itself is left to code graders
QuestionsThree ideas, one numberFive criteria, one question each, one score each
What a grader looks atIts own impressionSomething it can point to: a source, a rule id, a next step, a person
Levels1 to 5, unnamed0 missing, 1 partly, 2 fully
What counts moreDecided in each reader's headWritten down as weights
When the answer passesNot stated0.70, written before any trial is graded
What can never passNot statedTwo hard-fail conditions
When two graders disagreeThey argue about "quality"They argue about one criterion of one answer

The last row is the reason to do this work. A precise rubric does not end disagreement. It makes a disagreement small enough to settle.

This rubric is not finished either. Its scale says "partly" once for all five criteria and never says what "partly" means for each. On trial 3 the stand-in judge gives names_sources a 1, because the word "sources" appears. The senior underwriter's label gives it a 0, with the note that the answer "does not say which source gave which age". Both read the same rubric. A production rubric must describe each level of each criterion, with an answer that shows it. Finding such gaps is the work of calibration, and Judges and calibration does it with this same answer.

5. From a checklist to a weighted score#

Five criterion scores are not yet a result. There are three ways to combine them, and each fixes a problem of the one before.

A checklist. Every criterion must be fully met, or the answer fails. It is easy to explain and it is harsh: an answer with every fact and source in one long sentence fails on clear exactly as hard as an answer with no grounding at all, and the result hides which of the two happened.

A rubric with levels. Each criterion gets 0, 1 or 2, and the score is the share of the points available. The stand-in judge scores trial 3 as 1, 2, 1, 2, 2. That is 8 of 10 points: 0.80. A partly good answer now gets partial credit. But every criterion counts the same, and that is a claim about the work. A long sentence costs the underwriter a few seconds. A reason that nobody can trace to evidence can mislead her.

Weights. A weight says how much one criterion counts compared with the others. In rubric.json, grounded has weight 3, three criteria have weight 2, and clear has weight 1. The weighted score takes three steps: multiply each score by its weight, add the products, and divide by the most that sum could be. For trial 3 as the stand-in judge scored it:

CriterionWeightScoreWeight × scoreMost possible
grounded3136
cites_rule2244
names_sources2124
actionable2244
clear1222
Total1520

15 of 20 is 0.75. The function that does this is weighted_total in rubric.py:

def weighted_total(scores: dict[str, int], rubric: dict[str, Any] = RUBRIC) -> float:
    """Weighted share of the maximum, from 0 to 1. A missing criterion counts as 0."""
    got = sum(c["weight"] * scores.get(c["id"], 0) for c in rubric["criteria"])
    best = sum(c["weight"] * rubric["max_score"] for c in rubric["criteria"])
    return got / best

The next script calls that function on trial 3 twice: with the stand-in judge's criterion scores, and with the senior underwriter's from human_labels.json (label HL-04). Each is computed with equal weights and with the weights of the rubric. A successful run proves the arithmetic of the table. It does not prove that either set of criterion scores is right. Run it from examples/uw-evals.

from rubric import RUBRIC, weighted_total

judge = {"grounded": 1, "cites_rule": 2, "names_sources": 1, "actionable": 2, "clear": 2}
person = {"grounded": 2, "cites_rule": 2, "names_sources": 0, "actionable": 0, "clear": 2}
equal = {**RUBRIC, "criteria": [{**c, "weight": 1} for c in RUBRIC["criteria"]]}

for name, scores in (("stand-in judge", judge), ("senior underwriter", person)):
    print(f"{name:<19} equal weights {weighted_total(scores, equal):.2f}   rubric weights {weighted_total(scores):.2f}")
print("pass line", RUBRIC["pass_threshold"])
stand-in judge      equal weights 0.80   rubric weights 0.75
senior underwriter  equal weights 0.60   rubric weights 0.60
pass line 0.7

The weights moved the judge's score from 0.80 to 0.75, because the criterion it marked down, grounded, is the heaviest. And the two graders land on opposite sides of the pass line with the same rubric and the same arithmetic. Weights decide how much a criterion counts. They do nothing about who is right on a criterion.

The lab takes the same steps with five answers: boxes, then levels, then weights. The table after it holds the same rubric.

CriterionQuestionWeight
GroundedIs every reason backed by a tool result or a retrieved rule?3
Cites the ruleDo the reasons name the rule that decides, or say plainly that no rule blocks?2
Names its sourcesDo the reasons say where each fact came from (the application, the provider)?2
Next stepDoes the underwriter know what to do next, and who does it?2
ClearAre the sentences short and plain?1

Each criterion is scored 0, 1 or 2. The pass line is 0.70. Two hard-fail conditions cap the score at 0.20: the reasons claim a final decision, or they cite a rule that no search_guidance call returned in that trial. In the lab the starting scores come from the stand-in judge.

Plain English. A weighted score is a share of the points available: 15 of 20. It says how well the written reasons meet this guide. A rubric score is never the probability that a recommendation is correct.

The example has a trial that proves the last sentence. In trial 2 of UW-010 the stand-in drops the provider's answer and recommends approve, which is wrong for this input. Its reasons are tidy, so the rubric grader gives 0.90. These are that trial's grader results, printed from out/report.json after run_evals.py --case UW-010:

trial 2 approve wording C passed False
   FAIL recommendation_match  expected escalate, got approve
   FAIL citations             expected R-ROOF-03; got none
   PASS missing_fields        expected none; got none
   PASS required_process      eligibility checked and cited rules retrieved
   PASS human_required        recommendation stored; decision left to a person
   PASS rubric                score 0.90 (threshold 0.7)

A high rubric score sits next to a wrong recommendation. The trial fails all the same, because a code grader checks the recommendation and a trial passes only if every grader passes.

6. The failure that no score may outweigh#

In a weighted sum, a high score on one criterion pays for a low score on another. Usually that is the point. Sometimes it must not happen.

Case UW-008 shows when. The facts are those of UW-001: a roof of 31 years in wind zone C. Under the policy fixture, rule R-ROOF-01 makes that not eligible, so the expected recommendation is decline. The example can run the agent in a variant named shortcut, in which the stand-in skips the search_guidance call and writes the rule text from memory.

The written reasons come out word for word the same as in a careful trial. So the stand-in judge, which reads only the text, gives 2 on every criterion: 1.00. On weights alone, an answer that cites a rule it never read gets the best score there is.

The rule text was right, but by luck. The earlier version of the policy fixture, 2025.4, set the limit of R-ROOF-01 at 30 years, and the current version sets it at 25. An agent that cites from memory is correct until the day the rule changes, and on that day nothing in its answer looks different.

The first idea is to add a criterion, "retrieved its rules", with a high weight. As an illustration: weight 3, score 0. The answer now has 20 of 26 points, 0.77, and still passes. No weight fixes this, because any weighted sum lets the other criteria buy the failure back.

So the rubric needs a second mechanism. A hard-fail is a condition that fails the answer by itself, whatever the other scores are. rubric.json names two. cites_unretrieved_rule: a cited rule was not returned by any search_guidance call in the trial. claims_final_decision: the reasons say the agent approved, declined or bound the submission, when only a person decides.

Diagram: From criterion scores to pass or fail · where the hard-fail sits

Read left to right. The judge gives a level per criterion and the weights turn the levels into one total. Code checks the hard-fail conditions separately, from the output and the trace. If one is found, the total is capped and the answer fails, whatever the total was.

  1. 1 · Read the reasons
    1. Criterion scores: One level per criterion, given by the stand-in judge or by a person.
  2. 2 · Add them up
    1. Weighted total: Each score times its weight, added, divided by the most that sum could be.
  3. 3 · Check what may never happen
    1. Hard-fail check: Code looks for the two conditions: a claimed final decision in the text, a cited rule that the trace shows was never retrieved.
    2. Cap: If a condition was found, the total becomes at most 0.20. With no condition it is unchanged.
  4. 4 · Decide
    1. Pass or fail: Pass needs both: no hard-fail, and a total of at least 0.70.

In rubric.py, detect_hard_fails finds the conditions, apply_hard_fail caps the total, and rubric_result puts the result together.

def apply_hard_fail(total: float, hard_fails: list[str], rubric: dict[str, Any] = RUBRIC) -> float:
    """A hard-fail caps the total. Good prose cannot buy back a critical failure."""
    return min(total, rubric["hard_fail_cap"]) if hard_fails else total


def detect_hard_fails(output: dict[str, Any], trace: dict[str, Any] | None) -> list[str]:
    """Hard-fail conditions that code can check for certain."""
    found: list[str] = []
    if FINAL_DECISION.search(" ".join(output["reasons"])):
        found.append("claims_final_decision")
    if trace is not None and not set(output["citations"]) <= retrieved_rule_ids(trace):
        found.append("cites_unretrieved_rule")
    return found


def rubric_result(scores: dict[str, int], hard_fails: list[str], rubric: dict[str, Any] = RUBRIC) -> dict[str, Any]:
    """Put criterion scores and hard-fails together into one result."""
    raw = weighted_total(scores, rubric)
    total = apply_hard_fail(raw, hard_fails, rubric)
    return {"scores": scores, "raw_total": round(raw, 4), "total": round(total, 4), "hard_fails": hard_fails,
            "passed": not hard_fails and total >= rubric["pass_threshold"]}

Three decisions in this code matter.

The hard-fails are found by code, not by the judge. cites_unretrieved_rule compares the citations with the rule ids that the trace shows were retrieved. (A trace is the ordered record of what one trial did; Agent traces is its home.) The judge could never have found this, because the text carries no sign of it.

The cap uses min, so it only ever lowers a score. A total of 1.00 becomes 0.20. A total of 0.15 stays 0.15.

And passed does not rely on the cap: it reads not hard_fails and total >= pass_threshold. The flag decides pass or fail. The cap is there for the number that goes into reports, so that an average over many trials does not count a hard-failed answer as 1.00.

The next script shows the cap alone, then the full result for an answer with 2 on every criterion and one hard-fail. A successful run proves that the cap and the flag behave as described. Run it from examples/uw-evals.

from rubric import apply_hard_fail, rubric_result

full = {"grounded": 2, "cites_rule": 2, "names_sources": 2, "actionable": 2, "clear": 2}
print(apply_hard_fail(1.0, []))
print(apply_hard_fail(1.0, ["cites_unretrieved_rule"]))
print(apply_hard_fail(0.15, ["cites_unretrieved_rule"]))
print(rubric_result(full, ["cites_unretrieved_rule"]))
1.0
0.2
0.15
{'scores': {'grounded': 2, 'cites_rule': 2, 'names_sources': 2, 'actionable': 2, 'clear': 2}, 'raw_total': 1.0, 'total': 0.2, 'hard_fails': ['cites_unretrieved_rule'], 'passed': False}

Those scores were typed in by hand. The eval harness (the program that runs the cases and calls the graders) does the same on a trial of UW-008 in the shortcut variant. The second block prints that trial's six grader results from out/report.json.

cd examples/uw-evals
/opt/homebrew/bin/python3 run_evals.py --case UW-008 --variant shortcut
Eval run · suite UW-008 · variant shortcut · seed 7 · agent: scripted stand-in (no model called)
case     trials  pass   rate  recommendation            failed graders
UW-008        1     0   0.00  decline                   required_process, rubric
trials 1 · passed 0 · pass rate 0.00 · 95% interval 0.00 to 0.79 (Wilson)
critical cases: 1 failed trials, 1 with a hard-fail · measured cases: 0 of 0 trials passed
wrote out/report.json and 1 trace file(s) in out/traces/
PASS recommendation_match  expected decline, got decline
PASS citations             expected R-ROOF-01; got R-ROOF-01
PASS missing_fields        expected none; got none
FAIL required_process      search_guidance was not called; cited but never retrieved: R-ROOF-01
PASS human_required        recommendation stored; decision left to a person
FAIL rubric                hard-fail (cites_unretrieved_rule): 1.00 capped at 0.20

The last line keeps both numbers: the 1.00 the text earned and the 0.20 that is reported, with the name of the condition. required_process is a second grader, which looks at the path the agent took; What to evaluate is about that kind of check. Under the careful variant the same case passes all six graders.

In our insurance agent. The two hard-fails are the two things an underwriter must be able to rely on without re-checking: every cited rule was read in this trial, and the text never presents a recommendation as a decision.

7. When the grader is the one that is wrong#

A grader can be wrong in both directions. The stand-in judge passed too much: it gave trial 3 a 0.75 where the senior underwriter's label fails it. Here is a grader that fails too much, and it is a hard-fail, the one with the most power.

claims_final_decision is found with a pattern over the text, and one part of that pattern is the two words "final decision". The script gives detect_hard_fails four sentences, with no trace, so that only this condition is checked. The first sentence breaks the rule. The other three are correct sentences an agent could write.

from rubric import detect_hard_fails

def check(sentence):
    output = {"reasons": [sentence], "citations": []}
    print(detect_hard_fails(output, None), "<-", sentence)

check("I have approved this submission and coverage is bound.")
check("Next step: the underwriter reviews this recommendation and decides.")
check("Next step: a senior underwriter confirms the roof age before any decision.")
check("Next step: the underwriter makes the final decision.")
['claims_final_decision'] <- I have approved this submission and coverage is bound.
[] <- Next step: the underwriter reviews this recommendation and decides.
[] <- Next step: a senior underwriter confirms the roof age before any decision.
['claims_final_decision'] <- Next step: the underwriter makes the final decision.

The fourth sentence says what the rule wants: the person decides. The pattern sees two words and reports a hard-fail. This is the weakness of code graders from section 2 where it hurts most: a valid answer in a form nobody expected, and a verdict that cannot be outweighed. None of the ten cases produces that sentence, so the example's results are not affected.

Two habits follow, and both come from Anthropic's article. Read the failing trials, not only the totals; a failure should look fair to a person who reads it. And test a grader like any other code, with answers it must fail and answers it must pass.

Production pitfall. A hard-fail that fires on a correct answer blocks a good change and teaches the team to ignore the gate. Before a condition becomes a hard-fail, collect correct answers that come close to it and confirm that it stays silent on them.

The four blocks below hold the detailed layer.

Arithmetic on ordinal scoresWhat a weighted total assumes about 0, 1 and 2

weighted_total multiplies and adds the levels 0, 1 and 2. That treats "partly" as half of "fully" on every criterion. Nothing measured that. It is a convention, and the total inherits it.

Three consequences. Two different answers can have the same total: a 2 and a 0 on two criteria of equal weight give the same sum as a 1 and a 1, although the first answer has a hole and the second has two soft spots. So report the criterion scores next to the total; the rubric grader does, in the field criteria. The mean of ordinal scores over many trials carries the same assumption, where a count per level (how many trials got 0, 1 and 2 on grounded) assumes nothing. And more levels give more information only if graders can tell them apart; the way to check is to measure how often two graders pick the same level, which Judges and calibration covers.

Choosing weights, and what a change of weights doesWeights are a decision by people, with a version

Weights are not measured. They record what the people responsible for the work think matters, and here those are underwriters, not engineers. Set them by consequence (what does it cost the underwriter when this criterion is missing?), then check them against answers that people have already ranked. If the weighted score puts an answer above one the underwriters call better, the weights or the criteria are wrong.

The pass line is the same kind of thing: 0.70 is a choice written into rubric.json, not a finding. Scores near the line deserve the least trust. Trial 3 sits at 0.75 by one grader and 0.60 by another.

A change of weights changes every score without any change in the agent: section 5 gave 0.80 and 0.75 for the same criterion scores. So a rubric needs a version (rubric.json has "version": "1"), every stored score must say which version produced it, and scores from two versions must not share a chart as one series. When the rubric changes, grade the stored answers again before comparing.

Designing hard-fail conditionsWhat qualifies, why a cap, and how to count them

A condition qualifies as a hard-fail when three things hold. The failure is unacceptable whatever else the answer does well. It can be decided for certain, in practice by code over the output, the trace or the end state. And it stays silent on correct answers, which section 7 shows is something to test. A condition that needs judgment ("the tone is unprofessional") is a criterion. Keep the list short: every hard-fail is a place where one mistake of the grader fails a good answer.

Why a cap and not zero? Both work if the capped value is below the pass line. This example uses min(total, 0.20) and keeps raw_total next to it. What matters more is that pass or fail comes from the list of hard-fails, not from the capped number.

A capped score is no longer a share of points, so a mean over trials that include capped scores is hard to read. Count hard-fails as their own metric. The report does: its summary line says critical cases: 1 failed trials, 1 with a hard-fail, and the release gate reads that count apart from the pass rate (CI/CD quality gates).

Combining graders, and what each kind costsAll must pass, a weighted mix, or both

The results of several graders have to become one result for the trial. Anthropic's article names three ways: weighted (the combined score must reach a threshold), binary (every grader must pass), or a hybrid. The example is a hybrid. Across graders it is binary; the docstring of grade_trial in graders.py reads "A trial passes only if all of them pass." Inside one grader, rubric, it is weighted with hard-fails. That allows compensation between qualities of the prose and forbids it between a wrong recommendation and good prose. The article also recommends partial credit for tasks with several parts; the levels 0, 1 and 2 are partial credit inside the rubric.

The kinds of grader differ in where they can run. Code graders are cheap enough for every trial of every suite. A model grader costs a model call per trial, or per criterion if each criterion gets its own call, which the article recommends so that one judgment does not color another. It also recommends letting a model grader answer "unknown" when the answer does not hold enough to decide. Human grading is the scarcest: spend it on labelling the cases, on checking the model grader, and on the trials where graders disagree.

In one breath#

A grader gives the score and a rubric tells it what to check, so when a score looks wrong you ask which of the two is at fault. Facts go to code, reading goes to a model or a person, and expertise goes to a person. A rubric becomes precise when each criterion is one question a grader can answer by pointing at something, with named levels. Weights turn the criterion scores into one number, and a hard-fail is for the few failures that no number may buy back.

What is still open#

You can now say what would be checked in a pile of written answers, by which kind of grader, on which scale, and what fails no matter what. Every grader on this page leaned on something it did not produce: the expected result. recommendation_match passes escalate on UW-004 because the case file says escalate, and the case file says so because a senior underwriter labelled it. If that label is wrong or out of date, every score built on it is wrong in the same direction. Golden datasets is about those cases: who decides the expected result, how cases are chosen, and how a dataset stops telling the truth.

The same idea for a coding agent: unit tests are the code graders, a rubric covers what tests cannot see, such as whether the change stays inside the task, and "deleted a failing test to make the suite pass" is a hard-fail.