The underwriting agent has finished trial 2 of case UW-010. The submission has a conflict in it: the application says the roof is 9 years old, and the roof-data provider says 22. The expected result is escalate, so that a senior underwriter settles the roof age. This trial recommends approve instead. (The model recommends; a person decides. An approve here is a recommendation to the underwriter.)
Here are the reasons the agent wrote:
The application gives a roof age of 9 years.
No rule in the policy fixture version 2026.2 blocks this submission.
Next step: the underwriter reviews this recommendation and decides.
Three short sentences. A source is named. A next step is given. Nothing in the text is false. The grader that scores written reasons gives it 0.90 out of 1.00, well above the pass line of 0.70.
A senior underwriter scores the same three sentences with the same scoring guide and gives 0.40. Her note: "Reads well, but it ignores the provider's roof age of 22 years. The conflict is never mentioned."
Two graders, one text, one scoring guide: 0.90 and 0.40. The first grader stands in for a model. This page is about what happens when a model grades a model's work: why we need it, how it goes wrong, and how far to trust it.
One thing first: the agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is. The grader that gave 0.90 is a scripted stand-in too. No model is called anywhere in the example.
Why code cannot grade the written reasons#
Most of what the agent produces can be checked by code. The recommendation is one of four words, so a code grader compares it with the expected word. The cited rule is an id, so a code grader looks for R-ROOF-03 in a list. Grading covered these graders, and they are the best kind when they fit: fast, free, and the same every time.
The written reasons do not fit. In the twenty trials of UW-010 the agent words its reasons in three different ways. Two of them:
Wording A
The application says the roof is 9 years old. The roof-data provider says 22 years.
R-ROOF-03: Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter.
Next step: escalate to a senior underwriter to settle the roof age.
Wording C
Two sources disagree on the roof age (9 and 22 years), so R-ROOF-03 applies.
Please review.
Both recommend escalate. Both cite the right rule. Every code grader passes both. Yet an underwriter who opens wording C does not learn which source said 9 and which said 22, and "Please review" does not say who should do what.
The question we want answered is: does the underwriter know what to do next, and who does it? Code can search for the words "next step", but an answer can contain those words and say nothing, or say it well in other words. The question needs a reader.
A senior underwriter can be that reader for 20 answers. She cannot read 20,000 on every change to the agent. So teams give the reading to a model.
Plain English. A judge is a model used as a grader. We give it the answer and a scoring guide, and it gives back scores. Tiny example: we show a model wording C and ask "do the reasons say where each fact came from?" It answers "no: two sources are mentioned, neither is named", and gives 0 for that item.
Elsewhere this is called an LLM judge (LLM stands for large language model). On this site the word is judge, and it always means a model. A person who grades is a human grader; a program that grades is a code grader.
What a judge is given: the rubric prompt#
A judge needs three things: the scoring guide, the answer, and instructions on how to reply. Together they form the rubric prompt.
The scoring guide is the rubric. A rubric is the list of things we check and how much each one counts (Grading is its home). The example's rubric for written reasons, from rubric.json, has five criteria. Each gets 0 (missing), 1 (partly) or 2 (fully):
| Criterion | The question the grader answers | Weight |
|---|---|---|
grounded | Is every reason backed by a tool result or a retrieved rule? | 3 |
cites_rule | Do the reasons name the rule that decides, or say plainly that no rule blocks? | 2 |
names_sources | Do the reasons say where each fact came from (the application, the provider)? | 2 |
actionable | Does the underwriter know what to do next, and who does it? | 2 |
clear | Are the sentences short and plain? | 1 |
The total is the weighted share of the maximum, from 0 to 1, and the pass line is 0.70. That total is a score: an observed result of grading one answer. It is not the probability that the recommendation is correct.
A rubric prompt for a real judge puts those questions in front of a model. The block below is a design, not something this site ran: no model was called with it. It shows the parts such a prompt needs for one criterion.
You are grading the written reasons of one underwriting recommendation.
You grade ONE criterion. Do not grade anything else.
Criterion: names_sources
Question: Do the reasons say where each fact came from (the application, the provider)?
Scale: 0 = missing, 1 = partly, 2 = fully.
Example of 2: "The application says the roof is 9 years old. The roof-data provider says 22 years."
Example of 0: "Two sources disagree on the roof age (9 and 22 years)."
Evidence the agent had (tool results from the trace):
{tool_results}
The reasons to grade:
{reasons}
First quote the sentences you rely on. Then explain in one or two sentences.
Then give the score. If the evidence is not enough to decide, answer "unknown".
Reply as JSON: {"quotes": [...], "why": "...", "score": 0, 1, 2 or "unknown"}
Each part is there for a cause. The single criterion keeps a good impression on one question from lifting the score on another. The two examples pin the scale, because "partly" means different things to different readers. The tool results are there because of the opening of this page: a judge that sees only the answer cannot see what the answer left out. The explanation comes before the score so that the score follows from the reading. And "unknown" lets the judge say it cannot tell, which is better than a confident guess.
Three of these ideas come from Anthropic's article Demystifying evals for AI agents: structured rubrics, one isolated judge per dimension, and a way out such as "Unknown". The same article notes that vague rubrics produce inconsistent judgments. The wording of the prompt above is this site's own design.
The judge in this example is a scripted stand-in#
The example does not call a model, so it cannot contain a real judge. It contains judge_sim.py: a short Python function that plays the part of a judge with keyword rules.
This is the function judge in examples/uw-evals/judge_sim.py, up to the point where the five scores are set. It shows what "reads only the text" means in code: every line works on the string text and nothing else. It looks for a rule id, for the words "application", "provider" and "policy fixture", for an action word, and it counts words per sentence.
def judge(output: dict[str, Any]) -> dict[str, Any]:
"""Score the written reasons on each rubric criterion (0, 1 or 2), with a one-line why."""
text = " ".join(output["reasons"])
low = text.lower()
has_rule = bool(RULE_ID.search(text))
sources = [w for w in ("application", "provider", "policy fixture") if w in low]
sentences = [s for s in re.split(r"(?<=[.!?])\s+", text) if s]
mean_words = sum(len(s.split()) for s in sentences) / max(1, len(sentences))
evidence = len(sources) + has_rule
names = 2 if {"application", "provider"} <= set(sources) else 1 if sources or "sources" in low else 0
scores = {"grounded": min(2, evidence),
"cites_rule": 2 if has_rule or "no rule" in low else 0,
"names_sources": names,
"actionable": 2 if ACTION.search(text) else 0,
"clear": 2 if mean_words <= 22 else 1 if mean_words <= 30 else 0}
Be clear about what this stand-in can teach. It shares one property with a model judge: it reads only the answer and is sometimes wrong. It does not share the rest. A real judge reads language far better than keyword rules, it can give different scores to the same answer on two runs, and it fails in ways a keyword list cannot. So the numbers below show how to measure a judge. They say nothing about how good any real judge is.
The next block runs the stand-in on the three sentences from the top of the page. A successful run shows where the 0.90 came from, criterion by criterion. It does not show that 0.90 is right. Run from examples/uw-evals; run here on 2026-11-17.
import judge_sim, rubric
answer = {"reasons": [
"The application gives a roof age of 9 years.",
"No rule in the policy fixture version 2026.2 blocks this submission.",
"Next step: the underwriter reviews this recommendation and decides."]}
judged = judge_sim.judge(answer)
for criterion, score in judged["scores"].items():
print(f"{criterion:14} {score} {judged['why'][criterion]}")
result = rubric.rubric_result(judged["scores"], [])
print("total", result["total"], "passed", result["passed"])
grounded 2 2 kinds of evidence named
cites_rule 2 a rule id or 'no rule' appears
names_sources 1 sources found: application, policy fixture
actionable 2 an action word appears
clear 2 10 words per sentence on average
total 0.9 passed True
Read the first line of the output. grounded gets full marks because the text names two kinds of evidence. The underwriter gave grounded a 0, because the most important evidence, the provider's 22 years, is missing. The stand-in counts what is on the page. She compared the page with what the agent knew.
Two ways to ask, and two kinds of evidence#
So far the judge looked at one answer and gave it scores. That is one of two ways to ask.
One answer at a time. The judge reads one answer and scores it against the rubric. Wording C gets a number, and the number is compared with the pass line. This is called pointwise judging. It suits a release gate, because a gate needs each trial to pass or fail by itself.
Two answers side by side. The judge reads two answers to the same input and says which one is better, or that they are equal. Show it wording A and wording C for UW-010 and ask which gives the underwriter a clearer next step. This is called pairwise judging. It suits a comparison of two versions of the agent, because "which is better" is often an easier question to answer consistently than "how good, on a scale". It does not tell you whether either answer is good enough. Two poor answers still produce a winner.
The second choice is what the judge may look at besides the answer.
With a reference. A reference is an answer people already accepted for this case, for example the reasons a senior underwriter approved for UW-004. The judge compares the new answer with it: are the same facts there, is the same rule cited? This is called reference-based judging. It is the stricter kind, and it needs someone to have written the reference.
Without a reference. The judge has only the rubric and the answer, as in the stand-in above. This is reference-free judging. It is the only kind available on live traffic, where no one has written the expected answer yet.
| Choice | The judge sees | It answers | Good for | Weak spot |
|---|---|---|---|---|
| Pointwise | One answer and the rubric | How well does this answer meet each criterion? | A gate with a pass line | The whole scale can run lenient or strict |
| Pairwise | Two answers to one input | Which answer is better? | Comparing two versions of the agent | Says nothing about good enough; the order of the two can sway it |
| Reference-based | The answer and an accepted answer | Does this say what the accepted answer says? | Golden cases | Punishes a correct answer that is worded or built differently |
| Reference-free | The answer and the rubric only | Does this meet the rubric? | Live traffic, new cases | Cannot see what the answer left out |
The two choices combine: a judge can be pointwise and reference-free, as the stand-in is, or pairwise with a reference. The last cell of the table is the opening of this page again: a reference-free judge that sees only the text passed the fluent approve.
Is the judge right? Ask people the same question#
A judge gives a score to everything you hand it, in the same confident format, whether it is right or wrong. Nothing in its output tells the two apart. The only way to find out is to have people grade some of the same answers and compare.
The example does this with six answers. A senior underwriter (a role, not a person: the labels are synthetic) scored each one with the same rubric. Her result is called a human label. The file is human_labels.json, and the ids run from HL-01 to HL-06.
Now two words for the two ways a judge can be wrong, with the human label as the reference:
Plain English. A false pass: the judge says pass, the person says fail. A poor answer gets through. A false fail: the judge says fail, the person says pass. A good answer is rejected. Tiny example: the 0.90 at the top of the page is a false pass, because the person gave 0.40.
In classification these are called false positive and false negative. This site says false pass and false fail, because "positive" does not tell you which side is the good one.
The two errors do not cost the same. A false fail costs time: someone looks at a good answer and clears it. A false pass is silent: the report says pass, nobody looks, and the poor answer counts toward the pass rate that a release gate reads.
Here are the six answers with all three graders. The lab under the table shows the same rows with the full text and the criterion scores.
| Label | Answer | Code graders | Stand-in judge | Human label | Final label |
|---|---|---|---|---|---|
HL-01 | UW-001, decline, full reasons | pass | 1.00 pass | 1.00 pass | pass |
HL-02 | UW-010 trial 6, escalate, wording A | pass | 1.00 pass | 1.00 pass | pass |
HL-03 | UW-010 trial 1, escalate, wording B | pass | 1.00 pass | 1.00 pass | pass |
HL-04 | UW-010 trial 3, escalate, wording C | pass | 0.75 pass | 0.60 fail | fail |
HL-05 | UW-010 trial 2, approve, reads well | fail | 0.90 pass | 0.40 fail | fail |
HL-06 | UW-008 shortcut run, decline, full reasons | fail | 1.00 pass | 1.00 pass | fail |
The stand-in judge passes all six. The human label passes four. So the judge agrees with the person on four answers, gives two false passes (HL-04 and HL-05), and gives no false fail. All numbers here are synthetic.
Three graders, three blind spots#
The last column of the table, the final label, is not the human label. Look at HL-06. The person said pass and the final label is fail. Someone had to settle each row where the graders disagreed.
Plain English. Adjudication is that settling: when graders disagree on an answer, a named person looks at all the evidence, decides the final label, and writes down why. Tiny example: for HL-06 the adjudicator opened the trace, saw that the guidance was never searched, and set the label to fail.
The three disagreements are worth reading one at a time, because each one exposes a different grader.
HL-04: the code graders are blind. Wording C recommends escalate and cites R-ROOF-03, and that is all the code graders check. They pass. The stand-in judge gives 0.75: it found the word "sources" and the word "review" and gave credit for both. The person gives 0.60: no source is named and "Please review" names no one. The human label stands. The recorded fix is in the judge: add this answer to its examples as a failing one.
HL-05: the judge is blind. This is the fluent approve. The code grader fails it at once, because the expected recommendation is escalate. The person fails it too. Only the judge passes it, because a text can be clear, sourced and actionable and still be about the wrong thing. The recorded fix: give the judge the tool results, and keep the code grader as the check that decides.
HL-06: the person is blind. This answer comes from the run of UW-008 in which the agent skipped search_guidance and wrote the rule from memory. Its text is word for word the text of a careful run, so the judge gives 1.00 and so does the person. Only the code check that reads the trace knows the cited rule was never retrieved. The code check stands. The recorded fix: show the labeller the trace next to the answer.
Diagram: Three graders, three blind spots: each one is right only about what it was shown
Read left to right. One trial leaves two things behind: the answer text and the trace. The judge and the person read the text. The code graders read the output fields and the trace. When they disagree, a person looks at everything and sets the final label. All labels here are synthetic.
- 1 · What one trial leaves
- Answer: The output of one trial: the recommendation, the cited rules and the written reasons.
- Trace: The ordered record of the trial: which tools ran and what they returned.
- 2 · Three graders
- Code graders: Programs that compare output fields with the expected result and check the trace. They cannot read prose.
- Stand-in judge: A scripted stand-in for a model judge: keyword rules that score the written reasons on five criteria.
- Human label: A person scores the same reasons with the same rubric. The best reference available, as good as the evidence the person had.
- 3 · Settle and reuse
- Adjudication: When graders disagree, a named person reads the answer and the trace, sets the final label and writes down why.
- Calibration set: The answers with their final labels, kept to tune the judge and to measure it.
The diagram makes one point: each grader can only be right about what it was shown. The judge and the person read the answer text. The code graders read the output fields and the trace (Traces explained that record). Whoever lacks a piece of evidence is wrong in exactly the cases where that piece matters.
That includes people. Human labels are often called "ground truth", and HL-06 is a human label that was wrong, because the labeller was handed less than she needed. A human label is the best reference available. It is as good as the evidence and the instructions the person had.
In our insurance agent. The harness does not let the stand-in judge decide alone. Ingraders.pythe functionrubric_gradertakes the judge's criterion scores and then runsrubric.detect_hard_fails, which is code that reads the trace. For theUW-008shortcut run the report sayshard-fail (cites_unretrieved_rule): 1.00 capped at 0.20. The judge gave full marks; code capped them. ForUW-010trial 2 the rubric grader passes with 0.90, and the trial still fails, becauserecommendation_matchfails and a trial passes only if every grader passes.
One kind of trial slips through all of this. UW-010 trial 3 is wording C. Its code graders pass and its rubric score is 0.75, so the run counts it as a pass. The adjudicated label says fail. Trials 5, 9, 14 and 20 carry the same text and the same 0.75. So the headline of that run, 16 of 20 trials passed, includes five trials whose reasons a senior underwriter would not accept. A false pass does not announce itself. It sits inside a pass rate.
How much agreement is that? Count it two ways#
"Agrees four times out of six" is a start. Two numbers make it precise, and the second one exists because the first one flatters.
Raw agreement#
Raw agreement is the share of answers on which the two graders give the same label. Here the judge and the person agree on HL-01, HL-02, HL-03 and HL-06:
raw agreement = answers with the same label / all answers = 4 / 6 = 0.67
That sounds like a judge that is right two times in three. Now look at how it got there: the judge passed everything. A grader that prints "pass" without reading a word would get the same 0.67, because four of the six answers deserve a pass. Raw agreement cannot tell a careful grader from a rubber stamp when most answers are good, and in a working system most answers are good.
Cohen's kappa: agreement beyond chance#
The repair is to ask: how much agreement would these two graders reach by accident, given how often each one says pass? Then count only the agreement above that.
Start with a table of the six answers, judge against person:
| Person: pass | Person: fail | Total | |
|---|---|---|---|
| Judge: pass | 4 | 2 | 6 |
| Judge: fail | 0 | 0 | 0 |
| Total | 4 | 2 | 6 |
The judge says pass 6 times in 6, so its pass share is 1.00. The person says pass 4 times in 6, so her share is 0.67. If two graders with those habits labelled at random, they would both say pass on 1.00 × 0.67 = 0.67 of the answers and both say fail on 0.00 × 0.33 = 0.00 of them. The agreement expected by chance is the sum:
expected by chance = (6/6 × 4/6) + (0/6 × 2/6) = 0.67 + 0.00 = 0.67
Cohen's kappa, introduced by Jacob Cohen in 1960, puts the two numbers together:
kappa = (observed − expected) / (1 − expected)
= (0.67 − 0.67) / (1 − 0.67)
= 0.00
Kappa is 1 when two graders always agree and 0 when they agree no more often than chance would produce. Here it is 0. Raw agreement said 0.67; kappa says this judge adds nothing beyond its habit of saying pass. Both numbers are correct. They answer different questions.
The chance level assumes that the two graders label independently of each other: neither sees the other's labels. That holds in this example by construction.
To see kappa move, suppose the judge is repaired so that it fails HL-05 and still passes HL-04. This is a what-if, not a result of the example. The judge then says pass 5 times in 6. Observed agreement is 5/6 = 0.83. Expected by chance is (5/6 × 4/6) + (1/6 × 2/6) = 0.61. Kappa is (0.83 − 0.61) / (1 − 0.61) = 0.57. One repaired answer moves raw agreement from 0.67 to 0.83 and kappa from 0.00 to 0.57.
The same numbers from the example's code#
The count of agreements, false passes and false fails is the function agreement in examples/uw-evals/stats.py:
def agreement(judge_pass: list[bool], human_pass: list[bool]) -> dict[str, Any]:
"""How often a judge agrees with human labels. The human label is the reference."""
pairs = list(zip(judge_pass, human_pass))
return {"n": len(pairs), "agree": sum(j == h for j, h in pairs),
"false_pass": sum(j and not h for j, h in pairs), "false_fail": sum(h and not j for j, h in pairs),
"agreement_rate": sum(j == h for j, h in pairs) / len(pairs) if pairs else 0.0}
The example has no kappa function, so the block below adds one of six lines for this page. It checks the hand calculation above and then asks two more questions: how does the judge compare with the final labels, and how does the person? A successful run proves the arithmetic. It proves nothing about any real judge. Run from examples/uw-evals; run here on 2026-11-17.
import stats
# One entry per labelled answer, HL-01 to HL-06. True means "pass".
judge = [True, True, True, True, True, True]
human = [True, True, True, False, False, True]
final = [True, True, True, False, False, False] # after adjudication
def cohen_kappa(a: list[bool], b: list[bool]) -> float | None:
"""Cohen's kappa for two graders and two labels. None when chance agreement is 1."""
n = len(a)
observed = sum(x == y for x, y in zip(a, b)) / n
by_chance = (sum(a) / n) * (sum(b) / n) + (1 - sum(a) / n) * (1 - sum(b) / n)
return None if by_chance == 1 else (observed - by_chance) / (1 - by_chance)
print("judge against human labels:", stats.agreement(judge, human))
print("kappa:", cohen_kappa(judge, human))
print("judge against final labels:", stats.agreement(judge, final))
print("kappa:", cohen_kappa(judge, final))
print("human against final labels:", stats.agreement(human, final))
print("kappa:", round(cohen_kappa(human, final), 2))
low, high = stats.wilson_interval(4, 6)
print("95% interval for 4 of 6:", stats.two(low), "to", stats.two(high))
judge against human labels: {'n': 6, 'agree': 4, 'false_pass': 2, 'false_fail': 0, 'agreement_rate': 0.6666666666666666}
kappa: 0.0
judge against final labels: {'n': 6, 'agree': 3, 'false_pass': 3, 'false_fail': 0, 'agreement_rate': 0.5}
kappa: 0.0
human against final labels: {'n': 6, 'agree': 5, 'false_pass': 1, 'false_fail': 0, 'agreement_rate': 0.8333333333333334}
kappa: 0.67
95% interval for 4 of 6: 0.30 to 0.90
Three things to notice. Against the final labels the stand-in judge has three false passes, not two, because HL-06 fooled it as well. The person agrees with the final labels five times in six, with a kappa of 0.67: better than the judge, and not perfect. And the last line is a warning about all of it: with six answers, the 95% interval around an agreement of 4 in 6 runs from 0.30 to 0.90. Six labels illustrate the method. They measure nothing. Reliability explains that interval.
Kappa in depthThe general formula, when it is undefined, why it depends on the mix of answers, and a weighted form for 0, 1, 2 scores
The formula. For two graders and any number of labels, kappa = (p_o − p_e) / (1 − p_e). p_o is the observed share of items with the same label. p_e is the agreement expected by chance: for each label, multiply the share of items the first grader gave that label by the share the second grader gave it, then add the products.
Values. Kappa is 1 for complete agreement and 0 when agreement equals the chance level. It can be negative, when two graders agree less often than chance would produce.
When it is undefined. If p_e is 1 the denominator is 0. That happens when both graders give one and the same label to every item, for example a judge and a person who both pass all six answers. The data then contain no information about how the two behave on a failing answer. The function above returns None for that case. The right response is to add failing answers to the labelled set, not to report a number.
It depends on the mix. Kappa is not a property of the judge alone. It depends on how common each label is in the labelled set. The Wikipedia article on Cohen's kappa states that, other things being equal, kappas are higher when the labels are equally common. Two consequences. A kappa measured on a set that is half failures does not carry over to live traffic that is mostly passes. And two kappas are comparable only when they were measured on the same set.
Bands such as "substantial" are convention. A widely repeated table by Landis and Koch calls 0.61 to 0.80 "substantial" and 0.81 to 1.00 "almost perfect". The same Wikipedia article reports that the authors supplied no evidence for those bands and based them on personal opinion. Set your own line from the cost of a false pass in your system, and report the count of false passes next to any kappa.
Criterion scores are ordered. The pass or fail label throws information away. The rubric gives 0, 1 or 2 per criterion, and a judge that says 1 where the person says 2 is closer than one that says 0. Weighted kappa counts near misses as partial agreement. In this example the stand-in judge and the person give the same criterion score in 24 of 30 places (six answers, five criteria). All six differences are in HL-04 and HL-05, and the largest is grounded on HL-05: 2 against 0.
Report it per criterion. One kappa over the total hides which question the judge cannot answer. In the example the trouble is concentrated in grounded, names_sources and actionable. clear agrees on all six answers. A judge can be trusted with one criterion and not with another.
Why a judge goes wrong#
The stand-in went wrong for two visible reasons: it rewards words it recognizes, and it sees only the answer. A real judge is as blind as its evidence too, and it has documented habits of its own. A 2023 study by Zheng and colleagues, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, names three biases in its abstract, together with limited reasoning ability.
Position. In a pairwise comparison, whether an answer is shown first or second can sway the judge. A common defence is to ask twice with the order swapped and to count only a verdict that survives.
Verbosity. A longer answer can look better to the judge even when the extra words add nothing. In the insurance agent this would reward reasons that restate the submission at length over three sentences that name the facts.
Self-enhancement. A judge may favor answers written by the same model as itself. If the underwriting agent and its judge run on one model, the judge may be that agent's most lenient reader.
The same abstract reports that a strong judge reached over 80% agreement with human preferences, the same level as agreement between humans. That is one study, on chat answers, with the models of 2023. It shows that a judge can agree with people about as well as people agree with each other. It does not show that your judge does on your rubric. Only your own labels can.
Common confusion. A judge's score is not the judge's confidence, and neither one is the chance that the answer is right. The 0.90 on the fluent approve was an observed result of applying one rubric to one text. The recommendation was wrong.
Calibrating a judge#
Calibrating a judge means adjusting it until its labels match people's labels closely enough, and proving that with labels it has not seen. The tool for both halves is a set of labelled answers.
Plain English. A calibration set is a collection of answers that people have already labelled, kept for one purpose: to tune a judge and to measure it. Tiny example: the six answersHL-01toHL-06are a very small calibration set.
The work is a loop, and the loop is where judge quality comes from.
Diagram: Calibrating a judge: tune on one part of the labelled answers, measure on the other
Read left to right. People label answers and the set is split before anyone touches the judge. The judge is tuned on the first part by reading each disagreement and changing one thing. The numbers that are reported come from the part that was set aside. A change to the judge sends it back to the start.
- 1 · People label
- Label answers: People score real answers with the rubric, with the trace next to the answer. Disagreements between people are adjudicated.
- Split the set: One part is put aside before the judge is changed, so that the final numbers are not the judge's memory of the fixes.
- 2 · Tune the judge
- Run the judge: The judge scores the same answers the people scored, with the same rubric.
- Compare: Count raw agreement, kappa, false passes and false fails, per criterion.
- Read disagreements: Each disagreement has a cause: a vague rubric, missing evidence, a wrong label or a habit of the judge.
- Change one thing: Sharpen one question, add the disagreement as a worked example, or give the judge more evidence. Then run again.
- 3 · Prove and keep
- Measure: The agreement that is reported comes from answers the judge was not tuned on, with the size of the set and the date.
- Freeze the version: The judge's model, prompt and examples are recorded as one version. A change to any of them means measuring again.
- Sample for a person: A small share of judged answers goes to a person, so that drift shows as falling agreement.
Collect and label. Take real answers and make sure failures are well represented: borderline answers, fluent wrong ones, correct but terse ones. A set in which everything passes cannot show a false pass. Give the labellers the trace next to the answer, and adjudicate where two people disagree.
Split. Put one part aside before touching the judge. You tune on the first part and measure on the second. If you tune and measure on the same answers, the agreement you report is the judge's memory of your fixes. Golden datasets explains this split under the name holdout.
Run, compare, read every disagreement. Count raw agreement, kappa, false passes and false fails, per criterion. Then read each disagreement, because each has a cause and the causes are few: the rubric was vague, the judge lacked evidence, the label was wrong, or the judge has a habit. Anthropic's article makes the same point about graders in general: you will not know whether they work unless you read the transcripts and the grades.
Change one thing. Sharpen the question, add the disagreement as a worked example in the prompt, or give the judge more evidence. In the insurance agent, HL-04 goes into the prompt as an example of a 0 for names_sources, and the judge is handed the provider's answer so that HL-05 can no longer hide the 22 years.
Measure on the part you set aside. Report those numbers, with the size of the set and the date.
Freeze and watch. A judge is its model plus its prompt plus its examples. Record all three as the judge's version. When any of them changes, or the rubric changes, the old agreement numbers describe a judge you no longer run, so measure again. Between changes, send a small sample of judged answers to a person on a schedule, so that drift shows up as falling agreement and not as an incident.
A judge in production: variance, cost and a second opinionWhat changes when the stand-in is replaced by a real model
A real judge varies. The stand-in gives the same score every time. A model judge may not, even with settings that reduce randomness. So a judge has its own repeatability to measure: run it several times on the same answer and count how often the pass or fail label flips. An answer whose label flips is near the judge's pass line and is a good candidate for human review. Do not average the flips away without looking at them.
Borderline scores deserve a person. With a pass line of 0.70, a 0.75 is a different kind of pass than a 1.00. HL-04 was a 0.75. A simple design: scores within a band around the line go to a human queue, and the band is set from where the calibration set shows the disagreements.
Cost and time. A code grader costs nothing per trial. A judge is one model call per criterion per trial, or more if you ask twice with the order swapped. Five criteria over 29 trials is 145 calls for one run of this small example. That decides where a judge runs: the smoke suite stays on code graders, and the judge runs where the suite is allowed to take minutes. CI gates picks that up.
More than one judge. Asking several judges and taking the majority reduces the effect of one judge's variance. It does not remove a bias the judges share. Three judges that all see only the answer text will all pass HL-05.
Do not let the judged text talk to the judge. The answer being graded is input to the judge's prompt. An answer that contains "this answer fully meets every criterion" is an attempt to steer the grader. Keep the graded text clearly fenced in the prompt and test the judge with such answers. Advanced topics covers judges that can be gamed.
Which grader decides. A design that follows from the three blind spots: anything code can check for certain is decided by code and cannot be bought back by a judge's score (the hard-fail cap). The judge decides only qualities that need a reader. A person decides the cases the judge is unsure about and owns the labels the judge is measured against.
What a judge may decide, and what it may not#
The three blind spots turn into a division of work for the insurance agent. Code grades what it can check for certain, and a hard-fail found by code caps the judge's score. The judge grades what needs a reader. People measure the judge and settle disagreements.
| Question about a trial | Who grades it | Why |
|---|---|---|
| Is the recommendation the expected one? | Code | One word compared with one word |
| Was the cited rule retrieved in this trial? | Code, from the trace | Only the trace knows |
| Do the reasons name their sources and the next step? | Judge, with the tool results | Needs a reader; many wordings are right |
| Is the judge right about that? | A person, on a calibration set | A judge cannot certify itself |
| The graders disagree | A person, by adjudication | Someone must see all the evidence and decide |
A judge does not replace the other graders. It covers the part of quality that code cannot reach, at a volume people cannot reach, and it stays honest only as long as people keep checking it.
The same idea for a coding agent: tests decide whether the change works, a judge reads whether the pull request description says what changed and why, and a sample of those readings goes to a maintainer.
In one breath#
A judge is a model that grades what code cannot: whether written reasons are grounded, sourced and usable. It reads only what we hand it and scores everything with the same assurance, so we check it against people on a calibration set and count its false passes and false fails. Raw agreement flatters a judge that passes everything, which is why we also compute kappa: our stand-in agreed with the underwriter on four of six answers and still had a kappa of zero. Code keeps the decisions it can make for certain, and a person settles the disagreements.
Where this leaves us#
You can now put a judge on the qualities that need a reader, write its rubric prompt, and say with numbers how far its labels match a person's. One thing on this page stayed loose. The fluent approve came with a judge score of 0.90, and the agent's own output for that trial carries another number, a stated_confidence of 0.9. Both look like "how sure are we", and neither one is. Confidence and uncertainty takes five numbers that look alike and says what each one lets you claim.