Every term this site uses, in one place. Each card opens the same drill-down as the underlined phrases in the text: one line, why it matters here, what it is, what it is not, and how it shows up in the insurance example.
Three sentences hold the words apart. A score is an observed result. Confidence is a claim about uncertainty. Reliability needs repeated evidence.
AbstentionThe agent gives no answer of its own and says why, when the evidence is missing or in conflict.AdjudicationWhen graders disagree about an answer, a named person looks at all the evidence, sets the final label and writes down why.Adversarial evalAn eval whose cases are written on purpose to push the agent toward a wrong result.Agent harnessThe software that calls the model, runs the tools it asks for, enforces limits and records what happened.AgentA program that calls a language model again and again, running the tools the model asks for in between, until the work is done.AssertionOne check that compares something observed with something expected and answers pass or fail.BaselineA result from before a change, stored so that later results can be compared with it.BenchmarkA dataset and a fixed way of scoring it, shared so that different systems can be compared on the same footing.Blocking thresholdThe value a gate compares a metric against. Below it, the gate blocks the change.Calibration setA collection of answers that people have already labelled, used to adjust a judge and to measure how far it agrees with people.CalibrationA stated confidence is calibrated when, among all answers given with confidence 0.8, about 8 in 10 turn out right, and the same holds for every other value.CanaryGiving a new version to a small part of real traffic first, watching the signals, and rolling back if they get worse.Capability suiteA set of hard or variable cases, run many times, that shows how often the agent succeeds and whether that rate is moving.CaseOne input together with the result people expect for it, and labels that describe it.Code graderA grader written as code. It checks something that is true or false, and gives the same result every time for the same trial.Confidence intervalA range around a measured rate that says how far the measurement can be trusted. Few trials give a wide range; many trials give a narrow one.ContaminationThe case or its expected result reached the agent before the trial, so a pass shows recall and not ability.Counterfactual testA copy of a case with one fact changed, to check that the answer changes when the rule says so and stays when the fact should not matter.DeepEvalA code library that lets you write evals as test files: a test case, one or more metrics with a threshold, and a command that runs them like unit tests.DriftA slow change in the inputs, the rules, the model, a tool or a grader that makes an old measurement stop describing the present.End stateWhat is true in the stored rows and events after the trial, apart from what the agent said.Eval harnessThe program that feeds cases to an agent, gives every trial a clean start, calls the graders and writes the report.EvalA measurement of how a system behaves over a set of cases, scored in a way that was written down before the run.Expected calibration errorOne number for a whole reliability table: the gap between stated confidence and observed accuracy in each group, averaged with each group weighted by its share of the answers.Expected resultWhat people decided the result for one input should be, written down before any run.False pass and false failA false pass: the grader says pass and the reference label says fail. A false fail: the grader says fail and the reference says pass.Feedback loopThe path that turns something noticed on real traffic into a stored case, so the same failure is checked before every later release.FixtureWhat is put in place before a trial so that every trial starts from the same surroundings.Flaky evalA case that sometimes passes and sometimes fails on the same agent, the same graders and the same data.Gold labelThe expected result of one case, decided by a person with the authority to decide it, with a date.Golden caseOne case whose expected result people agreed on, stored with everything that result depends on.Golden datasetThe golden cases of one task, kept under version control, with an owner and agreed rules for changing them.GraderThe thing that looks at a trial and gives a score with a reason. It can be a function, a model or a person.Ground truthThe correct answer for an input. For a judgment call it is what qualified people agreed on.Hard-failA condition that fails an answer by itself. No score on any other criterion can make up for it.HoldoutA part of the golden dataset that is kept out of day-to-day development and run rarely.Human graderA person with the knowledge to judge an answer. Their labels are what the other graders are checked against.Inter-rater agreementA measure of how far two graders, for example a judge and a person, label the same answers the same way.IsolationThe property that each trial starts from a clean world and shares no changeable state with any other trial.JudgeA model that is given an answer and a scoring guide and gives back scores. LLM stands for large language model.Judge gamingAn answer scores well with a model grader because of how it is written, not because it is correct.LangSmithA platform from LangChain that keeps traces, datasets and the results of each eval run, and shows them side by side.LeakageInformation from the eval cases reaches the agent or the people tuning it, so a pass no longer shows the agent can do the task.MetricOne number that summarizes the scores of many trials.Non-determinismThe property that running the same input again can give a different answer, a different path, or both.Offline evalAn eval that runs the agent on cases you stored in advance, each with an expected result, away from real traffic.Online evalAn eval that scores what the agent did on real requests, where no expected result was written in advance.OpenAI eval toolingThe eval features of the OpenAI platform: an Evals interface with graders, datasets in the dashboard, and grading of agent traces.OpenTelemetryAn open-source project that defines how programs describe and export traces, metrics and logs, so that the data does not depend on one vendor's format.Ordinal scoreA score that is one of a few named levels in a fixed order, such as missing, partly, fully.Output evalAn eval that compares what the agent handed back with the expected result of the case, and looks at nothing else.OverrideA decision by the underwriter that differs from what the agent recommended, stored with the reason.Paired comparisonComparing a candidate with a baseline by running both on the same cases, then looking at each pair of results, not at two separate pass rates.Pairwise judgingThe judge reads two answers to the same input and says which is better, or that they are equal.pass@kThe chance that at least one of k trials of a case passes. It answers: if we may try k times and keep a good result, do we get one?pass^kThe chance that every one of k trials of a case passes. It answers: if k people depend on this, is every one of them served correctly?PhoenixA server from Arize that receives traces over OpenTelemetry and adds datasets, experiments and graders on top of them.Pointwise judgingThe judge reads one answer and gives it scores. The scores are compared with a pass line.Policy complianceWhether the output, the path and the end state obey the rules that are in force for this trial.Policy fixtureA fixed, versioned set of invented underwriting rules that the agent retrieves and that every expected result is written against.Process evalAn eval that checks the steps of a trial against what the process requires.Prompt injectionText that reaches the model as material to read and is written to act as an instruction.PromptfooA command-line tool and library in which an eval is a configuration file: prompts, providers, test cases and assertions.ProvenanceThe record of who decided the gold label, when, from which source, and whether the case is synthetic.Quality gateA program that reads an eval report, applies a written rule, and answers yes or no to the question: may this change go on?RAGA design in which the system first retrieves relevant text from a store and then gives that text to the model with the question.RagasA Python library of ready-made and custom metrics for model applications, with small helpers for datasets and experiments.RedactionTaking out or replacing the values in a trace that identify a person, a property or a secret, while keeping the shape that graders and readers need.Reference-based judgingWith a reference, the judge compares a new answer with one that people already accepted. Without a reference, it has only the rubric and the answer.Regression suiteThe set of cases the agent is known to handle. Its job is to notice when a change breaks one of them.RegressionA behavior that held before a change and no longer holds after it.ReplayStoring what happened in a trial and using the stored copy later, to grade it again or to feed its tool answers to a new trial.ReproducibilityThe property that a run of the eval harness can be repeated, with the same cases, code and seed, and gives the same result.Retrieval similarityThe number a search step attaches to each passage it returns: how much the passage resembles the query.Risk scoreA number a business rule computes from the facts of a case to say how much is at stake. It is about the case, not about the agent.RubricA scoring guide: the list of things we check and how much each is worth.Sample sizeThe number of trials a metric was computed from. It decides how wide the interval is, and so what the metric lets you claim.SamplingPicking a part of real traffic to grade or review, because nobody can read all of it.ScoreWhat a grader gives for one trial: pass or fail, or a number.SeedA starting value for a random number generator. The same seed gives the same sequence of random choices, so a run can be repeated exactly.Selective predictionA system answers only when a number passes a threshold and hands every other case to something more careful, usually a person.Shadow testRunning a new version on a copy of real requests while the current version keeps serving, and storing the new version's answers for comparison only.SignificanceA claim that a difference between two variants is unlikely to be chance. It needs enough trials, a stated test and a line drawn before the run.Simulated userA script or a second model that plays the person on the other side, so a case with several turns can run without a human.SliceA subset of the cases, chosen by a tag or a field, with its own pass count.Smoke suiteA small set of critical cases, cheap enough to run on every change, that catches a change which breaks the agent outright.SpanThe record of one step: what kind of step it was, when it started, how long it took, what went in, what came out, and whether it failed.Stand-in toolA program with the same name, inputs and outputs as a real tool, which answers from data stored in the case.Stated confidenceThe number a model writes about how sure it is of its own answer. It is a claim about uncertainty, not a measurement.SubmissionOne JSON record that holds the facts of a broker's quote request. It is the input of the underwriting agent.SuiteA named group of cases that are run together for one purpose.Synthetic dataInputs, tool answers or whole cases that were written or generated, not taken from real traffic.TaskWhat the agent is asked to do, together with the definition of success.ThresholdThe value that decides whether a number counts as a pass.Tool callOne step in which the agent asks for a named function to be run with given arguments, and receives its result.Trace gradingGiving a score by reading the trace: which steps ran, how often, in what order, with what results, and whether the answer used them.Trace idThe identifier that says which trial a span belongs to. Every span of one trial carries the same one.TraceThe record of everything one trial did: each model call, retrieval, tool call, check and hand-off, in order, with times and results.TrajectoryThe sequence of steps the agent took in one trial, read from the trace.TrialOne run of the agent on one case, graded.Underwriter cockpitThe screen where the human underwriter reads the agent's recommendation, its reasons and citations, and approves or declines.UnderwriterThe person at the insurer who is authorized to approve or decline a submission.Underwriting agentThe agent of the insurance example: it reads a submission, gathers evidence, and writes a recommendation for an underwriter.VarianceThe spread of a measurement across repeats of the same setup. If the pass rate moves between runs of the same agent, that movement is variance, not progress.Weighted scoreOne number made from several criterion scores: each score times its weight, added, then divided by the most that sum could be.
Each term is explained in full on one page and used everywhere else. Where the words come from, and which of them this site uses differently from its sources, is on About.