The example has been in use for a month. Ten cases, ten graders, a trace for every trial and a gate, all in one folder of plain Python: examples/uw-evals/. The underwriting agent recommends, the underwriter decides, and before each release the gate prints PASS or BLOCK. The model recommends; a person decides.
Then three requests arrive in one week.
A second team wants to add 400 cases and asks where to put them. Two senior underwriters will label the answers that the graders disagree about, and they ask for a screen, because they do not read JSON files. And the head of underwriting asks what the pass rate of UW-004 was four weeks ago. UW-004 is the case in which the application says the roof is 9 years old and the roof-data provider says 22, so the agent should recommend escalate and cite rule R-ROOF-03. Nobody can answer. Every run overwrites out/report.json.
Someone writes: "We should adopt an eval framework." Within an hour the thread holds seven names: DeepEval, Promptfoo, LangSmith, Phoenix, Ragas, the eval tooling of OpenAI, OpenTelemetry. Each of them describes itself with the word evaluation. They are not seven versions of one thing. One is a library that runs inside the test runner. One is a server that stores traces. One is a list of agreed names and runs nothing at all. And one, on the day this page was written, carries a shutdown date in its own documentation.
The question that sorts them is not which one is good. It is this: which jobs of my eval setup does this tool run for me, and where is my data while it does?
A framework runs jobs. It does not know your task. The cases, the expected results and the rule for release stay yours, whichever tool runs them.
This is a reference page. It deepens the eval harness, the page about the program that runs the evals, and it is the only page of this site that makes statements about named tools. One note before any result appears: the agent here is a scripted stand-in, so these results show how the evaluation works, not how good any model is.
1. Six jobs, whoever writes the code#
Look at what examples/uw-evals/ does when you type one command. It loads the cases. It calls the agent, once for most cases and 20 times for UW-010. It writes down every step the agent took. It grades each trial. It writes a report. Then a second program reads the report and decides.
Those are six jobs. They exist in every eval setup, including one made of a spreadsheet and a person. The diagram names them.
Diagram: Six jobs behind every eval run, whoever writes the code
Read left to right. Something has to do each of these six jobs. In the example each job is one small file. An eval tool takes over some of the jobs and leaves the others with you. For each tool on this page, ask which boxes it runs and where your data sits while it does.
- 1 · Before the run
- Hold the cases: Keep each case with its expected result and labels, under a version, where the runner can load it.
- 2 · The run
- Run the trials: Call the agent once per trial in a clean state, as many times as the case asks for.
- Record the trace: Write down each step the agent took, so a grader can check the path and a person can read it.
- 3 · Grading
- Grade: Give each trial its scores: code checks, a rubric scored by a model, or a label from a person.
- 4 · After the run
- Keep and compare: Store the results of every run so that two runs can be put side by side later.
- Decide: Turn the results into a release decision with a stated rule and an exit code a pipeline can read.
In the example each job is a small file, and the files are not equally strong. The table says where each job lives and how far the example takes it.
| Job | In examples/uw-evals/ | How far the example takes it |
|---|---|---|
| Hold the cases | cases/UW-001.json to UW-010.json | Files in the repository, edited by engineers, versioned by git |
| Run the trials | run_evals.py, function run_trial | One after another, a fresh World for each trial, a seed |
| Record the trace | tracing.py | One JSON file per trial in a format of its own |
| Grade | graders.py, rubric.py, judge_sim.py | Nine code graders and one rubric grader with a scripted stand-in judge |
| Keep and compare | out/report.json | One file. The next run overwrites it |
| Decide | gate.py | A written rule, a sentence that explains the decision, exit code 0 or 1 |
Now the three requests have an address. The 400 cases belong to the first job. The labelling screen belongs to the fourth. The pass rate of four weeks ago belongs to the fifth, and the table shows why nobody could answer: that job is the thinnest one in the example.
Plain English. An eval framework is somebody else's code for some of these six jobs. Read each tool by asking which rows of this table it takes over.
2. How the tools were read#
Each tool below was read on its own documentation on 2026-11-17. The log of every claim, with the address of the page, is in the repository under docs/facts/frameworks.md. Three rules hold for the whole page.
- Only what the documentation says. Where the pages read do not state something, the text says "the pages read here do not say". That is a statement about what was read, not about the tool.
- No prices, no ranking. Tools change faster than a page. Check the linked documentation before you rely on a row.
- The example code for each tool is labelled "documented shape; not run here". It was put together from the calls the documentation shows and checked for syntax only. Nothing was installed. The only code on this page that was run is the plain Python of
examples/uw-evals/.
Every tool gets the same nine parts, so that you can read across:
- What it solves.
- What it is not.
- How it runs.
UW-004in this tool.- Cases, graders, traces, runs, pipeline: what the documentation covers.
- Where it runs and where the data goes.
- What it gives, where it stops, and what ties you to it.
- When it fits, and when plain Python does the same.
- Sources and the date read.
Parts 6, 7 and 9 are folded under each tool, so the first reading stays short.
Two words come up in almost every section. An LLM judge is a model used as a grader: it reads an answer and a rubric and returns a score. Its home on this site is Judges and calibration. And RAG, short for retrieval-augmented generation, means that a system looks up text first and gives it to the model with the question. The underwriting agent does that when it calls search_guidance and gets rule R-ROOF-03 back.
One more thing makes the tools hard to compare: they use different words for the same things. This table translates them into the words of this site.
| This site | DeepEval | Promptfoo | LangSmith | Phoenix | Ragas | OpenAI Evals |
|---|---|---|---|---|---|---|
| case | golden | test case | example | example | sample | item |
| grader | metric | assertion | evaluator | evaluator | metric | grader |
| one run of a suite | test run | eval | experiment | experiment | experiment | run |
| repeat a case | flag -r | flag --repeat | num_repetitions | not on the pages read | not on the pages read | not on the pages read |
Common confusion. On this site a metric is a number computed over many scores, such as a pass rate. In DeepEval and Ragas a "metric" is the thing that scores one answer, which this site calls a grader. The word is the same and the object is not.
DeepEval#
1 · What it solves. Your team already runs pytest. DeepEval lets an eval be a test file: you build a test case, attach one or more metrics with a threshold, and call assert_test. Its documentation calls it an open-source LLM evaluation framework and describes it as local-first.
2 · What it is not. It is not an observability tool. Its own introduction draws that line: observability tools show what happened, and DeepEval is about whether the behavior is good enough. It is also not a set of code checks. Its metrics page says almost all predefined metrics use a model as judge, so a score from one of them is a judge's opinion with a judge's errors.
3 · How it runs. In your process. The command deepeval test run collects test files the way pytest would. Each metric returns a score between 0 and 1 with a reason, and it passes when the score reaches the threshold, which defaults to 0.5. A decorator, @observe, records spans so that a metric can be attached to one step of the agent.
4 · UW-004 in this tool. The block shows one pytest function. A successful run would prove two things by code (the recommendation and the citation) and one thing by a judge (the reasons name both sources). Documented shape; not run here.
# test_uw_004.py · documented shape; not run here
import json
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from my_agent import underwrite # your agent's entry point: a submission in, the output record out
SUBMISSION = {"submission_id": "SUB-1004", "roof_age_years": 9, "wind_zone": "C"} # UW-004, shortened
EXPECTED = "Escalate under R-ROOF-03. The application says 9 years, the provider says 22 years."
def test_uw_004_names_both_sources():
output = underwrite(SUBMISSION)
assert output["recommendation"] == "escalate" # ordinary pytest: code can check this
assert "R-ROOF-03" in output["citations"]
names_both_sources = GEval(
name="Names both sources",
criteria="The actual output must state both roof ages and which source gave each, as the expected output does.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
threshold=0.5,
)
test_case = LLMTestCase(
input=json.dumps(SUBMISSION),
actual_output=json.dumps(output),
expected_output=EXPECTED,
)
assert_test(test_case, [names_both_sources])
The documented command is deepeval test run test_uw_004.py. The judge needs a model key; the quickstart uses OPENAI_API_KEY and says any other model can be configured.
5 · What the documentation covers.
| Job | What the pages read say |
|---|---|
| Cases | A dataset is a collection of goldens. A golden is a case that does not yet hold the agent's output. Saved locally as JSON, CSV or JSONL, or pushed to Confident AI |
| Graders | Metrics: custom ones such as G-Eval, and groups for agents (task completion, tool correctness, argument correctness and others), for RAG and for safety |
| Traces | @observe makes each call a span; a metric can sit on a span or on the whole trace. Tracing works locally with no account |
| Runs and repeats | deepeval test run with -n for parallel processes, -c to read the local cache, -r to repeat each test case |
| Pipeline | A page on running in a CI/CD pipeline, with a GitHub Actions workflow. Plain pytest works but the page advises against it |
8 · When it fits. When evals should sit next to the unit tests, run by the same pipeline, and the main need is a rubric scored by a judge. When every check is a code check, as nine of the ten graders in the example are, pytest alone does the same work.
DeepEval: where it runs, where it stops, sourcesParts 6, 7 and 9: data, limits, lock-in, links
6 · Where it runs and where the data goes. The quickstart says everything in it runs locally. The calls of the judge go to the model provider you configure. Logging in is optional and sends traces and test runs to Confident AI, the platform of the same team; the privacy page says data sent there is stored in the vendor's private cloud on AWS. The library collects basic usage telemetry by default and the page names DEEPEVAL_TELEMETRY_OPT_OUT=1 to turn it off. What drives cost is the number of judge calls: one or more per metric per test case, times the repeats. No prices are given here.
7 · What it gives, where it stops, what ties you to it. It gives many ready-made metrics, a reason with every score, and metrics on single spans. Its own metrics page states two limits: the default metrics are optimized for OpenAI's models, and a custom judge model may not follow the evaluation instructions reliably. The tracing page read here does not mention OpenTelemetry. This site adds one caution: a default threshold of 0.5 is a default, not a line anyone calibrated for your task. On lock-in, this is this site's reading of the documented formats: goldens export to JSON, CSV or JSONL, so the cases stay portable; the test files import the library's classes, so the metrics do not.
9 · Sources, read 2026-11-17. Introduction · Getting started · Metrics · Datasets · Tracing · CI/CD · Flags · Data privacy · Repository, which names the Apache 2.0 licence. No release number was visible on the pages read. Sample: the block above, compiled only.
Promptfoo#
1 · What it solves. You want the eval to be a file that a reviewer can read without reading code. In Promptfoo an eval is a configuration: prompts, providers (the thing under test), test cases and assertions. Its documentation describes it as a command-line tool and library for evaluating and red-teaming LLM apps. Red-teaming means attacking your own application to find its weak points; this page covers the evaluating half.
2 · What it is not. It is not a hosted store for production traces. And the server it offers for sharing results is, in the words of its own self-hosting page, built for individual or experimental use and not recommended for production use cases.
3 · How it runs. On your machine. The introduction says evals run locally and call the configured providers. A provider can be a Python file with a function call_api, which is how your own agent gets called. Each test can carry deterministic assertions, model-graded ones and, when tracing is enabled, assertions on the traced tool calls. The eval command returns exit code 100 when at least one test fails.
4 · UW-004 in this tool. Two blocks: the configuration, then the two small Python files it points to. A successful run would show one test with three assertions passed: valid JSON, the right recommendation and citation by code, and the wording by a judge. Documented shape; not run here.
# promptfooconfig.yaml · documented shape; not run here
prompts:
- '{{submission}}'
providers:
- id: 'file://uw_provider.py'
tests:
- description: 'UW-004: the application says 9 years, the provider says 22'
vars:
submission: '{"submission_id": "SUB-1004", "roof_age_years": 9, "wind_zone": "C"}'
assert:
- type: is-json
- type: python
value: file://uw_asserts.py:escalates_and_cites
- type: llm-rubric
value: States the roof age from the application and the roof age from the provider, and says which source gave which.
# uw_provider.py and uw_asserts.py, shown together · documented shape; not run here
import json
from my_agent import underwrite # your agent's entry point
def call_api(prompt, options, context): # uw_provider.py
output = underwrite(json.loads(prompt))
return {"output": json.dumps(output)}
def escalates_and_cites(output, context): # uw_asserts.py
record = json.loads(output)
ok = record["recommendation"] == "escalate" and "R-ROOF-03" in record["citations"]
return {"pass": ok, "score": 1.0 if ok else 0.0, "reason": "expected escalate citing R-ROOF-03"}
The documented command is npx promptfoo@latest eval -c promptfooconfig.yaml. The flag --repeat 20 runs each test 20 times.
5 · What the documentation covers.
| Job | What the pages read say |
|---|---|
| Cases | The tests list, inline or loaded from YAML, JSON, JSONL or CSV files |
| Graders | Assertions: deterministic (equals, regex, is-json, your own Python or JavaScript, latency, cost), model-graded (llm-rubric, factuality, context faithfulness and others), with weights and thresholds |
| Traces | Receives OpenTelemetry traces through a built-in receiver and stores them in a local SQLite file. Assertions such as trajectory:tool-used and trajectory:tool-sequence read them |
| Runs and repeats | --repeat, a cache, a concurrency limit, filters to rerun the failed tests |
| Pipeline | Exit code 100 on a failed test or a pass rate under PROMPTFOO_PASS_RATE_THRESHOLD. Output as JSON, HTML or JUnit XML, a test-report format that pipelines read. Guides for several pipeline products |
8 · When it fits. When cases and checks should be reviewable as data, when you compare prompts or models side by side, and when the pipeline needs an exit code. When your checks need the whole trial record (the output, the trace and the stored rows together, as required_process does for UW-008), a plain function over your own record is shorter than fitting the record into assertions.
Promptfoo: where it runs, where it stops, sourcesParts 6, 7 and 9: data, limits, lock-in, links
6 · Where it runs and where the data goes. Results are stored locally; the self-hosting page names ~/.promptfoo/ as the default data directory and a SQLite database. Results leave the machine when you run promptfoo share or pass --share. The introduction notes that red-team generation may use hosted services. Usage telemetry is on by default; the telemetry page says it does not include prompts, model outputs, test cases or keys, and names PROMPTFOO_DISABLE_TELEMETRY=1. The repository names the MIT licence and states that Promptfoo is now part of OpenAI. Cost is driven by the calls to the providers and to the judge model. No prices are given here.
7 · What it gives, where it stops, what ties you to it. It gives one readable file, many assertion types and a direct fit to a pipeline. Its documentation states limits of its own: the self-hosted server has no built-in authentication, no role-based access control and no horizontal scaling; the tracing page lists cases in which redactAttributes does not remove a value, so it should not be the only protection for sensitive data. On lock-in, this site's reading: the configuration is YAML and the cases can live in CSV or JSONL, so both move easily. Checks written as Python functions move with a change of signature. Model-graded assertions have to be validated again wherever they go.
9 · Sources, read 2026-11-17. Intro · Getting started · Configuration · Assertions · Python assertions · Python provider · Tracing · Command line · CI/CD · Self-hosting · Telemetry · Repository. No release number was visible on the pages read. Sample: the two blocks above; the YAML parsed and the Python compiled, nothing more.
LangSmith#
1 · What it solves. The fifth job: keep and compare. LangSmith is a platform that stores traces, datasets and the result of every eval run, and shows two runs side by side. Its documentation calls the stored result of evaluating one application version on a dataset an experiment.
2 · What it is not. It is not a library that works alone. The evaluation quickstart lists a LangSmith account and an API key as prerequisites, and the results live on the platform. It is also not limited to one agent framework: the tracing pages describe a decorator for any Python function and an endpoint for OpenTelemetry.
3 · How it runs. Your code runs the agent and the graders on your machine; the client sends inputs, outputs, scores and traces to the platform. An evaluator, which is a grader in this site's words, can be code, an LLM judge, a person working through an annotation queue, or a pairwise comparison. The same platform runs online evaluators on production traces, with a filter and a sampling rate.
4 · UW-004 in this tool. The block creates a dataset with one example, defines a target function and one code grader, and runs an experiment with 20 repetitions. A successful run would leave one experiment on the platform with 20 scored runs of UW-004. Documented shape; not run here.
# uw_langsmith.py · documented shape; not run here
from langsmith import Client
from my_agent import underwrite # your agent's entry point
client = Client() # needs LANGSMITH_API_KEY
dataset = client.create_dataset(dataset_name="uw-golden", description="Synthetic underwriting cases")
client.create_examples(
dataset_id=dataset.id,
examples=[
{
"inputs": {"submission": {"submission_id": "SUB-1004", "roof_age_years": 9, "wind_zone": "C"}},
"outputs": {"recommendation": "escalate", "citations": ["R-ROOF-03"]},
}
],
)
def target(inputs: dict) -> dict:
return underwrite(inputs["submission"])
def recommendation_match(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
return outputs["recommendation"] == reference_outputs["recommendation"]
results = client.evaluate(
target,
data="uw-golden",
evaluators=[recommendation_match],
experiment_prefix="uw-004-careful",
num_repetitions=20,
max_concurrency=4,
)
5 · What the documentation covers.
| Job | What the pages read say |
|---|---|
| Cases | Datasets of examples: inputs, optional reference outputs, metadata. Created by code or in the interface |
| Graders | Evaluators: code, LLM judge with or without a reference, human review in annotation queues, pairwise. For the path of an agent, the open-source agentevals package with strict, unordered, subset and superset matching |
| Traces | Runs grouped into traces and threads. Each trace is kept for the retention period of its tier. An endpoint accepts OpenTelemetry and maps gen_ai attributes |
| Runs and repeats | evaluate with max_concurrency and num_repetitions; the interface shows the average per score and the individual runs |
| Pipeline | A pytest plugin (@pytest.mark.langsmith) and Vitest or Jest. LANGSMITH_TEST_TRACKING=false runs the tests without sending results |
8 · When it fits. When the open questions are the ones from the opening: who labels, where is last month's run, what did production do. When one team reads one report per release and keeps it as a build artifact, a file per run answers the same question.
LangSmith: where it runs, where it stops, sourcesParts 6, 7 and 9: data, limits, lock-in, links
6 · Where it runs and where the data goes. The documentation lists three ways to run it. Cloud: the vendor runs the infrastructure and the data is in the vendor's cloud. Hybrid: traces and evaluations stay on the cloud service while agent workloads run in your infrastructure. Self-hosted: you run it in your own Kubernetes cluster and the data stays in your environment; the page says self-hosting is an add-on to the Enterprise plan and needs a licence key. For an insurer this is the first question to settle, because a trace of a real submission holds the applicant's data. The pages read here state no source licence for the platform. Cost depends on the plan, on trace volume and retention, and on judge calls. No prices are given here.
7 · What it gives, where it stops, what ties you to it. It gives history, comparison, human review and production traces in one place. The RAG tutorial read here writes its four graders (correctness, relevance, groundedness, retrieval relevance) as the user's own judge functions, so plan to write and validate yours. On lock-in, this site's reading: graders are plain functions with a documented signature and move easily. Datasets, experiments, annotation queues and their history live on the platform, and moving them is a project of its own.
9 · Sources, read 2026-11-17. Evaluation concepts · Evaluation quickstart · Evaluate an application · Repetitions · pytest · Observability concepts · Online evaluations · Trajectory evals · OpenTelemetry · Hosting options · Self-hosted · RAG tutorial. No release number was visible on the pages read. Sample: the block above, compiled only.
Arize Phoenix#
1 · What it solves. The third job first: record the trace, and then build the rest on it. Phoenix is a server that receives traces and adds datasets, experiments and graders. Its documentation says it is built on top of OpenTelemetry and powered by OpenInference instrumentation, the libraries that create the spans.
2 · What it is not. It is not Arize AX. The documentation describes AX as a separate managed enterprise platform of the same company, built on the same open standards. And the pages read here do not describe Phoenix as a test runner with an exit code for a pipeline.
3 · How it runs. As a server that you start. One command, phoenix serve, starts it locally with the interface on port 6006, and the documentation says a local server needs no key. Your agent sends spans to it over the OpenTelemetry protocol. An experiment runs a task function over a dataset and scores the outputs with evaluators, which are graders in this site's words.
4 · UW-004 in this tool. The block creates a dataset with one example, defines a judge with two labels, and runs one experiment. A successful run would leave an experiment in the local Phoenix with the judge's label for UW-004. Documented shape; not run here.
# uw_phoenix.py · documented shape; not run here
import json
from phoenix.client import Client
from phoenix.evals import LLM, bind_evaluator, create_classifier
from my_agent import underwrite # your agent's entry point
phoenix = Client() # reads PHOENIX_ENDPOINT, default http://localhost:6006
dataset = phoenix.datasets.create_dataset(
name="uw-golden",
examples=[
{
"input": {"submission": {"submission_id": "SUB-1004", "roof_age_years": 9, "wind_zone": "C"}},
"output": {"expected_reasons": "Escalate under R-ROOF-03. The application says 9 years, the provider says 22."},
}
],
)
judge = create_classifier(
name="names_both_sources",
llm=LLM(provider="openai", model="gpt-5-mini"), # the judge model named in the documentation's example
prompt_template="""Label true if the answer recommends escalation and states both roof ages with their sources.
Label false if it leaves out a source, an age or the escalation.
Reference: {{reference}}
Answer: {{output}}""",
choices={"true": 1.0, "false": 0.0},
)
names_both_sources = bind_evaluator(evaluator=judge, input_mapping={"reference": "reference.expected_reasons"})
def task(input):
return json.dumps(underwrite(input["submission"]))
experiment = phoenix.experiments.run_experiment(
dataset=dataset, task=task, evaluators=[names_both_sources], experiment_name="uw-004-careful"
)
5 · What the documentation covers.
| Job | What the pages read say |
|---|---|
| Cases | Datasets: collections of examples with inputs and optional reference outputs, collected from production, staging or by hand |
| Graders | Evaluators by code (exact match, regex, your own function) and by LLM judge. Pre-built ones for RAG and for agents that call tools. Each returns a score object with a kind of llm, code or human |
| Traces | Its core. Accepts the OpenTelemetry protocol; projects, sessions and annotations on traces |
| Runs and repeats | run_experiment over a dataset. Repeating a case: the pages read here do not say |
| Pipeline | The pages read here do not say |
8 · When it fits. When you want to see traces first and keep them on your own machines, and when OpenTelemetry is already in the company. When nobody will open an interface and the release decision is a script, the trace files of the example and a gate cover the same ground.
Phoenix: where it runs, where it stops, sourcesParts 6, 7 and 9: data, limits, lock-in, links
6 · Where it runs and where the data goes. The documentation describes a local process, a container image and self-hosting on Kubernetes and several clouds, backed by PostgreSQL. The self-hosting page says the data stays entirely within your infrastructure and that the deployment can be cut off from the internet, with no feature limitations. The repository names the Elastic License 2.0, and its README calls Phoenix an open-source AI observability platform. That licence is not the Apache or the MIT licence, so read its terms before you build a service on it. Judge calls go to the model provider you configure. No prices are given here.
7 · What it gives, where it stops, what ties you to it. It gives a trace view built on an open protocol, with graders and experiments next to it. The pages read here are silent on two things this site's example relies on: repeating a case inside an experiment, and returning a result to a pipeline. On lock-in, this site's reading: traces arrive over OpenTelemetry, so the instrumentation in your agent is not tied to Phoenix. Datasets, experiments and annotations live in its database.
9 · Sources, read 2026-11-17. Overview · Start Phoenix · Tracing quickstart · Tracing · Evaluation · Evaluators · Datasets and experiments · Experiments quickstart · Self-hosting · Repository. No release number was visible on the pages read. Sample: the block above, compiled only.
Ragas#
1 · What it solves. The fourth job, and within it one hard part: graders for answers that depend on retrieved text and on tool calls. Ragas is a Python library of metrics, which are graders in this site's words. Its documentation describes it as a library for moving from informal checks to systematic evaluation loops.
2 · What it is not. It is not a trace store, a dashboard or a release gate; the pages read here describe none of those. It is a library you call from your own runner or from one of the other tools.
3 · How it runs. In your process, as function calls. Some metrics need a judge model: you create one with llm_factory and pass it in. Others are computed without a model. Tool call accuracy, for instance, compares the tool calls in a conversation with a list of expected tool calls. A small @experiment decorator runs a function over a dataset and saves the results as CSV files.
4 · UW-004 in this tool. The block defines one custom metric with two allowed values and scores one answer. A successful run would print pass and the judge's reason. Documented shape; not run here.
# uw_ragas.py · documented shape; not run here
import asyncio
import json
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics import DiscreteMetric
from my_agent import underwrite # your agent's entry point
evaluator_llm = llm_factory("gpt-4o", client=AsyncOpenAI()) # the judge model named in the documentation's example
names_both_sources = DiscreteMetric(
name="names_both_sources",
allowed_values=["pass", "fail"],
prompt="""The answer must recommend escalation and state both roof ages with their sources:
9 years from the application, 22 years from the provider.
Response: {response}
Answer with only 'pass' or 'fail'.""",
)
async def main():
output = underwrite({"submission_id": "SUB-1004", "roof_age_years": 9, "wind_zone": "C"})
score = await names_both_sources.ascore(llm=evaluator_llm, response=json.dumps(output))
print(score.value, score.reason)
if __name__ == "__main__":
asyncio.run(main())
5 · What the documentation covers.
| Job | What the pages read say |
|---|---|
| Cases | A Dataset with a local CSV or JSONL backend. Tools to generate synthetic test data for RAG and for agents |
| Graders | Its core. Groups for RAG (context precision, context recall, faithfulness, response relevancy and others), for agents (topic adherence, tool call accuracy, tool call F1, agent goal accuracy), for text comparison, and rubric-based scoring |
| Traces | The pages read here do not say |
| Runs and repeats | @experiment over a dataset; results saved as CSV files with a timestamp. Repeating a case: the pages read here do not say |
| Pipeline | The pages read here do not say |
8 · When it fits. When the weak spot is the retrieval or the tool calls and you want named, documented graders for them without writing the judge prompts from nothing. It joins a runner; it does not replace one. For UW-008, where the question is whether search_guidance was called at all, the 14 lines of required_process in graders.py, read from the trace, are enough.
Ragas: where it runs, where it stops, sourcesParts 6, 7 and 9: data, limits, lock-in, links
6 · Where it runs and where the data goes. It runs as a library on your machine and saves datasets and results to local files. Judge calls go to the model provider you configure; the quickstart lists several providers and a local model server. The repository names the Apache 2.0 licence, says the project collects minimal, anonymized usage data, and names RAGAS_DO_NOT_TRACK to turn that off. Cost is driven by judge calls. No prices are given here.
7 · What it gives, where it stops, what ties you to it. It gives graders with stated definitions, which makes a score easier to explain. Read the definition before you use one: tool call accuracy, as documented, multiplies the accuracy of the arguments by whether the order matched, and strict order is the default. An agent that reaches the same result by another valid order then scores 0 unless you turn strict order off. The source of the older evaluate() function carries a deprecation warning that points to @experiment, so examples written for the older call may age. On lock-in, this site's reading: low. It is a library, the data is in CSV, and a grader can be replaced one at a time.
9 · Sources, read 2026-11-17. Overview · Quickstart · Available metrics · Agent metrics · Experiments · evaluate() reference · Test data generation · Repository. No release number was visible on the pages read. Sample: the block above, compiled only.
OpenAI eval tooling#
Read this section with its date in mind. On 2026-11-17 the Evals guide in the OpenAI platform documentation opened with a notice: OpenAI is deprecating the Evals platform, Evals become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026. The deprecations page lists the entry "Evals and graders", affecting the Evals dashboard, the API and graders for eval workflows, and names the replacement: migrate to Promptfoo.
1 · What it solves. It let a team that already used the OpenAI platform define an eval there: a schema for the test items, graders, and runs whose results appear in the dashboard. A second feature, trace grading, assigns scores or labels to the trace of an agent.
2 · What it is not. In the guide read here it is not a runner for your own agent code. The documented run sends each item through a model with a prompt template and grades the text that comes back. And after the notice above it is not a place to put new cases.
3 · How it runs. On the vendor's platform. An eval has a data source configuration, which is the shape of each item, and testing criteria, which are the graders. A run names a model, a prompt template and an uploaded JSONL file. Runs are asynchronous, and the result carries counts of passed and failed items and a link to a report.
4 · UW-004 in this tool. The block shows the shape of an eval with one string check on the recommendation. It is shown for readers who have such evals and must move them. Documented shape; not run here, and on the date read the documentation says Evals are read-only for existing users.
# documented shape; not run here. The documentation read on 2026-11-17 marks this platform read-only.
from openai import OpenAI
client = OpenAI()
eval_obj = client.evals.create(
name="UW recommendation",
data_source_config={
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"submission": {"type": "string"},
"expected_recommendation": {"type": "string"},
},
"required": ["submission", "expected_recommendation"],
},
"include_sample_schema": True,
},
testing_criteria=[
{
"type": "string_check",
"name": "Recommendation matches the expected one",
"input": "{{ sample.output_text }}",
"operation": "eq",
"reference": "{{ item.expected_recommendation }}",
}
],
)
5 · What the documentation covers.
| Job | What the pages read say |
|---|---|
| Cases | Items in an uploaded JSONL file that match the item schema. A Datasets feature in the dashboard, which carries the same deprecation notice |
| Graders | String check, text similarity, a model that returns a score, and a Python function with no network access and a time limit. The graders page says graders are being deprecated with the workflows they support |
| Traces | The Agents SDK traces by default and shows traces in a dashboard. Trace grading scores them there |
| Runs and repeats | Runs on the platform, with webhooks for finished runs. Repeating an item: the pages read here do not say |
| Pipeline | An API exists. A documented pipeline recipe: the pages read here do not say |
8 · When it fits. For new work, the documentation itself points elsewhere. If you have evals there, the cookbook guide on moving to Promptfoo describes a manual rebuild of prompts, provider, test cases and assertions, and warns that similarity-based scores may not be numerically identical across systems.
OpenAI eval tooling: what the notice covers, sourcesParts 6, 7 and 9: data, open questions, the lesson, links
6 · Where it runs and where the data goes. Everything in this section runs on the vendor's platform, and the items, outputs and traces are stored there. The Agents SDK page says tracing is on by default, that the default export can be replaced with your own trace processors, that the capture of sensitive data in spans can be turned off, and that tracing is unavailable for organizations under a Zero Data Retention policy. No prices are given here.
7 · What it gives, where it stops, what ties you to it. What is not clear from the pages read: the deprecation entry names the Evals dashboard, the API and graders for eval workflows. The trace grading page carries no notice, and the page on agent evals still recommends starting with traces. The documentation read here does not say whether trace grading continues after November 30, 2026. Ask the vendor before you plan on it. The Promptfoo command-line page adds one fact for people who are moving: promptfoo import reads an exported file of results as a historical record, and does not rebuild a runnable configuration from it.
This is the lock-in question of every other section, answered by an event. A team whose cases, expected results and graders lived only on that platform has work to do before the shutdown date. A team that kept them as files in its own repository, and used the platform to run them, rewrites an adapter.
9 · Sources, read 2026-11-17. The addresses under platform.openai.com/docs/guides/ answered with a redirect to developers.openai.com. Evals guide · Deprecations · Graders · Trace grading · Agent evals · Datasets · Moving from OpenAI Evals to Promptfoo · Agents SDK tracing. Sample: the block above, compiled only.
OpenTelemetry GenAI conventions#
1 · What it solves. A problem between tools, not inside one. Phoenix, LangSmith and Promptfoo all accept traces over the OpenTelemetry protocol. For a trace to mean the same thing in each of them, the steps of an agent need agreed names. The GenAI semantic conventions are that agreement: names for the spans and attributes of model calls, tool calls, retrievals and agent steps.
2 · What it is not. It is not an eval tool, and it is not a place where traces are stored. The OpenTelemetry documentation says the project is not an observability backend. The conventions grade nothing and run nothing. They are a document.
3 · How it runs. It does not run. Your agent, or a library that instruments it, creates spans with these names and exports them to a backend of your choice. On the date read, the conventions were marked with the status Development and had moved out of the main OpenTelemetry site into a repository of their own. Development means the names can still change.
4 · UW-004 in this convention. The table maps one real span from the example, the roof look-up of UW-004, and one grader result onto the documented names. The left column is from out/traces/UW-004-01.json. The mapping is this site's; documented shape; not run here.
| In the example's trace | Under the conventions, as documented |
|---|---|
name: get_roof_assessment, kind: tool | Span name execute_tool get_roof_assessment; gen_ai.operation.name = execute_tool; gen_ai.tool.name = get_roof_assessment (required) |
input: {"submission_id": "SUB-1004"} | gen_ai.tool.call.arguments, opt-in, marked as possibly sensitive |
output: {"roof_age_years": 22, ...} | gen_ai.tool.call.result, opt-in, marked as possibly sensitive |
status: ok (in UW-006 it is error) | error.type, set only when the operation ended in an error |
name: search_guidance, kind: retrieval | gen_ai.operation.name = retrieval; the query text and the documents are opt-in |
Grader result required_process, passed | Event gen_ai.evaluation.result with gen_ai.evaluation.name, a score label, a score value and an explanation, attached to the span it evaluated |
The last row matters for this site. The conventions define an event for the result of an evaluation, so a score can travel in the same trace as the step it judges.
5 · What the documentation covers.
| Job | What the documents read say |
|---|---|
| Cases | Nothing; out of scope |
| Graders | Nothing runs. One event, gen_ai.evaluation.result, names how a result is recorded |
| Traces | Its subject: spans for inference, embeddings, retrieval, tool execution, agent creation and invocation, planning; metrics; provider-specific pages |
| Runs and repeats | Nothing; out of scope |
| Pipeline | Nothing; out of scope |
8 · When it fits. When more than one tool will read your traces, or when you may change the tool later. Then naming your spans this way from the start is cheap. When the only reader of a trace is your own grader, the format of tracing.py is enough, and the mapping above can wait. The home of traces on this site is Agent traces.
OpenTelemetry GenAI conventions: content capture, gaps, sourcesParts 6, 7 and 9: sensitive data, what is not defined, links
6 · Where it runs and where the data goes. Wherever you export. The conventions take a position on content: model instructions, user messages and model outputs are called sensitive and often large, and instrumentations should not capture them by default but offer an opt-in. The documented switch is OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT, with NO_CONTENT as the default. For production the document recommends storing content elsewhere and recording a reference on the span. For the insurer this is the right default: a submission holds an applicant's data, and a trace without content still shows which tools ran, in which order, and which failed.
7 · What it gives, where it stops, what ties you to it. It gives a shared vocabulary, which is what makes the trace part of your setup movable. Two limits follow from the documents read. The status is Development, so pin the version you instrument against and expect renames. And the documents read here define no span type for a deterministic policy check or for a human approval. In the example those are check_eligibility and underwriter_review, and they stay custom spans with names of your own. A grader that depends on them, such as required_process, depends on your names, not on a standard.
9 · Sources, read 2026-11-17. What is OpenTelemetry · The moved notice · Repository of the GenAI conventions and, in its folder docs/gen-ai/, the files README.md, client-inference.md, execute-tool.md, retrieval.md, gen-ai-agent-spans.md, gen-ai-spans.md and gen-ai-events.md. No release was visible on the repository page. Sample: the mapping table above; no code.
The seven side by side#
You have now read each tool against the same six jobs. The table puts the answers in one place. It contains nothing the sections did not prepare, and it carries the same date: read on 2026-11-17. Use the buttons to filter the rows.
A cell says yes when a page read describes the feature, no when the tool is out of that business by its own description, and unstated when the pages read here do not say. unstated is not no.
| Tool | Runs graders | Records traces | Holds cases | Repeats a case | Pipeline entry | Ready RAG graders | Account needed | Where results live | Licence, as read |
|---|---|---|---|---|---|---|---|---|---|
| DeepEval | yes | yes | yes | yes | yes | yes | no | Local files; Confident AI if you log in | Apache 2.0 |
| Promptfoo | yes | yes | yes | yes | yes | yes | no | Local SQLite; a server if you share | MIT |
| LangSmith | yes | yes | yes | yes | yes | unstated | yes | The platform: vendor cloud, hybrid, or your cluster on the Enterprise plan | Not stated on the pages read |
| Phoenix | yes | yes | yes | unstated | unstated | yes | no | Your Phoenix server | Elastic License 2.0 |
| Ragas | yes | unstated | yes | unstated | unstated | yes | unstated | Local CSV files | Apache 2.0 |
| OpenAI eval tooling | yes | yes | yes | unstated | unstated | unstated | yes | The vendor's platform; shutdown scheduled for 2026-11-30 | Hosted service |
| OpenTelemetry GenAI | no | spec | no | no | no | no | n/a | Wherever you export | Not read |
Four notes keep the table honest. "Account needed" is about the tool's own vendor; a judge model needs a key from a model provider in every row that uses one. For LangSmith, "Ready RAG graders" is unstated because the tutorial read writes its own judge functions. "Pipeline entry" means a documented command or plugin for a pipeline, not that a pipeline is impossible without one. And spec means the row defines the format of a trace and records none.
Nothing in this table says which tool to choose. Anthropic's article on agent evals, the source of this site's vocabulary, makes the point that settles it: a framework is worth as much as the eval tasks you run through it. Its advice is to pick one that fits how you work and to spend the effort on the tasks.
The part that stays yours#
The OpenAI section showed what the question of lock-in looks like when it stops being a thought. The defence is the same for every tool: keep three things in your own repository, in plain formats, and let a thin adapter hand them to whichever tool runs them.
Diagram: What stays yours when a tool comes in
Read left to right. The left lane is the part that holds your knowledge of the task. Keep it in your repository in plain formats. The middle lane is one small file per tool. The right lane is what the tool does with what the adapter hands it. Changing tools then means rewriting the middle lane only.
- 1 · Yours, in the repository
- Cases as data: Inputs, expected results and labels in a format any tool can read.
- One way to call the agent: A single function that runs one trial and returns what the agent said and did.
- Graders as functions: Each code check is a plain function. The rubric is a text file a judge prompt is built from.
- 2 · A thin adapter
- Adapter: Translates your cases, your call and your graders into the shapes one tool expects.
- 3 · The tool
- Runs and repeats: Calls the adapter for every case, in parallel and as often as asked.
- Stores and shows: Keeps each run, shows the traces and puts two runs side by side.
The three things are the cases as data, one function that runs a trial, and the graders as functions. The example already has all three, which is why each tool example above needed so little: a dictionary for the case, one call to the agent, one comparison.
The script below shows how small the neutral form is. It was written for this page and run against examples/uw-evals/ on 2026-11-17. It turns the nine regression cases into rows with an input, an expected result, the output, the tool calls and the scores, and prints the row of UW-004. A successful run proves that everything a tool would need can be produced from the example with one loop.
"""Written for this page, not part of examples/uw-evals. Run from examples/uw-evals."""
import json
from run_evals import load_cases, run_trial
rows = []
for case in load_cases("regression"):
trial = run_trial(case, variant="careful", seed=7, trial=1)
rows.append({
"id": case["id"],
"input": case["input"]["submission"],
"expected": case["expected"],
"output": trial["output"],
"tool_calls": [s["name"] for s in trial["trace"]["spans"] if s["kind"] in ("tool", "retrieval", "policy_check")],
"scores": {g["name"]: g["passed"] for g in trial["graders"]},
})
row = next(r for r in rows if r["id"] == "UW-004")
print(len(rows), "rows")
print("input keys:", sorted(row["input"]))
print("expected:", row["expected"]["recommendation"], row["expected"]["citations"])
print("output:", row["output"]["recommendation"], row["output"]["citations"])
print("tool_calls:", row["tool_calls"])
print("scores:", row["scores"])
9 rows
input keys: ['area_sq_ft', 'building_value_usd', 'construction', 'occupancy', 'prior_losses_5y', 'roof_age_years', 'sprinklered', 'submission_id', 'wind_zone', 'year_built']
expected: escalate ['R-ROOF-03']
output: escalate ['R-ROOF-03']
tool_calls: ['get_roof_assessment', 'search_guidance', 'find_similar_decisions', 'check_eligibility', 'record_recommendation']
scores: {'recommendation_match': True, 'citations': True, 'missing_fields': True, 'required_process': True, 'human_required': True, 'rubric': True}
Each row is a case in LangSmith's or Phoenix's sense, a test in Promptfoo's, a golden plus an output in DeepEval's. The list of tool calls is what a tool-call grader compares with a reference. The scores are what an evaluation event would carry. An adapter is the loop that writes these rows into one tool's shape.
Production pitfall. A judge prompt typed into a vendor's screen is a grader that exists in one place. Keep the rubric as a text file in the repository and let the adapter send it. Then a change to the rubric is reviewed like any other change, and the grader survives a change of tool.
When a plain Python harness is enough#
Go back to the three requests and take them one at a time, this time with the six jobs in hand.
The pass rate of four weeks ago needs no framework. The runner already has an option, --json FILE, that writes the report to a file of your choice. A pipeline that names the file after the date and keeps it as a build artifact answers the question for every week from now on. Here is the run that would be stored, and the gate's reading of it:
cd examples/uw-evals
python3 run_evals.py --suite all
python3 gate.py
Eval run · suite all · variant careful · seed 7 · agent: scripted stand-in (no model called)
case trials pass rate recommendation failed graders
UW-001 1 1 1.00 decline -
UW-002 1 1 1.00 approve -
UW-003 1 1 1.00 request_info -
UW-004 1 1 1.00 escalate -
UW-005 1 1 1.00 decline -
UW-006 1 1 1.00 request_info -
UW-007 1 1 1.00 decline -
UW-008 1 1 1.00 decline -
UW-009 1 1 1.00 approve -
UW-010 20 16 0.80 escalate x16, approve x4 recommendation_match (4), citations (4)
trials 29 · passed 25 · pass rate 0.86 · 95% interval 0.69 to 0.95 (Wilson)
critical cases: 0 failed trials, 0 with a hard-fail · measured cases: 16 of 20 trials passed
wrote out/report.json and 29 trace file(s) in out/traces/
Gate · suite all · variant careful · threshold 0.70
BLOCK: 16 of 20 trials passed (0.80); the 95% interval is 0.58 to 0.92. The lower bound 0.58 is below the threshold 0.70. The observed rate looks high enough, but the sample is too small to claim it.
That is about 600 lines of standard-library Python doing all six jobs: cases, trials, traces, ten graders, a report, and a decision that explains itself. With the stand-in agent, its tools and the tests, the folder holds 1,112 lines. The numbers are synthetic and come from a scripted stand-in. Why the gate blocks a rate of 0.80 is the subject of CI/CD quality gates.
The 400 cases need a decision about who edits them. If engineers do, files in the repository scale to hundreds of cases, and review of a new case is a code review. If underwriters do, they need a screen, and that is the first real argument for a tool that holds datasets.
The labelling screen is the request plain Python does not meet. Showing two people an answer, its trace and a rubric, collecting two labels and recording who disagreed is an application of its own. The documentation of LangSmith describes annotation queues for this, and Phoenix describes annotations on traces. Building it yourself is seldom the cheaper path.
So the rule of thumb has two sides.
| Plain Python is enough while | A tool earns its place when |
|---|---|
| Engineers write and review the cases | People who do not read JSON must add or label cases |
| Most graders are code, and the few judge prompts fit in one file | Many judge-scored graders across several models need caching, retries and rate limits |
| One team reads one report per release | Several teams compare runs over months |
| Traces are read by your own graders | Traces from production must be stored, searched and sampled |
| The release decision is a script with an exit code | The same results must also feed dashboards and alerts |
When you do bring a tool in, bring it in for a job. Name the row of the table in section 1 that it takes over, write the adapter, and leave the other rows where they are. The gate is the last job to hand over, because its rule is the shortest code you have and the one you must be able to explain to an underwriter.
Four checks before a tool's score may block a releaseParity, judge agreement, repeats, and where the data sits
A grader that moves into a tool is a new grader until shown otherwise. Four checks, in the order in which they tend to fail.
Parity on the cases you already have. Run the old grader and the new one on the same stored trials and compare result by result. For code graders the results must be identical. The example makes this cheap, because every trial is a file under out/traces/ and the scores are in the report.
Agreement of a ready-made judge with your people. A ready-made grader named "faithfulness" or "tool correctness" is a judge prompt written by someone who has not seen your policy. Before its score counts, compare it with human labels on your own answers, as Judges and calibration describes. In the example the scripted stand-in judge agrees with four of six human labels and gives two false passes; six answers are an illustration, not a measurement.
Repeats that mean trials. A repeat flag reruns a case. Check that each repeat starts from a clean state, as a fresh World does in the example, and that a cache is not handing back the first answer. Promptfoo's tracing page gives one concrete instance: a cache hit emits no spans for the model turns, so checks that count them need the cache off. A pass rate computed over cached repeats is one trial counted many times. The formulas pass@k = 1 − (1 − p)^k and pass^k = p^k hold only if trials are independent and each succeeds with the same probability p. A cache breaks the first condition completely.
Where the content of a trace sits. A trace of a real submission contains an applicant's data. For each tool, write down which machine stores inputs and outputs, who can read them, and how long they are kept. The sections above give the documented starting points: local files, your own server, a vendor's cloud with a retention period, or a convention whose default is to capture no content.
A score is an observed result. Confidence is a claim about uncertainty. Reliability needs repeated evidence. No tool on this page changes that. A dashboard with a number on it is still one of the three, and it is worth asking which.
In one breath#
Every eval setup does six jobs: hold the cases, run the trials, record the trace, grade, keep the results, decide. A framework is someone else's code for some of them, so I read each tool by asking which jobs it runs and where my data sits while it does. DeepEval and Promptfoo run and grade from my machine, LangSmith and Phoenix store and compare, Ragas supplies graders, OpenTelemetry names the spans, and the OpenAI Evals platform showed, with a shutdown date in its own documentation, why the cases and graders belong in my repository behind a thin adapter. Until someone who does not read JSON has to label, or several teams have to compare runs over months, a few hundred lines of plain Python do all six.
What comes next#
You can now look at an eval tool and say which of the six jobs it takes over, what its documentation leaves open, and what you would keep in your own repository whatever you choose. The same idea holds for a coding agent: the tasks, the tests that define success and the rule for merging stay in the repository, and the tool that runs them is replaceable.
What this page did not do is show the whole example working as one thing. It used UW-004 seven times and the other nine cases hardly at all. The insurance lab puts everything on one page: the ten cases, the policy fixture, the traces, every grader, the results and the release decision, with the code that produced them.