Evaluators

An evaluator scores output automatically by handing a model a rubric and asking it to judge. Its scores go to Scores like any other, tagged with their source, which is how you tell an evaluator's judgement apart from a reviewer's on the same trace.

The page has two specific tabs. The Evaluator Library lists every evaluator available to use, whether it came with the platform or your team created it. Running Evaluators, on the other hand, are the ones you have put to work, each pointed at a particular target and each either active or inactive. The same evaluator can back several running evaluators with different targets, and an evaluator sitting in the library produces nothing until you put them to work.

Evaluators come from three sources, shown in the Maintainer column: InteractiveAI for the general quality dimensions, Ragas for a set drawn from its open-source evaluation framework, and User for anything you build yourself. Each entry shows how often it has been used and which version is current, and Use starts a setup from it.

Setting one up

Set Up Evaluator leads somewhere different depending on which half you are in. From the library it opens the form for writing a new custom evaluator. From Running Evaluators it opens the flow that puts one to work, where you pick a managed or custom evaluator, or create one on the spot.

Once you have picked one, decide what it watches.

The generated score name is what these scores will be called in Scores. It starts as the evaluator's own name, which is worth changing when you run the same evaluator more than once, since otherwise two configurations produce scores you cannot tell apart.

Evaluator runs on and Target data together settle what gets judged. An evaluator can watch new traces as they arrive, work through traces you already have, or both at once, against either live tracing data or the output of experiment runs. A target filter narrows it further, and Preview sample matched traces shows a sample from the last 24 hours of what that filter actually catches, which is the check worth doing before you execute.

Sampling is the control that matters most. It sets the percentage of matching items that actually get evaluated, so you might judge every item in a dataset run but only a fraction of production traffic. Every evaluation is a model call, which makes this the main cost control on the page.

Delay sets how long to wait after a trace or dataset run arrives before evaluating it, so that everything belonging to that item has landed first.

Variable mapping is where the rubric meets your data. The evaluation prompt contains variables such as {{query}} and {{generation}}, and each one is bound to a source: which object it comes from, the trace or the dataset item, which part of that object, its input, output, or expected output, and optionally a JsonPath to reach a specific field inside the payload. Bind one to the wrong place and the judge scores the wrong text, which is the most common reason an evaluator returns something that makes no sense.

Custom evaluators

A custom evaluator takes a name, a model, and three prompts that do different jobs. The evaluation prompt is the rubric itself. The score range prompt tells the judge what scale to use. The score reasoning prompt tells it how to explain the score it gives. Leave the model on the project default or override it for this evaluator alone.

The default evaluation model is set once for the project and shown at the top of the page, so changing it there changes what every evaluator uses unless it overrides.

Execution logs

Logs opens one row per evaluation: when it ran, the score and comment it produced, any error, the trace it judged, and whether it completed. A completed row with no score value is not a failure. It means the judge had nothing to work with, an empty generation or an empty input, and the comment says which. When an evaluator returns something you did not expect, this is where you find out whether it failed, had nothing to judge, or simply disagreed with you.