Datasets

A dataset is a fixed set of test cases. Each item holds an input, the expected output, and metadata, so the same inputs can be replayed against a new prompt, a different model, or your own code, and the results compared against what came before.

Dataset Items

There are three ways to fill a dataset. New Item adds one by hand, Upload CSV loads them in bulk, and Add to Datasets on a trace in Observability turns a real interaction into a test case, carrying its input, output, and metadata across.

Opening an item shows its input, expected output, and metadata alongside every run that has used it, which is how you tell whether one particular case has got better or worse over time.

Experiments

An experiment runs your system over every item in the dataset and records what came out. There are two kinds.

A prompt experiment is configured here with no code. Pick a context and the version of it to test, a provider and model, and the evaluators that should score the results. This is what lets you hold the dataset still and vary exactly one thing, a prompt version or a model, so that any difference in the output is attributable.

The prompt's variables take their values from the dataset by name, so a prompt containing {{input}} receives each item's input.

A custom experiment runs through the SDK instead, for when the thing under test is your own code rather than a prompt and a model.

Evaluators can be attached to the dataset itself as well as to a single experiment, so every run gets scored the same way without setting it up each time. Attached this way their variables map to fields from the experiment runs rather than from traces.

Runs and comparison

Every experiment produces a run. The Runs tab charts latency and average model cost across all of them, so a change that improved outputs while doubling the cost is visible before you read a single result. Any score in the project can be added as a chart of its own, which is how you watch quality move from one run to the next rather than only speed and spend.

Opening a run lists each item with its output beside the expected output, and links to the trace that produced it, so a disappointing result leads straight to the execution behind it.

Compare takes two or more runs and gives each one its own column next to the item's input and expected output, carrying the output that run produced along with its latency, cost, and any scores. That side-by-side is the point of holding the dataset still in the first place.