Datasets

Datasets are collections of test cases used to evaluate your LLM application systematically. Each dataset contains items with inputs, ideally expected outputs, and metadata that you run your AI system against to measure quality, detect regressions, and compare configurations.

Why Datasets Matter

LLM applications need structured evaluation beyond ad-hoc testing. Datasets let you:

Test your system against consistent, repeatable inputs
Compare performance across prompt versions, models, or configurations
Detect regressions before they reach production
Build a library of edge cases and known failure modes
Establish baselines for quality measurement

Creating Datasets

You can create datasets through the UI or programmatically via the InteractiveAI SDK.

Navigate to Improve > Datasets in the sidebar
Click + New Dataset
Enter a name, optional description, and optional metadata
Click Create Dataset

interactiveai.create_dataset(
    name="test-dataset",
    description="Question-answering test cases for model evaluation",
    metadata={
        "author": "evaluation-team",
        "purpose": "regression-testing",
        "version": "1.0"
    }
)

You can optionally provide JSON Schemas to validate dataset items on creation:

interactiveai.create_dataset(
    name="validated-dataset",
    description="Dataset with schema validation",
    metadata={
    "author": "evaluation-team",
    "purpose": "regression-testing",
    "version": "1.0"
    },
    input_schema={
        "type": "object",
        "properties": {
            "prompt": {"type": "string"},
            "context": {"type": "string"}
        },
        "required": ["prompt"]
    },
    expected_output_schema={
        "type": "object",
        "properties": {
            "answer": {"type": "string"}
        },
        "required": ["answer"]
    }
)

When input_schema or expected_output_schema are set, all new items will be validated against these schemas.

For the full create_dataset() API reference including upsert behavior and status options, see the SDK Documentation.

Adding Items to a Dataset

Each item represents a single test case with an input, expected output, and metadata. You can populate datasets through the UI or SDK.

There are three ways to add items through the UI:

From the Items tab:
1. Open a dataset from the Datasets list
2. Click the Items tab
3. Click + New Item
4. Enter the input, expected output, and metadata
5. Save the item
From a production trace:
1. Open any trace in the Govern > Traces view
2. Click Add to Dataset in the trace detail view
3. Select the target dataset
4. Optionally edit the input, expected output, and metadata
5. Confirm to add the item

This method is useful for building test cases from real user interactions.

Via CSV upload:
1. When creating a new dataset, drag and drop a CSV file or click to upload
2. Map CSV columns to input, expected output, and metadata fields
3. The items will be created automatically from your CSV rows

There are three ways to add items through the SDK:

You can add a single item:

interactiveai.create_dataset_item(
    dataset_name="test-dataset",
    input={"prompt": "What is the capital of Portugal?"},
    expected_output={"answer": "Lisbon", "confidence": "high"},
    metadata={"category": "geography", "difficulty": "easy"}
)

You can add multiple items at once:

dataset_items = [
    {
        "input": {"prompt": "Explain quantum computing in simple terms"},
        "expected_output": {"answer": "Quantum computing uses quantum mechanics..."},
        "metadata": {"category": "science", "difficulty": "medium"}
    },
    {
        "input": {"prompt": "What is 15 + 27?"},
        "expected_output": {"answer": "42"},
        "metadata": {"category": "math", "difficulty": "easy"}
    },
    {
        "input": {"prompt": "Translate 'Hello' to Spanish"},
        "expected_output": {"answer": "Hola"},
        "metadata": {"category": "translation", "difficulty": "easy"}
    }
]

for item in dataset_items:
    interactiveai.create_dataset_item(
        dataset_name="test-dataset",
        input=item["input"],
        expected_output=item["expected_output"],
        metadata=item["metadata"]
    )

You can create dataset items linked to production traces:

interactiveai.create_dataset_item(
    dataset_name="test-dataset",
    input={"prompt": "User's actual question from production"},
    expected_output={"answer": "Validated correct response"},
    source_trace_id="your-trace-id",
    source_observation_id="your-observation-id",
    metadata={"source": "production", "date": "2026-01-19"}
)

For the full create_dataset_item() API reference including upsert behavior and status options, see the SDK Documentation.

Translating Content

Click the translate icon in the dataset item header to translate the Input, Expected Output, and Metadata sections into your preferred language. The Runs section is not included in the translation.

PreviousScores NextExperiments

Last updated 3 months ago

Was this helpful?