Datasets

Datasets are collections of test cases used to evaluate your LLM application systematically. Each dataset contains items with inputs, ideally expected outputs, and metadata that you run your AI system against to measure quality, detect regressions, and compare configurations.

Why Datasets Matter

LLM applications need structured evaluation beyond ad-hoc testing. Datasets let you:

Test your system against consistent, repeatable inputs
Compare performance across prompt versions, models, or configurations
Detect regressions before they reach production
Build a library of edge cases and known failure modes
Establish baselines for quality measurement

Creating Datasets

You can create datasets through the UI or programmatically via the InteractiveAI SDK.

Navigate to Improvement > Datasets in the sidebar
Click + New Dataset
Enter a name, optional description, and optional metadata
Click Create Dataset

interactiveai.create_dataset(
    name="test-dataset", # your dataset name
    description="Question-answering test cases for model evaluation",
    metadata={
        "author": "evaluation-team",
        "purpose": "regression-testing",
        "version": "1.0"
    }
)

interactiveai.flush()  # Ensures all pending data is sent

Adding Items to a Dataset

Each item represents a single test case with an input, expected output, and metadata. You can populate datasets through the UI or SDK.

There are three ways to add items through the UI:

From the Items tab:
1. Open a dataset from the Datasets list
2. Click the Items tab
3. Click + New Item
4. Enter the input, expected output, and metadata
5. Save the item
From a production trace:
1. Open any trace in the Observability view
2. Click Add to Dataset in the trace detail view
3. Select the target dataset
4. Optionally edit the input, expected output, and metadata
5. Confirm to add the item

This method is useful for building test cases from real user interactions.

Via CSV upload:
1. When creating a new dataset, drag and drop a CSV file or click to upload
2. Map CSV columns to input, expected output, and metadata fields
3. The items will be created automatically from your CSV rows

There are three ways to add items through the SDK:

You can add a single item:

interactiveai.create_dataset_item(
    dataset_name="test-dataset",
    input={"prompt": "What is the capital of Portugal?"},
    expected_output={"answer": "Lisbon", "confidence": "high"},
    metadata={"category": "geography", "difficulty": "easy"}
)

You can add multiple items at once:

# Add a single item manually
interactiveai.create_dataset_item(
    dataset_name="test-dataset",
    input={"prompt": "What is the capital of Portugal?"},
    expected_output={"answer": "Lisbon", "confidence": "high"},
    metadata={"category": "geography", "difficulty": "easy"}
)

# Add multiple items manually 
dataset_items = [
    {
        "input": {"prompt": "Explain quantum computing in simple terms"},
        "expected_output": {"answer": "Quantum computing uses quantum mechanics..."},
        "metadata": {"category": "science", "difficulty": "medium"}
    },
    {
        "input": {"prompt": "What is 15 + 27?"},
        "expected_output": {"answer": "42"},
        "metadata": {"category": "math", "difficulty": "easy"}
    },
    {
        "input": {"prompt": "Translate 'Hello' to Spanish"},
        "expected_output": {"answer": "Hola"},
        "metadata": {"category": "translation", "difficulty": "easy"}
    }
]

for item in dataset_items:
    interactiveai.create_dataset_item(
        dataset_name="test-dataset",
        input=item["input"],
        expected_output=item["expected_output"],
        metadata=item["metadata"]
    )

interactiveai.flush() # Ensures all pending data is sent

You can create dataset items linked to production traces:

interactiveai.create_dataset_item(
    dataset_name="test-dataset",
    input={"prompt": "User's actual question from production"},
    expected_output={"answer": "Validated correct response"},
    metadata={"source": "production", "date": "2026-01-19"},
    source_trace_id="your-trace-id"
)

interactiveai.flush() # Ensures all pending data is sent

PreviousUser Tracking NextExperiments

Last updated 8 days ago

Was this helpful?

hashtagWhy Datasets Matter

hashtagCreating Datasets

hashtagAdding Items to a Dataset

Why Datasets Matter

Creating Datasets

Adding Items to a Dataset