# Annotations

Annotation enables **domain experts** to **manually evaluate** AI outputs by adding scores and comments to traces, observations, or sessions. This approach establishes **ground truth** for your evaluation infrastructure and provides reference points for **benchmarking** automated evaluators.

{% hint style="info" %}
Before creating annotations, you need at least one **Score Config** defined in your project. Score configs determine which scoring dimensions are available during annotation. Refer to [Score Configs](/settings/score-configs.md) to know more about it.
{% endhint %}

### Why Annotation Matters

Automated evaluation can scale, but human judgment remains essential for establishing what "good" actually means in your specific context. Human annotation serves **two purposes**:

* **Direct Assessment** allows multiple team members to review AI outputs and score them against defined criteria. This builds high-quality labeled datasets and surfaces issues that automated systems might miss.
* **Calibration** aligns your LLM-as-a-Judge evaluators with human judgment. By comparing automated scores against expert annotations, you can identify where your evaluators drift from human expectations and adjust accordingly.

***

### Annotation Queues

Annotation Queues streamline the process of working through **batches of items** that need review. Instead of hunting through traces one by one, queues let you organize work and track progress across your team.

#### **Creating a Queue**

1. Navigate to **Improve → Annotations**
2. Click **+ New Queue**
3. Configure the queue:
   * **Name**: Identifier for this annotation task
   * **Description**: Optional context about what reviewers should focus on
   * **Score Config**: Select which scoring dimensions annotators will use

<div data-with-frame="true"><figure><img src="/files/954Mv2CqsBNK0wPWx8rN" alt=""><figcaption></figcaption></figure></div>

#### **Adding Items to a Queue**

You can add traces to a queue individually or in bulk from the Traces view:

| Method         | How to do it                                                                                                 |
| -------------- | ------------------------------------------------------------------------------------------------------------ |
| Single item    | Open a trace, click **Annotate** dropdown, select the **Score Config** of your choosing and score that trace |
| Bulk selection | Select traces using checkboxes, click **Actions** dropdown, then **+ Add to Annotation Queue**               |

Use filters on the Traces view to narrow down to the specific subset you want reviewed before adding to a queue.

#### **Processing a Queue**

Click **Process Queue** to enter the annotation interface. Each item displays its content alongside the scoring panel.

<div data-with-frame="true"><figure><img src="/files/DDHbamhgyPo6A755pkVD" alt=""><figcaption></figcaption></figure></div>

* **Focused view** shows Input, Output, and Metadata in a simplified layout for rapid review.
* **Detailed view** displays the full trace with execution graph, latency, cost, and the Run/Scores tabs. Use this when you need complete context to make a judgment.

For each item:

1. Review the content in the main panel
2. Enter scores for each dimension in the **Annotate** panel on the right
3. Click **Complete + Next** to save and move to the next item

The queue tracks completion status and who completed each item, giving you visibility into annotation progress.

***

### Single-Trace Annotation

For **ad-hoc review** outside of queues, you can annotate any trace directly:

1. Open a trace, observation, or session detail view
2. Click **Annotate**
3. Select which Score Configs to use
4. Enter score values
5. Scores appear in the **Scores** tab on the detail view

<div data-with-frame="true"><figure><img src="/files/6KLbgh8j1gEZGpHKzXaN" alt=""><figcaption></figcaption></figure></div>

This approach works well for investigating flagged issues, validating specific outputs, or quick spot-checks.

***

### Viewing Annotation Results

Scores created through annotation appear in multiple places:

* **Trace detail view**: Click the Scores tab to see all scores attached to that trace
* **Scores page**: View all scores across your project with filtering and export options
* **Dashboards**: Aggregate annotation data appears in your quality metrics

{% hint style="info" %}
Use the Source filter on the Scores page to isolate human annotations (ANNOTATION) from automated evaluations (EVAL) or evaluation made via the InteractiveAI SDK (API) .
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.interactive.ai/improve/annotations.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.