Score Configs

Score configs are schema definitions that standardize how your team evaluates AI outputs. They ensure that every score follows a consistent structure, making your evaluation data comparable and reliable for analysis over time.

Why Score Configs Matter

Without standardized schemas, scoring becomes inconsistent. One team member might rate quality on a 1-5 scale while another uses 1-10. Someone might use "good/bad" categories while others use "excellent/acceptable/poor". Score configs eliminate this ambiguity by defining exactly what values are valid for each score type.

Score configs are required for Human Annotation. When you create an annotation queue, you select which score configs annotators will use to evaluate traces. This guarantees that all human reviewers apply the same criteria and scale.

---

Data Types

Score configs support three data types:

• Numeric scores accept decimal values within an optional range. You can set minimum and maximum bounds to constrain valid values. If no bounds are specified, any number is accepted.

• Categorical scores accept only predefined labels. Each category has a numeric value and a text label. For example, a quality score might define categories like 0 = "Poor", 1 = "Acceptable", 2 = "Good", 3 = "Excellent".

• Boolean scores accept only 0 or 1, representing false and true respectively.

---

Creating a Score Config:

1. Navigate to Settings → Score Configs

2. Click + Add New

3. Configure the following fields:

   • Name: A descriptive identifier for this scoring dimension

   • Data Type: Select Numeric, Categorical, or Boolean

   • Values: Define the valid range or categories

   • Description (optional): Guidance for annotators on how to apply this score

4. Click Create Config

The config becomes immediately available for assessment.

---

Managing Score Configs

The Score Configs table displays all configs in your project with their name, data type, range or categories, description, and status.

You cannot edit or delete score configs. If a config is no longer needed, you can archive it. Archiving removes the config from the selection list when creating annotation queues, but scores already recorded against that config remain visible in your project. To restore an archived config, click the archive icon again.