> For the complete documentation index, see [llms.txt](https://docs.interactive.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.interactive.ai/agents/concepts/models.md).

# Models

Per-stage model configuration: two lanes — customer-facing and evaluation — each with a default, per-stage overrides, and its own fallback mechanism. The lanes never cross.

> **Context** — Every model call an agent makes is configured by the manifest's `llms` block and routed through the InteractiveAI **LLM router** (the agent never calls a model provider directly). This page explains the two-lane, per-stage design — the most operationally important thing to understand about model configuration.
>
> YAML examples follow **manifest schema 1.4.0**. Manifest and content shapes are schema-versioned and differ across runtime versions — see [Versioning & compatibility](https://docs.interactive.ai/agents/operations/versioning.md).

## The `llms` block

```yaml
agent_config:
  llms:
    api_key: ${ROUTER_API_KEY}                   # router credential (required)

    # Customer-facing lane
    default: interactive/anthropic/claude-haiku-4.5          # lane default
    fallback:                                    # backup models, tried in order
      - interactive/anthropic/claude-sonnet-4-6
    response: interactive/anthropic/claude-sonnet-4-6        # final message (optional)
    preamble: interactive/anthropic/claude-haiku-4.5         # preamble + tool announcement (optional)
    search_query: interactive/anthropic/claude-haiku-4.5     # KB search-query writing (optional)

    # Evaluation lane (internal engine inference)
    evaluation:
      default: interactive/openai/gpt-6-luna                 # lane default
      fallback: interactive/anthropic/claude-haiku-5.5       # escalation on retry exhaustion
      tools: interactive/anthropic/claude-haiku-4.5          # tool-call argument inference
      think: interactive/openai/gpt-5.6-luna                # structured think-step output
      policy_matching: interactive/google/gemini-3-flash-preview
      routine_navigation: interactive/google/gemini-3.1-pro-preview
      judge: interactive/google/gemini-3.1-pro-preview
```

| Field                           | Type                      | Required | Default                                    | Meaning                                                                                                                                                                          |
| ------------------------------- | ------------------------- | -------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key`                       | string (`${VAR}` env-ref) | yes      | —                                          | Router credential — shared by **all** calls: customer-facing, evaluation, and embeddings                                                                                         |
| `default`                       | string                    | no       | `interactive/openai/gpt-6-luna`            | Customer-facing lane default                                                                                                                                                     |
| `fallback`                      | list of string            | no       | `[interactive/anthropic/claude-haiku-5.5]` | Router-side alternates, ordered; set `[]` for a single-model deployment                                                                                                          |
| `response`                      | string                    | no       | inherits `default`                         | Final customer-facing message                                                                                                                                                    |
| `preamble`                      | string                    | no       | inherits `default`                         | Preamble **and** tool-call announcement (one generation stage drives both)                                                                                                       |
| `search_query`                  | string                    | no       | inherits `default`                         | Writes the knowledge-base search queries (one call per query field per retrieval). No fallback chain — a failed rewrite falls back to the raw customer message                   |
| `evaluation`                    | object or string          | no       | all evaluation defaults                    | Evaluation lane; a bare string is shorthand for `evaluation.default`                                                                                                             |
| `evaluation.default`            | string                    | no       | `interactive/openai/gpt-6-luna`            | Evaluation lane default                                                                                                                                                          |
| `evaluation.fallback`           | string                    | no       | `interactive/anthropic/claude-haiku-5.5`   | Escalation model when an evaluation stage exhausts its retries                                                                                                                   |
| `evaluation.tools`              | string                    | no       | inherits `evaluation.default`              | Tool-call argument inference                                                                                                                                                     |
| `evaluation.policy_matching`    | string                    | no       | inherits `evaluation.default`              | Per-turn policy matching + response analysis                                                                                                                                     |
| `evaluation.routine_navigation` | string                    | no       | inherits `evaluation.default`              | Next-step selection and backtracking                                                                                                                                             |
| `evaluation.think`              | string or model object    | no       | inherits `evaluation.default`              | Structured think-step output; independent model, reasoning, and decisions confidence threshold                                                                                   |
| `evaluation.judge`              | string                    | no       | inherits `evaluation.default`              | Replay judge: scores a replay scenario's rubric against the finished transcript. Pin it so that retuning `evaluation.default` does not move the verdicts replays are compared on |

Model names are `provider/model` aliases as served by the router's model catalog. There is no environment-variable override for model ids — change a model by editing the manifest and redeploying.

**Resolution:** a stage that is not set inherits its lane's default — `response`/`preamble`/`search_query` from `default`, the `evaluation.*` slots from `evaluation.default`. Stages never inherit across lanes: leaving `evaluation` out entirely gives you the evaluation defaults above, not your customer-facing model.

## Two lanes, two jobs

### Customer-facing lane — what the customer reads

`default` (with the `fallback` list behind it) serves the **customer-visible** generations: the final reply (`response`) and the mid-turn niceties (`preamble` — which also produces the tool-call announcement). Optimise this lane for voice quality and instruction-following.

**Fallback semantics:** `fallback` is an ordered list of backup models. Writing the reply gets up to **3 attempts** (`runtime.generation_attempts`) on its model. An attempt fails when the model returns an error **or** stays silent past the 3-minute attempt limit. Once those attempts are used up, the reply moves to the first `fallback` model with the same number of attempts, then the next, and only fails the turn once every model has failed. Every switch is logged with a `[retry-fallback]` marker and `lane="response"`.

A streamed reply can only switch before any text reaches the customer; if it breaks off after that, the message is ended and the turn fails, since another attempt would send a second reply.

The preamble and tool-call announcement don't switch models: they are skipped if not written within 30 seconds, and send `fallback` to the router as alternates within each request instead.

### Evaluation lane — decisions the customer never sees

`evaluation.*` serves the engine's **internal structured-JSON decisions**:

* tool-call argument inference (`tools`) — which tools to run, with what arguments,
* policy matching and response analysis (`policy_matching`) — does this condition apply, and did the reply fulfil it?,
* think steps (`think`) — structured output from the active think node,
* routine navigation (`routine_navigation`) — next-step selection and backtrack checks (typically the largest, hardest prompts of a turn).

Startup is [deterministic](https://docs.interactive.ai/agents/concepts/startup-evaluation.md) and makes no model calls.

These are narrow, high-volume, schema-constrained calls — a fast, inexpensive model is the right lane default. Quality shows up as *correct routing and correct arguments*, not prose. `tools` is the highest-stakes stage of the lane: a wrong argument does real damage, so it's the first stage to promote to a stronger model.

**Fallback semantics:** each evaluation call retries up to **3 attempts** on its stage's model; if all three fail (typically the model not conforming to the required output schema, transport errors, or the model staying silent past the 3-minute attempt limit), the runtime swaps to `evaluation.fallback` and retries up to 3 more times on the fallback model. This is per call, not per process — one stubborn decision escalates alone. Every escalation is logged with a `[retry-fallback]` marker; see [Observability](https://docs.interactive.ai/agents/guides/observability.md#retry-fallback-signals).

### The lanes never cross

A failed customer-facing call falls through the `fallback` list — never to the evaluation models. A failed evaluation call escalates to `evaluation.fallback` — never to a customer-facing model. This is deliberate:

* The customer-facing lane optimises for **voice**; the evaluation lane for **cheap, fast, narrow JSON**. A model great at one is often mediocre at the other.
* Their failure profiles are orthogonal. A throughput problem on the chat model shouldn't degrade routing decisions, and schema-conformance issues on the evaluation model shouldn't change the agent's voice.

## Decisions-only models

The agent supports decisions-only models in `evaluation.policy_matching`, `evaluation.routine_navigation`, and `evaluation.think`. The supported models are `interactive/typesafe/jev-1.13` and `interactive/typesafe/jev-1.14` (Jev). Schema validation and runtime routing share the explicit `SUPPORTED_DECISION_MODELS` set in `schemas/model_capabilities.py`; a model must support the current decisions protocol and all three operations before being added. This is integration support, not discovery of models available to a router key. Unknown IDs in the reserved `interactive/typesafe/` namespace remain rejected, not sent to chat.

These models cannot fill text-generation slots, including either lane default, `evaluation.fallback`, or customer-facing fallback lists. The current adapter does not accept `reasoning`. An optional `confidence_threshold` requests an ordinary LLM second opinion; ordinary LLMs do not accept that threshold.

For activation matching, only uncertain eligible policy conditions receive second opinions. Reapplication and response analysis stay on `evaluation.default`.

The second opinion uses `evaluation.fallback`. If it remains unavailable after bounded retries, activation continues with the primary's original decisions and traces record the degraded outcome. Independently, `evaluation.routine_navigation` can use a decisions-only model for all three routine traversal tasks with its own optional `confidence_threshold`. Uncertain traversal triggers a whole-operation second opinion; fallback failure retains and executes the complete original valid path. Next-step and backtrack-path selection each use one decisions question per operation. Backtracking selects a complete path, rather than separately predicting its target and advancement. Both model types share path validation, exit normalization, tool completion boundaries, and invalid-output recovery. Activation answers must cover every requested policy exactly once for either model type. These selectors are shared by conversational and autonomous routines, so the navigation setting affects both. Autonomous lifecycle handling remains unchanged.

Think-step inference uses `evaluation.think`, independently of `evaluation.routine_navigation` and `evaluation.tools`. If omitted, it inherits `evaluation.default` including reasoning, or the built-in evaluation default. The decisions adapter handles one or more required string-enum or boolean output fields together in one model request. Optional fields may be omitted. Any required free-text, numeric, or other unsupported field sends the entire step directly to `evaluation.fallback`, with a warning identifying each unsupported field and why it cannot use decisions-only inference. Conditional or referenced requirements also fall back. There are no field-name exceptions: a required free-text `rationale` requires fallback. Every output is validated against the original schema before persistence.

Think decisions use their own optional `evaluation.think.confidence_threshold` (inclusive); omission disables confidence second opinions. A low-confidence decision requests a complete ordinary-LLM output from the original context; invalid or unavailable fallback retains the valid primary output. Think inference does not execute or persist anything until the normal `built-in:reason` tool execution validates and stores the chosen output.

See [Decisions-only models](https://docs.interactive.ai/agents/operations/jev-experiment.md) for configuration, tracing, validation, and release checks.

## Embeddings

When the [knowledge base](https://docs.interactive.ai/agents/concepts/knowledge-base.md) is `type: collections`, vector fields are embedded with the model bound to their target collection slot — never a model declared in the manifest — routed through the same router with the same `api_key`. There is no separate embeddings credential.

## When each stage fires

| Moment                                         | Stage                               | Notes                                                                                                       |
| ---------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Every reply                                    | `response`                          |                                                                                                             |
| Preamble / tool announcement                   | `preamble`                          |                                                                                                             |
| Search query rewrite                           | `search_query` (inherits `default`) | One completion per query field per retrieval                                                                |
| Tool-call inference for your tools             | `evaluation.tools`                  |                                                                                                             |
| Policy matching + response analysis, each turn | `evaluation.policy_matching`        | Batched: `policy_batch_size` policies per call                                                              |
| Routine next-step / backtrack, each turn       | `evaluation.routine_navigation`     | Activation uses `evaluation.policy_matching`                                                                |
| Active think node                              | `evaluation.think`                  | Categorical decisions or ordinary structured generation                                                     |
| Routine metadata derivation, at boot           | No model                            | Deterministic — see [Startup evaluation](https://docs.interactive.ai/agents/concepts/startup-evaluation.md) |
| Knowledge-base embedding                       | Target slot's embedding binding     | `type: collections` vector fields only                                                                      |

## Prompt caching

Most of a prompt repeats from one call to the next: the instructions, the conversation so far, and the tool results already gathered. The agent marks where that repeated start ends so the model provider can read it back from its cache, which costs a fraction of fresh input and returns sooner. `runtime.prompt_cache` in the [manifest](https://docs.interactive.ai/agents/reference/manifest.md) controls it:

```yaml
agent_config:
  runtime:
    prompt_cache:
      enabled: true            # default
      tool_results: autonomous # off | autonomous | always (default autonomous)
```

* **One provider host per conversation.** A model is often served by several hosts, each with its own cache. The agent keeps a conversation's calls on one host, so the next call finds what the previous one cached.
* **Tool results.** A turn that calls many tools, such as an autonomous run reading a document page by page, sends every earlier result again on each call. With `tool_results`, those results are cached once they reach about 2,000 tokens, and each call reads back what the previous one sent (past 40 results, all but the last few results). The default, `autonomous`, applies it to autonomous routine runs, where a whole run is one turn; `always` adds conversational turns.
* **Gemini.** A cache write costs less than fresh input on Gemini, so every call caches nearly its whole prompt, tool results included, whatever `tool_results` says.
* **Turning it off.** `enabled: false` sends every prompt uncached. Use it only to rule caching out while diagnosing a provider error.

Each model call in a trace records its cached and newly written input tokens, so the effect shows per call (see [Observability](https://docs.interactive.ai/agents/guides/observability.md)).

## Operational guidance

* **Token ceiling:** the operator env var `ROUTER_MAX_TOKENS` (default 100000) caps context size on router calls. See [Environment variables](https://docs.interactive.ai/agents/reference/environment.md).
* **Watch the escalation rate.** Frequent `[retry-fallback]` lines mean a stage's model is struggling with your content's complexity — either simplify conditions or promote that stage to a stronger model. Both models exhausting (logged at ERROR) fails the turn.
* **Changing an evaluation model does not change startup metadata.** Routing tables depend on authored content and derivation-version constants, not on models. Model changes do not invalidate these cached results (see [Startup evaluation](https://docs.interactive.ai/agents/concepts/startup-evaluation.md)).
* **Credential:** one router key serves everything. Rotate by updating the secret and restarting; see [Security](https://docs.interactive.ai/agents/operations/security.md).

## See also

* [Conversation lifecycle](https://docs.interactive.ai/agents/concepts/conversation-lifecycle.md) — the calls in context
* [Limits & defaults](https://docs.interactive.ai/agents/reference/limits-and-defaults.md) — every default in one table
* [Observability](https://docs.interactive.ai/agents/guides/observability.md) — tracing model calls


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.interactive.ai/agents/concepts/models.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
