> For the complete documentation index, see [llms.txt](https://docs.interactive.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.interactive.ai/agents/concepts/models.md). # Models > **Context** — Every model call an agent makes is configured by the manifest's `llms` block and routed through the InteractiveAI **LLM router** (the agent never calls a model provider directly). This page explains the two-lane, per-stage design — the most operationally important thing to understand about model configuration. > > YAML examples follow **manifest schema 7.0.0**. Manifest and content shapes are schema-versioned and differ across runtime versions — see [Versioning & compatibility](/agents/operations/versioning.md). ## The `llms` block ```yaml agent_config: llms: api_key: ${ROUTER_API_KEY} # router credential (required) # Customer-facing lane default: interactive/anthropic/claude-haiku-4.5 # lane default fallback: # router-side alternates (ordered) - interactive/anthropic/claude-sonnet-4-6 response: interactive/anthropic/claude-sonnet-4-6 # final message (optional) preamble: interactive/anthropic/claude-haiku-4.5 # preamble + tool announcement (optional) # Evaluation lane (internal engine inference) evaluation: default: interactive/google/gemini-3-flash-preview # lane default fallback: interactive/google/gemini-3.1-pro-preview # escalation on retry exhaustion tools: interactive/anthropic/claude-haiku-4.5 # tool-call argument inference startup: interactive/google/gemini-3.1-pro-preview # boot-time routine evaluation policy_matching: interactive/google/gemini-3-flash-preview routine_navigation: interactive/google/gemini-3.1-pro-preview ``` | Field | Type | Required | Default | Meaning | | ------------------------------- | ------------------------- | -------- | --------------------------------------------- | ---------------------------------------------------------------------------------------- | | `api_key` | string (`${VAR}` env-ref) | yes | — | Router credential — shared by **all** calls: customer-facing, evaluation, and embeddings | | `default` | string | no | `interactive/google/gemini-3-flash-preview` | Customer-facing lane default | | `fallback` | list of string | no | `[interactive/google/gemini-3.1-pro-preview]` | Router-side alternates, ordered; set `[]` for a single-model deployment | | `response` | string | no | inherits `default` | Final customer-facing message | | `preamble` | string | no | inherits `default` | Preamble **and** tool-call announcement (one generation stage drives both) | | `evaluation` | object or string | no | all evaluation defaults | Evaluation lane; a bare string is shorthand for `evaluation.default` | | `evaluation.default` | string | no | `interactive/google/gemini-3-flash-preview` | Evaluation lane default | | `evaluation.fallback` | string | no | `interactive/google/gemini-3.1-pro-preview` | Escalation model when an evaluation stage exhausts its retries | | `evaluation.tools` | string | no | inherits `evaluation.default` | Tool-call argument inference | | `evaluation.startup` | string | no | inherits `evaluation.default` | Boot-time routine evaluation | | `evaluation.policy_matching` | string | no | inherits `evaluation.default` | Per-turn policy matching + response analysis | | `evaluation.routine_navigation` | string | no | inherits `evaluation.default` | Next-step selection, backtrack check, backtrack step selection | Model names are `provider/model` aliases as served by the router's model catalog. There is no environment-variable override for model ids — change a model by editing the manifest and redeploying. **Resolution:** a stage that is not set inherits its lane's default — `response`/`preamble` from `default`, the four `evaluation.*` stages from `evaluation.default`. Stages never inherit across lanes: leaving `evaluation` out entirely gives you the evaluation defaults above, not your customer-facing model. ## Two lanes, two jobs ### Customer-facing lane — what the customer reads `default` (with the `fallback` list behind it) serves the **customer-visible** generations: the final reply (`response`) and the mid-turn niceties (`preamble` — which also produces the tool-call announcement). Optimise this lane for voice quality and instruction-following. **Fallback semantics:** `fallback` is an ordered list forwarded to the router along with the stage's primary; when the primary fails, the router tries each alternate in order. One request, router-side failover. ### Evaluation lane — decisions the customer never sees `evaluation.*` serves the engine's **internal structured-JSON decisions**: * tool-call argument inference (`tools`) — which tools to run, with what arguments, * policy matching and response analysis (`policy_matching`) — does this condition apply, and did the reply fulfil it?, * routine navigation (`routine_navigation`) — next-step selection and backtrack checks (typically the largest, hardest prompts of a turn), * routine metadata evaluation at startup (`startup`) — see [Startup evaluation](/agents/concepts/startup-evaluation.md). These are narrow, high-volume, schema-constrained calls — a fast, inexpensive model is the right lane default. Quality shows up as *correct routing and correct arguments*, not prose. `tools` is the highest-stakes stage of the lane: a wrong argument does real damage, so it's the first stage to promote to a stronger model. **Fallback semantics (different from the customer-facing lane!):** each evaluation call retries up to **3 attempts** on its stage's model; if all three fail (typically the model not conforming to the required output schema, or transport errors), the runtime swaps to `evaluation.fallback` and retries up to 3 more times on the bigger model. This is per call, not per process — one stubborn decision escalates alone. Every escalation is logged with a `[retry-fallback]` marker; see [Observability](/agents/guides/observability.md#retry-fallback-signals). ### The lanes never cross A failed customer-facing call falls through the `fallback` list — never to the evaluation models. A failed evaluation call escalates to `evaluation.fallback` — never to a customer-facing model. This is deliberate: * The customer-facing lane optimises for **voice**; the evaluation lane for **cheap, fast, narrow JSON**. A model great at one is often mediocre at the other. * Their failure profiles are orthogonal. A throughput problem on the chat model shouldn't degrade routing decisions, and schema-conformance issues on the evaluation model shouldn't change the agent's voice. ## Embeddings When the [knowledge base](/agents/concepts/knowledge-base.md) is `type: pgvector`, query embeddings use the manifest's `search.embedding_model`, routed through the same router with the same `api_key`. There is no separate embeddings credential. ## When each stage fires | Moment | Stage | Notes | | ----------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------ | | Every reply | `response` | | | Preamble / tool announcement | `preamble` | | | Knowledge-base query rewrite | `default` | One completion per retrieval | | Tool-call inference for your tools | `evaluation.tools` | | | Policy matching + response analysis, each turn | `evaluation.policy_matching` | Batched: `policy_batch_size` policies per call | | Routine activation / next-step / backtrack, each turn | `evaluation.routine_navigation` | | | Routine metadata evaluation, at boot (cold cache) | `evaluation.startup` | The slow part of cold startup — see [Startup evaluation](/agents/concepts/startup-evaluation.md) | | Knowledge-base embedding | `search.embedding_model` | | ## Operational guidance * **Token ceiling:** the operator env var `ROUTER_MAX_TOKENS` (default 100000) caps context size on router calls. See [Environment variables](/agents/reference/environment.md). * **Watch the escalation rate.** Frequent `[retry-fallback]` lines mean a stage's model is struggling with your content's complexity — either simplify conditions or promote that stage to a stronger model. Both models exhausting (logged at ERROR) fails the turn. * **Changing an evaluation model invalidates nothing**, but startup routine evaluation results are cached by *content* hash, so a model change does not bust the cache — re-evaluate deliberately if you change models and want fresh metadata (see [Startup evaluation](/agents/concepts/startup-evaluation.md)). * **Credential:** one router key serves everything. Rotate by updating the secret and restarting; see [Security](/agents/operations/security.md). ## See also * [Conversation lifecycle](/agents/concepts/conversation-lifecycle.md) — the calls in context * [Limits & defaults](/agents/reference/limits-and-defaults.md) — every default in one table * [Observability](/agents/guides/observability.md) — tracing model calls --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.interactive.ai/agents/concepts/models.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.