Models

The agent uses two independent model lanes — chat (customer-facing) and evaluation (internal decisions) — each with its own primary and fallback. The lanes never cross.

Context — Every model call an agent makes is configured by the manifest's llms block and routed through the InteractiveAI LLM router (the agent never calls a model provider directly). This page explains the two-lane design — the most operationally important thing to understand about model configuration.

YAML examples follow manifest schema 6.1.5. Manifest and content shapes are schema-versioned and differ across runtime versions — see Versioning & compatibility.

The llms block

agent_config:
  llms:
    default: anthropic/claude-haiku-4.5          # chat primary (required)
    api_key: ${ROUTER_API_KEY}                   # router credential (required)
    fallback:                                    # chat fallbacks (optional list)
      - anthropic/claude-sonnet-4-6
    evaluation: google/gemini-3-flash-preview    # evaluation primary (optional)
    evaluation_fallback: google/gemini-3.1-pro-preview  # evaluation backup (optional)

Field	Type	Required	Default	Meaning
default	string	yes	interactive/agent	Chat primary
fallback	list of string	no	[]	Chat fallbacks, ordered list forwarded to the router
api_key	string (${VAR} env-ref)	yes	—	Router credential — shared by all calls, chat, evaluation, and embeddings
evaluation	string	no	interactive/google/gemini-3-flash-preview	Evaluation primary
evaluation_fallback	string	no	interactive/google/gemini-3.1-pro-preview	Evaluation backup

Model names are provider/model aliases as served by the router's model catalog. There is no environment-variable override for model ids — change a model by editing the manifest and redeploying.

Two lanes, two jobs

Chat lane — what the customer reads

default (with fallback behind it) serves every customer-visible generation: replies, preambles, content the agent writes, and the tool-call reasoning for the agent's real tools. Optimise this lane for voice quality and instruction-following.

Fallback semantics: fallback is an ordered list forwarded to the router along with default; when the primary fails, the router tries each fallback in order. One request, router-side failover.

Evaluation lane — decisions the customer never sees

evaluation (with evaluation_fallback behind it) serves the engine's internal structured-JSON decisions:

• policy matching (does this condition apply?),

• routine activation, next-step selection, and backtrack checks,

• routine metadata evaluation at startup (step reachability).

These are narrow, high-volume, schema-constrained calls — a fast, inexpensive model is the right default. Quality shows up as correct routing, not prose.

Fallback semantics (different from chat!): each evaluation call retries up to 3 attempts on the primary; if all three fail (typically the model not conforming to the required output schema, or transport errors), the runtime swaps to evaluation_fallback and retries up to 3 more times on the bigger model. This is per call, not per process — one stubborn decision escalates alone. Every escalation is logged with a [retry-fallback] marker; see Observability.

The lanes never cross

A failed chat call falls through the chat fallback list — never to the evaluation models. A failed evaluation call escalates to evaluation_fallback — never to a chat model. This is deliberate:

• The chat lane optimises for voice; the evaluation lane for cheap, fast, narrow JSON. A model great at one is often mediocre at the other.

• Their failure profiles are orthogonal. A throughput problem on the chat model shouldn't degrade routing decisions, and schema-conformance issues on the evaluation model shouldn't change the agent's voice.

Embeddings

When the knowledge base is type: pgvector, query embeddings use the manifest's search.embedding_model, routed through the same router with the same api_key. There is no separate embeddings credential.

When each lane fires

Moment	Lane	Notes
Every reply / preamble	Chat
Tool-call inference for your tools	Chat
Knowledge-base query rewrite	Chat	One completion on the chat primary per retrieval
Policy matching, each turn	Evaluation	Batched: policy_batch_size policies per call
Routine activation / next-step / backtrack, each turn	Evaluation
Routine metadata evaluation, at boot (cold cache)	Evaluation	The slow part of cold startup — see Startup evaluation
Knowledge-base embedding	search.embedding_model

Operational guidance

• Token ceiling: the operator env var ROUTER_MAX_TOKENS (default 100000) caps context size on router calls. See Environment variables.

• Watch the escalation rate. Frequent [retry-fallback] lines mean the evaluation primary is struggling with your content's complexity — either simplify conditions or promote a stronger evaluation model. Both models exhausting (logged at ERROR) fails the turn.

• Changing evaluation invalidates nothing, but startup routine evaluation results are cached by content hash, so a model change does not bust the cache — re-evaluate deliberately if you change models and want fresh metadata (see Startup evaluation).

• Credential: one router key serves everything. Rotate by updating the secret and restarting; see Security.

See also

• Conversation lifecycle — the calls in context

• Limits & defaults — every default in one table

• Observability — tracing model calls