Models
Per-stage model configuration: two lanes — customer-facing and evaluation — each with a default, per-stage overrides, and its own fallback mechanism. The lanes never cross.
Context — Every model call an agent makes is configured by the manifest's
llmsblock and routed through the InteractiveAI LLM router (the agent never calls a model provider directly). This page explains the two-lane, per-stage design — the most operationally important thing to understand about model configuration.YAML examples follow manifest schema 9.10.0. Manifest and content shapes are schema-versioned and differ across runtime versions — see Versioning & compatibility.
The llms block
agent_config:
llms:
api_key: ${ROUTER_API_KEY} # router credential (required)
# Customer-facing lane
default: interactive/anthropic/claude-haiku-4.5 # lane default
fallback: # backup models, tried in order
- interactive/anthropic/claude-sonnet-4-6
response: interactive/anthropic/claude-sonnet-4-6 # final message (optional)
preamble: interactive/anthropic/claude-haiku-4.5 # preamble + tool announcement (optional)
search_query: interactive/anthropic/claude-haiku-4.5 # KB search-query writing (optional)
# Evaluation lane (internal engine inference)
evaluation:
default: interactive/openai/gpt-5.6-luna # lane default
fallback: interactive/google/gemini-3.8-flash # escalation on retry exhaustion
tools: interactive/anthropic/claude-haiku-4.5 # tool-call argument inference
think: interactive/openai/gpt-5.6-luna # structured think-step output
policy_matching: interactive/google/gemini-3-flash-preview
routine_navigation: interactive/google/gemini-3.1-pro-preview
judge: interactive/google/gemini-3.1-pro-previewapi_key
string (${VAR} env-ref)
yes
—
Router credential — shared by all calls: customer-facing, evaluation, and embeddings
default
string
no
interactive/openai/gpt-5.6-luna
Customer-facing lane default
fallback
list of string
no
[interactive/google/gemini-3.8-flash]
Router-side alternates, ordered; set [] for a single-model deployment
response
string
no
inherits default
Final customer-facing message
preamble
string
no
inherits default
Preamble and tool-call announcement (one generation stage drives both)
search_query
string
no
inherits default
Writes the knowledge-base search queries (one call per query field per retrieval). No fallback chain — a failed rewrite falls back to the raw customer message
evaluation
object or string
no
all evaluation defaults
Evaluation lane; a bare string is shorthand for evaluation.default
evaluation.default
string
no
interactive/openai/gpt-5.6-luna
Evaluation lane default
evaluation.fallback
string
no
interactive/google/gemini-3.8-flash
Escalation model when an evaluation stage exhausts its retries
evaluation.tools
string
no
inherits evaluation.default
Tool-call argument inference
evaluation.policy_matching
string
no
inherits evaluation.default
Per-turn policy matching + response analysis
evaluation.routine_navigation
string
no
inherits evaluation.default
Next-step selection and backtracking
evaluation.think
string or model object
no
inherits evaluation.default
Structured think-step output; independent model, reasoning, and decisions confidence threshold
evaluation.judge
string
no
inherits evaluation.default
Replay judge: scores a replay scenario's rubric against the finished transcript. Pin it so that retuning evaluation.default does not move the verdicts replays are compared on
Model names are provider/model aliases as served by the router's model catalog. There is no environment-variable override for model ids — change a model by editing the manifest and redeploying.
Resolution: a stage that is not set inherits its lane's default — response/preamble/search_query from default, the evaluation.* slots from evaluation.default. Stages never inherit across lanes: leaving evaluation out entirely gives you the evaluation defaults above, not your customer-facing model.
Two lanes, two jobs
Customer-facing lane — what the customer reads
default (with the fallback list behind it) serves the customer-visible generations: the final reply (response) and the mid-turn niceties (preamble — which also produces the tool-call announcement). Optimise this lane for voice quality and instruction-following.
Fallback semantics: fallback is an ordered list of backup models. Writing the reply gets up to 3 attempts (runtime.generation_attempts) on its model. An attempt fails when the model returns an error or stays silent past the 3-minute attempt limit. Once those attempts are used up, the reply moves to the first fallback model with the same number of attempts, then the next, and only fails the turn once every model has failed. Every switch is logged with a [retry-fallback] marker and lane="response".
A streamed reply can only switch before any text reaches the customer; if it breaks off after that, the message is ended and the turn fails, since another attempt would send a second reply.
The preamble and tool-call announcement don't switch models: they are skipped if not written within 30 seconds, and send fallback to the router as alternates within each request instead.
Evaluation lane — decisions the customer never sees
evaluation.* serves the engine's internal structured-JSON decisions:
tool-call argument inference (
tools) — which tools to run, with what arguments,policy matching and response analysis (
policy_matching) — does this condition apply, and did the reply fulfil it?,think steps (
think) — structured output from the active think node,routine navigation (
routine_navigation) — next-step selection and backtrack checks (typically the largest, hardest prompts of a turn).
Startup is deterministic and makes no model calls.
These are narrow, high-volume, schema-constrained calls — a fast, inexpensive model is the right lane default. Quality shows up as correct routing and correct arguments, not prose. tools is the highest-stakes stage of the lane: a wrong argument does real damage, so it's the first stage to promote to a stronger model.
Fallback semantics: each evaluation call retries up to 3 attempts on its stage's model; if all three fail (typically the model not conforming to the required output schema, transport errors, or the model staying silent past the 3-minute attempt limit), the runtime swaps to evaluation.fallback and retries up to 3 more times on the bigger model. This is per call, not per process — one stubborn decision escalates alone. Every escalation is logged with a [retry-fallback] marker; see Observability.
The lanes never cross
A failed customer-facing call falls through the fallback list — never to the evaluation models. A failed evaluation call escalates to evaluation.fallback — never to a customer-facing model. This is deliberate:
The customer-facing lane optimises for voice; the evaluation lane for cheap, fast, narrow JSON. A model great at one is often mediocre at the other.
Their failure profiles are orthogonal. A throughput problem on the chat model shouldn't degrade routing decisions, and schema-conformance issues on the evaluation model shouldn't change the agent's voice.
Decisions-only models
The agent supports decisions-only models in evaluation.policy_matching, evaluation.routine_navigation, and evaluation.think. The supported models are interactive/typesafe/jev-1.13 and interactive/typesafe/jev-1.14 (Jev). Schema validation and runtime routing share the explicit SUPPORTED_DECISION_MODELS set in schemas/model_capabilities.py; a model must support the current decisions protocol and all three operations before being added. This is integration support, not discovery of models available to a router key. Unknown IDs in the reserved interactive/typesafe/ namespace remain rejected, not sent to chat.
These models cannot fill text-generation slots, including either lane default, evaluation.fallback, or customer-facing fallback lists. The current adapter does not accept reasoning. An optional confidence_threshold requests an ordinary LLM second opinion; ordinary LLMs do not accept that threshold.
For activation matching, only uncertain eligible policy conditions receive second opinions. Reapplication and response analysis stay on evaluation.default.
The second opinion uses evaluation.fallback. If it remains unavailable after bounded retries, activation continues with the primary's original decisions and traces record the degraded outcome. Independently, evaluation.routine_navigation can use a decisions-only model for all three routine traversal tasks with its own optional confidence_threshold. Uncertain traversal triggers a whole-operation second opinion; fallback failure retains and executes the complete original valid path. Next-step and backtrack-path selection each use one decisions question per operation. Backtracking selects a complete path, rather than separately predicting its target and advancement. Both model types share path validation, exit normalization, tool completion boundaries, and invalid-output recovery. Activation answers must cover every requested policy exactly once for either model type. These selectors are shared by conversational and autonomous routines, so the navigation setting affects both. Autonomous lifecycle handling remains unchanged.
Think-step inference uses evaluation.think, independently of evaluation.routine_navigation and evaluation.tools. If omitted, it inherits evaluation.default including reasoning, or the built-in evaluation default. The decisions adapter handles one or more required string-enum or boolean output fields together in one model request. Optional fields may be omitted. Any required free-text, numeric, or other unsupported field sends the entire step directly to evaluation.fallback, with a warning identifying each unsupported field and why it cannot use decisions-only inference. Conditional or referenced requirements also fall back. There are no field-name exceptions: a required free-text rationale requires fallback. Every output is validated against the original schema before persistence.
Think decisions use their own optional evaluation.think.confidence_threshold (inclusive); omission disables confidence second opinions. A low-confidence decision requests a complete ordinary-LLM output from the original context; invalid or unavailable fallback retains the valid primary output. Think inference does not execute or persist anything until the normal built-in:reason tool execution validates and stores the chosen output.
See Decisions-only models for configuration, tracing, validation, and release checks.
Embeddings
When the knowledge base is type: collections, vector fields are embedded with the model bound to their target collection slot — never a model declared in the manifest — routed through the same router with the same api_key. There is no separate embeddings credential.
When each stage fires
Every reply
response
Preamble / tool announcement
preamble
Search query rewrite
search_query (inherits default)
One completion per query field per retrieval
Tool-call inference for your tools
evaluation.tools
Policy matching + response analysis, each turn
evaluation.policy_matching
Batched: policy_batch_size policies per call
Routine next-step / backtrack, each turn
evaluation.routine_navigation
Activation uses evaluation.policy_matching
Active think node
evaluation.think
Categorical decisions or ordinary structured generation
Knowledge-base embedding
Target slot's embedding binding
type: collections vector fields only
Operational guidance
Token ceiling: the operator env var
ROUTER_MAX_TOKENS(default 100000) caps context size on router calls. See Environment variables.Watch the escalation rate. Frequent
[retry-fallback]lines mean a stage's model is struggling with your content's complexity — either simplify conditions or promote that stage to a stronger model. Both models exhausting (logged at ERROR) fails the turn.Changing an evaluation model does not change startup metadata. Routing tables depend on authored content and derivation-version constants, not on models. Model changes do not invalidate these cached results (see Startup evaluation).
Credential: one router key serves everything. Rotate by updating the secret and restarting; see Security.
See also
Conversation lifecycle — the calls in context
Limits & defaults — every default in one table
Observability — tracing model calls
Last updated
Was this helpful?

