Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Models

Per-stage model configuration: two lanes — customer-facing and evaluation — each with a default, per-stage overrides, and its own fallback mechanism. The lanes never cross.

Context — Every model call an agent makes is configured by the manifest's llms block and routed through the InteractiveAI LLM router (the agent never calls a model provider directly). This page explains the two-lane, per-stage design — the most operationally important thing to understand about model configuration.

YAML examples follow manifest schema 9.10.0. Manifest and content shapes are schema-versioned and differ across runtime versions — see Versioning & compatibility.

The llms block

agent_config:
  llms:
    api_key: ${ROUTER_API_KEY}                   # router credential (required)

    # Customer-facing lane
    default: interactive/anthropic/claude-haiku-4.5          # lane default
    fallback:                                    # backup models, tried in order
      - interactive/anthropic/claude-sonnet-4-6
    response: interactive/anthropic/claude-sonnet-4-6        # final message (optional)
    preamble: interactive/anthropic/claude-haiku-4.5         # preamble + tool announcement (optional)
    search_query: interactive/anthropic/claude-haiku-4.5     # KB search-query writing (optional)

    # Evaluation lane (internal engine inference)
    evaluation:
      default: interactive/openai/gpt-5.6-luna               # lane default
      fallback: interactive/google/gemini-3.8-flash          # escalation on retry exhaustion
      tools: interactive/anthropic/claude-haiku-4.5          # tool-call argument inference
      think: interactive/openai/gpt-5.6-luna                # structured think-step output
      policy_matching: interactive/google/gemini-3-flash-preview
      routine_navigation: interactive/google/gemini-3.1-pro-preview
      judge: interactive/google/gemini-3.1-pro-preview
Field
Type
Required
Default
Meaning

api_key

string (${VAR} env-ref)

yes

—

Router credential — shared by all calls: customer-facing, evaluation, and embeddings

default

string

no

interactive/openai/gpt-5.6-luna

Customer-facing lane default

fallback

list of string

no

[interactive/google/gemini-3.8-flash]

Router-side alternates, ordered; set [] for a single-model deployment

response

string

no

inherits default

Final customer-facing message

preamble

string

no

inherits default

Preamble and tool-call announcement (one generation stage drives both)

search_query

string

no

inherits default

Writes the knowledge-base search queries (one call per query field per retrieval). No fallback chain — a failed rewrite falls back to the raw customer message

evaluation

object or string

no

all evaluation defaults

Evaluation lane; a bare string is shorthand for evaluation.default

evaluation.default

string

no

interactive/openai/gpt-5.6-luna

Evaluation lane default

evaluation.fallback

string

no

interactive/google/gemini-3.8-flash

Escalation model when an evaluation stage exhausts its retries

evaluation.tools

string

no

inherits evaluation.default

Tool-call argument inference

evaluation.policy_matching

string

no

inherits evaluation.default

Per-turn policy matching + response analysis

evaluation.routine_navigation

string

no

inherits evaluation.default

Next-step selection and backtracking

evaluation.think

string or model object

no

inherits evaluation.default

Structured think-step output; independent model, reasoning, and decisions confidence threshold

evaluation.judge

string

no

inherits evaluation.default

Replay judge: scores a replay scenario's rubric against the finished transcript. Pin it so that retuning evaluation.default does not move the verdicts replays are compared on

Model names are provider/model aliases as served by the router's model catalog. There is no environment-variable override for model ids — change a model by editing the manifest and redeploying.

Resolution: a stage that is not set inherits its lane's default — response/preamble/search_query from default, the evaluation.* slots from evaluation.default. Stages never inherit across lanes: leaving evaluation out entirely gives you the evaluation defaults above, not your customer-facing model.

Two lanes, two jobs

Customer-facing lane — what the customer reads

default (with the fallback list behind it) serves the customer-visible generations: the final reply (response) and the mid-turn niceties (preamble — which also produces the tool-call announcement). Optimise this lane for voice quality and instruction-following.

Fallback semantics: fallback is an ordered list of backup models. Writing the reply gets up to 3 attempts (runtime.generation_attempts) on its model. An attempt fails when the model returns an error or stays silent past the 3-minute attempt limit. Once those attempts are used up, the reply moves to the first fallback model with the same number of attempts, then the next, and only fails the turn once every model has failed. Every switch is logged with a [retry-fallback] marker and lane="response".

A streamed reply can only switch before any text reaches the customer; if it breaks off after that, the message is ended and the turn fails, since another attempt would send a second reply.

The preamble and tool-call announcement don't switch models: they are skipped if not written within 30 seconds, and send fallback to the router as alternates within each request instead.

Evaluation lane — decisions the customer never sees

evaluation.* serves the engine's internal structured-JSON decisions:

  • tool-call argument inference (tools) — which tools to run, with what arguments,

  • policy matching and response analysis (policy_matching) — does this condition apply, and did the reply fulfil it?,

  • think steps (think) — structured output from the active think node,

  • routine navigation (routine_navigation) — next-step selection and backtrack checks (typically the largest, hardest prompts of a turn).

Startup is deterministic and makes no model calls.

These are narrow, high-volume, schema-constrained calls — a fast, inexpensive model is the right lane default. Quality shows up as correct routing and correct arguments, not prose. tools is the highest-stakes stage of the lane: a wrong argument does real damage, so it's the first stage to promote to a stronger model.

Fallback semantics: each evaluation call retries up to 3 attempts on its stage's model; if all three fail (typically the model not conforming to the required output schema, transport errors, or the model staying silent past the 3-minute attempt limit), the runtime swaps to evaluation.fallback and retries up to 3 more times on the bigger model. This is per call, not per process — one stubborn decision escalates alone. Every escalation is logged with a [retry-fallback] marker; see Observability.

The lanes never cross

A failed customer-facing call falls through the fallback list — never to the evaluation models. A failed evaluation call escalates to evaluation.fallback — never to a customer-facing model. This is deliberate:

  • The customer-facing lane optimises for voice; the evaluation lane for cheap, fast, narrow JSON. A model great at one is often mediocre at the other.

  • Their failure profiles are orthogonal. A throughput problem on the chat model shouldn't degrade routing decisions, and schema-conformance issues on the evaluation model shouldn't change the agent's voice.

Decisions-only models

The agent supports decisions-only models in evaluation.policy_matching, evaluation.routine_navigation, and evaluation.think. The supported models are interactive/typesafe/jev-1.13 and interactive/typesafe/jev-1.14 (Jev). Schema validation and runtime routing share the explicit SUPPORTED_DECISION_MODELS set in schemas/model_capabilities.py; a model must support the current decisions protocol and all three operations before being added. This is integration support, not discovery of models available to a router key. Unknown IDs in the reserved interactive/typesafe/ namespace remain rejected, not sent to chat.

These models cannot fill text-generation slots, including either lane default, evaluation.fallback, or customer-facing fallback lists. The current adapter does not accept reasoning. An optional confidence_threshold requests an ordinary LLM second opinion; ordinary LLMs do not accept that threshold.

For activation matching, only uncertain eligible policy conditions receive second opinions. Reapplication and response analysis stay on evaluation.default.

The second opinion uses evaluation.fallback. If it remains unavailable after bounded retries, activation continues with the primary's original decisions and traces record the degraded outcome. Independently, evaluation.routine_navigation can use a decisions-only model for all three routine traversal tasks with its own optional confidence_threshold. Uncertain traversal triggers a whole-operation second opinion; fallback failure retains and executes the complete original valid path. Next-step and backtrack-path selection each use one decisions question per operation. Backtracking selects a complete path, rather than separately predicting its target and advancement. Both model types share path validation, exit normalization, tool completion boundaries, and invalid-output recovery. Activation answers must cover every requested policy exactly once for either model type. These selectors are shared by conversational and autonomous routines, so the navigation setting affects both. Autonomous lifecycle handling remains unchanged.

Think-step inference uses evaluation.think, independently of evaluation.routine_navigation and evaluation.tools. If omitted, it inherits evaluation.default including reasoning, or the built-in evaluation default. The decisions adapter handles one or more required string-enum or boolean output fields together in one model request. Optional fields may be omitted. Any required free-text, numeric, or other unsupported field sends the entire step directly to evaluation.fallback, with a warning identifying each unsupported field and why it cannot use decisions-only inference. Conditional or referenced requirements also fall back. There are no field-name exceptions: a required free-text rationale requires fallback. Every output is validated against the original schema before persistence.

Think decisions use their own optional evaluation.think.confidence_threshold (inclusive); omission disables confidence second opinions. A low-confidence decision requests a complete ordinary-LLM output from the original context; invalid or unavailable fallback retains the valid primary output. Think inference does not execute or persist anything until the normal built-in:reason tool execution validates and stores the chosen output.

See Decisions-only models for configuration, tracing, validation, and release checks.

Embeddings

When the knowledge base is type: collections, vector fields are embedded with the model bound to their target collection slot — never a model declared in the manifest — routed through the same router with the same api_key. There is no separate embeddings credential.

When each stage fires

Moment
Stage
Notes

Every reply

response

Preamble / tool announcement

preamble

Search query rewrite

search_query (inherits default)

One completion per query field per retrieval

Tool-call inference for your tools

evaluation.tools

Policy matching + response analysis, each turn

evaluation.policy_matching

Batched: policy_batch_size policies per call

Routine next-step / backtrack, each turn

evaluation.routine_navigation

Activation uses evaluation.policy_matching

Active think node

evaluation.think

Categorical decisions or ordinary structured generation

Routine metadata derivation, at boot

No model

Deterministic — see Startup evaluation

Knowledge-base embedding

Target slot's embedding binding

type: collections vector fields only

Operational guidance

  • Token ceiling: the operator env var ROUTER_MAX_TOKENS (default 100000) caps context size on router calls. See Environment variables.

  • Watch the escalation rate. Frequent [retry-fallback] lines mean a stage's model is struggling with your content's complexity — either simplify conditions or promote that stage to a stronger model. Both models exhausting (logged at ERROR) fails the turn.

  • Changing an evaluation model does not change startup metadata. Routing tables depend on authored content and derivation-version constants, not on models. Model changes do not invalidate these cached results (see Startup evaluation).

  • Credential: one router key serves everything. Rotate by updating the secret and restarting; see Security.

See also

Last updated

Was this helpful?