> For the complete documentation index, see [llms.txt](https://docs.interactive.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.interactive.ai/agents/operations/jev-experiment.md).

# Decisions-only models: activation, traversal, and thinking

Configure decisions-only models for activation, traversal, and thinking, with confidence fallback, tracing, and release verification.

Opt-in activation matching with independent decisions and optional LLM second opinions. Use a runtime and matching manifest schema that support this feature; changing the model on an older runtime is not sufficient.

Jev is the supported decisions-only model family (`interactive/typesafe/jev-1.13` and `interactive/typesafe/jev-1.14`); the examples below use 1.13. Validation and routing share an explicit supported-model set; adding a model requires confirming compatibility with the current decisions protocol and all three operations. There is no automatic provider or catalog discovery. Unknown `interactive/typesafe/` IDs remain rejected.

Merge these fields into an existing agent configuration (do not replace its other settings):

```yaml
agent_config:
  runtime:
    policy_batch_size: 5
  llms:
    evaluation:
      policy_matching: interactive/typesafe/jev-1.13
```

`policy_batch_size` remains configurable. Try 1, 5, 10, or 20; each eligible batch becomes one `/api/v1/decisions` request with one independent apply/skip choice per condition. There is no hardcoded batch size for Jev.

Jev handles observational conditions (including routine activation), actionable policies, and low-criticality policies. Reapplication, customer-dependent reapplication, response analysis, and disambiguation use `evaluation.default`. Routine navigation can independently opt in as described below, for both conversational and autonomous routines. Ordinary tool arguments and customer-facing generation remain on their existing models. Think-step inference has its own model setting as described below. Keep `evaluation.default` and `evaluation.fallback` on text-capable models; Jev is supported for policy matching, routine navigation, and eligible categorical think steps. It does not accept reasoning effort. Existing evaluation error retries/fallback remain active and logged.

Jev matches have empty rationales. Generation traces retain the exact decision request and a single `checks` list with apply/skip decisions (including skips), probabilities, confidence, config IDs, router request ID, timing, and billing. `reasoning_available: false` explicitly marks the lack of textual reasoning. No diagnostic categories, explanation model, or probability-based priority changes are used. Probabilities can vary between identical requests; do not assume exact determinism. The router currently lists Jev in the US region only.

## Optional confidence fallback

```yaml
agent_config:
  llms:
    evaluation:
      policy_matching:
        model: interactive/typesafe/jev-1.13
        confidence_threshold: 0.5
      fallback:
        model: interactive/openai/gpt-5.6-luna
        reasoning: high
```

The threshold is optional, limited to supported decisions-only models, and bounded from 0 to 1. The example value is not a calibrated recommendation. Omit it to preserve primary-only decisions. This setting controls agent routing; it is not sent to the decisions API. Normal LLMs reject `confidence_threshold`; the decisions adapter rejects `reasoning`. Both the published JSON Schema and runtime validation enforce this. Null means unset.

Conditions with `confidence <= confidence_threshold` are re-evaluated together in one smaller batch by `evaluation.fallback`, with its own reasoning setting. Both apply and skip decisions qualify. Original condition IDs, context, and examples are preserved; Jev's proposed answers are not sent to the fallback. Only selected decisions are replaced. Routine navigation and other non-activation schemas are unaffected. A threshold of 1 re-evaluates every condition; 0 still re-evaluates answers with zero confidence.

Jev's original generation span retains its decisions and confidence. The LLM span has a `(fallback)` suffix and metadata containing `fallback_reason: low_confidence`, `confidence_threshold`, and `reevaluated_policy_ids`. Each span records its own cost and latency. `decision_request_id` links the second opinion to its initial request. Events on the existing evaluation span record attempts, validation failures, and the final outcome, linked by `fallback_operation_id`. There is no separate `Confidence fallback` span. The engine consumes the merged decisions, including LLM rationales where the matching schema supports them.

Missing, extra, or duplicate fallback answers are rejected. Only the selected subset is retried, with the original Jev result retained throughout. If the LLM exhausts its attempts or time budget, the original Jev apply/skip decisions are used and the engine continues. The evaluation span's fallback event marks this degradation as `decisions_retained`; successful replacements are `llm_fallback`, while unaffected decisions remain `decisions`. Confidence requests a second opinion; it is not a guarantee that an unavailable LLM will provide one. Caller cancellation still propagates.

Setup or rendering failures in the second-opinion path also retain the original decisions, recording the exception and failure phase without retrying unexpected errors. Caller cancellation is never converted into retained decisions.

If Jev itself fails, ordinary batch retries and whole-batch error fallback still apply: there is no valid initial decision to retain. This availability-first exhaustion policy also applies to routine traversal: a complete, valid original decision/path is retained and executed if its second opinion fails.

Confidence describes the distribution's concentration, not a calibrated probability of correctness. Choose a threshold using replay outcomes and cost. This configuration needs a runtime and manifest schema containing the new field; the previously released `0.17.1rc60612` runtime does not support it.

## Routine traversal

```yaml
agent_config:
  llms:
    evaluation:
      routine_navigation:
        model: interactive/typesafe/jev-1.13
        confidence_threshold: 0.5
```

This enables next-step selection, backtrack checking, and backtrack destination selection together. The navigation threshold is independent of the activation threshold. Omitting it disables confidence second opinions; the inclusive threshold applies to staying, exiting, and declining backtracking too.

Next-step selection chooses one existing condition/path ID, not independent apply/skip answers. Backtrack selection also uses one question in one decisions request: it selects a complete candidate path starting at a visited node, or exits. There are no separate target-selection and advancement calls. Candidates respect the existing routine-map pruning boundary.

Both ordinary LLMs and decisions-only models use the same outcome validation and normalization. A starting tool may be complete if its execution result satisfies the current request; later tools require execution. Loops stop before repeating an action that needs fresh execution or customer input. Exit markers are normalized, including when the previous routine history already ended in an exit.

Only a unique, unconditional routing-only connector can be inserted into a path; missing customer actions or tool execution are never fabricated. Invalid paths, unknown next-step transitions, and incomplete activation answer sets trigger bounded error recovery for both model types, rather than silently becoming no matches.

Any uncertain choice escalates the entire affected traversal operation using its original LLM prompt, context and examples, without Jev's answers. A successful fallback replaces the whole decision/path. Exhaustion, timeout or setup failure retains the complete original Jev result; cancellation propagates.

Each actual model call has its own usage and billing. Jev navigation traces use `decisions` with question IDs and exclusive choices, not activation `checks`. Generation spans sit directly under the existing evaluation span, with no `Routine traversal` wrapper. A second opinion uses the same generation name with `(fallback)` appended; model names are metadata, not name suffixes. Events on the evaluation span record the final path and provenance; confidence fallback includes `reevaluated_question_ids` and `decision_request_ids`. These generic provenance labels replace the earlier `jev`/`jev_retained` and `jev_request_id(s)` labels; older traces retain their original labels. Actual model identity always comes from the configured model and router response. Completed speculative calls count even when their results are discarded. Conversational and autonomous routines share these selectors; enabling Jev changes navigation for both. Autonomous lifecycle handling is unchanged: completion still requires `built-in:emit_output`. Shared-selector tests do not replace end-to-end validation of callbacks, failures, cancellation, or iteration limits.

## Categorical think steps

`evaluation.think` independently controls inference at active `think` nodes. Omission inherits `evaluation.default` (including reasoning), not navigation or tools. Existing navigation settings no longer opt think nodes into Jev; explicitly configure `think` to retain that behavior:

```yaml
agent_config:
  llms:
    evaluation:
      think:
        model: interactive/typesafe/jev-1.13
        confidence_threshold: 0.5
```

One or more required string-enum or boolean fields can use Jev. Each field is a Choice question in the same model request; boolean choices map back to JSON booleans, not strings. String enums need at least two distinct nonempty values. The original instructions and context determine the choices; no tools-model classification runs first. Optional fields may be omitted.

Any required free-text string, numeric, or other unsupported field skips Jev and sends the whole step to `evaluation.fallback`. The warning identifies unsupported required fields and their types or enum constraints, along with the routine, step, and models. Conditional or referenced requirements also use fallback. No field name is special: a required free-text `rationale` causes fallback, while an enum named `rationale` is supported.

Both inference and the reason tool validate the original schema without relaxing any requirements. The think confidence threshold is inclusive; omission disables second opinions. Any uncertain field requests a second opinion for the complete output, never a patch of individual fields. An unavailable or invalid second opinion retains the complete valid original output. Invalid or incomplete primary output instead uses the existing bounded retries and evaluation fallback; if neither produces valid output, the operation fails without persisting invalid output.

Think nodes share an `Execute: Thinking` chain directly under their iteration. Actual inference appears beneath it as `Think: <step_id>`; only a confidence second opinion adds `(fallback)`. Jev trace choices include their category values and probabilities. Validation and session-state writes belong to the parent, without a duplicate `Thinking` tool observation. If enabled, the asynchronous `Response: Tool announcement` generation is another child of the same chain. See [thinking traces](https://docs.interactive.ai/agents/guides/observability.md#thinking-traces) for output and failure semantics.

## Checks

```bash
uv run pytest agent/tests/unit/test_decision_generator.py agent/tests/unit/test_jev_confidence_fallback.py agent/tests/unit/test_traversal_decisions.py agent/tests/unit/test_inference_parity.py agent/tests/unit/test_thinking.py

# Required before release: a limited test-project router key supplied securely.
# Exercises successful fallback, an unavailable model, exhausted budget,
# and cancellation using real calls.
for suite in test_jev_live test_jev_traversal_live test_jev_think_live; do
  JEV_ROUTER_API_KEY="$ROUTER_API_KEY" uv run pytest -s "agent/tests/integration/$suite.py"
done
```

The live check drives real matching batches at sizes 1, 5, and 10, validates matches and empty rationales, and inspects the emitted generation traces. It also forces confidence fallback for each supported schema and checks the paired spans. It verifies that an unavailable fallback does not repeat Jev or fail the batch, that an exhausted deadline retains original decisions, and that cancellation propagates. Agent image publication is gated on this check and targeted type/unit checks when relevant code, dependencies, or verification workflows changed since a successful ancestor image build. Missing history runs verification rather than skipping it. Version-only commits do not hide earlier unverified changes. The dedicated release key is stored as `JEV_ROUTER_API_KEY` in GitHub Actions secrets and must have a small spending limit. It does not deploy or modify an agent. Run it separately from other tracing tests because it installs an in-memory OpenTelemetry provider.

## Conversation comparison

Use the same content pins and other model settings on baseline and experimental candidate deployments. Prefer separate candidate agents rather than replacing an existing baseline. In `platform-development / interactive-agent`, compare:

* `agent-chat` and its Jev candidate on `replay-chat`.
* `agent-email` and its Jev candidate on `replay-email`.

For example, after deploying the experimental runtime/config to a candidate:

```bash
iai agents replay agent-chat --dataset replay-chat --repeat 3 \
  -o platform-development -p interactive-agent
iai agents replay <chat-jev-candidate> --dataset replay-chat --repeat 3 \
  -o platform-development -p interactive-agent
```

Repeat for email. Compare activation decisions, full-conversation assertions, latency, billed cost, and fallback frequency. Baseline decisions are not ground truth: review disagreements against the scenario expectations. Full replay results are still required before enabling Jev by default.

To disable Jev, restore the previous `evaluation.policy_matching` model (or remove the override to inherit `evaluation.default`).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.interactive.ai/agents/operations/jev-experiment.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
