> For the complete documentation index, see [llms.txt](https://docs.interactive.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.interactive.ai/agents/concepts/startup-evaluation.md).

# Startup evaluation

What the engine derives about your routines at boot, which behavioural flags you author instead, and how the routing table is cached.

> **Context** — Before an agent serves traffic, the engine derives each routine's routing table and reads the behavioural flags you authored. This page explains what the runtime needs, where each fact comes from, and how the routing table is cached. It assumes [Routines](https://docs.interactive.ai/agents/concepts/routines.md) (nodes, transitions) and [Policies](https://docs.interactive.ai/agents/concepts/policies.md); watching it run is in [Observability](https://docs.interactive.ai/agents/guides/observability.md).
>
> YAML examples follow **manifest schema 1.4.0**. Manifest and content shapes are schema-versioned and differ across runtime versions — see [Versioning & compatibility](https://docs.interactive.ai/agents/operations/versioning.md).

## Why evaluation exists

Your routines and policies are written in natural language. To run a turn quickly and consistently, the engine needs a few structured facts about them: *does this step wait for the customer's reply? does this policy stay in force every turn? which node can come next, and when?*

Those facts used to be worked out by a model at every cold boot and cached. Two boots of identical content did not agree on them, and the boot took minutes and cost money. They are now **authored or derived**: the flags are fields you set (with sensible defaults from the routine's shape), and the routing table is computed from your transitions — no model is needed to become ready. An advisory authoring review runs separately after the HTTP listener binds. What remains of graph "evaluation" is the routing table, derived once per routine and cached because the runtime reads it from node metadata.

## Where it sits in boot

Startup evaluation is the last phase of the [boot sequence](https://docs.interactive.ai/agents/concepts/architecture.md#boot-sequence), and it runs **before the agent begins serving**: the HTTP port binds and health checks start passing only once it settles. It is a graph walk over your routines and completes in seconds, cold or warm.

## Authoring checks

Four deterministic checks gate startup before policies and routines are created:

| Check                | Validates                                                                                                                                                                |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `schema-valid`       | Existing policy and routine schemas, including inline policies, transition targets, and external-tool instructions. Invalid policies are rejected, not silently skipped. |
| `references-resolve` | Unambiguous tool references, including policy `reevaluate_after` and routine reevaluation tools. Existing manifest prerequisite checks still apply separately.           |
| `steps-reachable`    | Every step is reachable from the routine entry, enforced by the shared routine semantic validator. This checks graph structure, not natural-language feasibility.        |
| `think-ids-unique`   | Think-step IDs are unique across all loaded routines.                                                                                                                    |

Routine-local semantics, including reachability, live in the schemas package and are shared by Pydantic validation at startup and platform semantic validation after JSON Schema validation. The agent does not implement a second graph traversal. Missing entry points, duplicate node IDs, or dangling targets are reported before reachability is attempted.

Tool resolution and cross-routine think-ID uniqueness remain agent-level checks: they require the deployed tool registry or the complete loaded routine set. Policy field validation stays in `PolicySchema`. Neither policy intent nor behavior-flag recommendations become schema constraints; all LLM checks remain agent-level advisory reviews.

**Upgrade warning:** invalid policies, unreachable steps, and cross-routine think-ID collisions can now prevent startup on content that older versions accepted. Once the updated schemas library is deployed on the platform, unreachable routines will also fail authoring validation. Every 1.x schema version is validated by the same rules, whatever version a routine pins; content written for schemas before 9.0 is not supported (see [Content written before schema 9.0](#content-written-before-schema-90)).

Correct these before upgrading. Failures identify the affected content in startup logs and the failed readiness state; these are hard gates, not advisory warnings. Routine schema failures stop config loading before MCP discovery and deployment-wide checks. Reachability-only schema failures use `steps-reachable`; when Pydantic combines different graph-error reasons, the diagnostic uses `schema-valid` and includes every combined message. Startup is fail-fast, not an exhaustive audit of every loaded resource.

After readiness and HTTP binding, each process runs three advisory checks using `llms.evaluation.judge` and its reasoning setting. They are enabled by default. Set `agent_config.runtime.startup_review: false` to opt out of LLM reviews; all four deterministic startup gates still run.

```yaml
agent_config:
  runtime:
    api_key: ${AGENT_API_KEY}
    startup_review: false
```

`runtime.startup_review` replaces `runtime.authoring_review`; rename existing overrides when upgrading to a release with the new name. Older published runtimes continue to use `authoring_review`; it is not an alias in the new schema.

The advisory checks are:

| Check                     | Reviews                                                                                                                                                                           |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `policy-process-boundary` | Whether a policy contains meaningful stages with conditional progression that belong in a routine. Complexity or tool use alone is not a violation.                               |
| `policy-behavior-flags`   | Whether `continuous`, `agent_intention`, and `awaits_customer` match policy intent and effective runtime behavior, including metadata overrides.                                  |
| `step-customer-waiting`   | Whether completing a chat step needs a customer reply, considering rendered instructions and transitions. Missing continuation requires author review, not an invented successor. |

Reviews never change readiness, content, flags, or graphs. They report `no_change`, `recommendation`, `needs_review`, `execution_failure`, or `not_applicable`. Invalid or incomplete judge output is an execution failure, not proof of a content problem. Evidence quotations come from server-owned passages, and proposed flag changes are interpreted by the server.

Findings appear in readable logs and the separate `Authoring validations` trace. Each log message names the check, subject, outcome, and rationale; structured fields retain `detail`, proposed changes, evidence references, and the actual review `trace_id`. Recommendations and ambiguities are warnings; execution failures are errors; no-change findings are debug logs. `Authoring review finished` reports outcome counts and model usage.

The trace has one aggregate span per check, with its judge generations directly underneath. The root input identifies the startup trigger, advisory mode, snapshot hash, review version, checks, judge configuration, and subject counts. Each check input also lists its subjects, aspects, and batch count. Credentials and unrelated runtime configuration are excluded.

Root and check outputs include actionable `findings` grouped by check, separate `execution_failures`, outcome counts, and unfinished-review details. Findings retain rationale, subject, aspect, authored/proposed changes, and evidence references; unchanged assessments contribute counts rather than cluttering the output. Full prompts, evidence quotations, and raw assessments remain on judge generations. Model usage is reported on the root. No endpoint or persistent review store is added.

Enabled reviews run once per process start, including warm boots and every replica, so restarts incur model costs again. Up to four calls run concurrently within a check, with up to five policies per batch and one chat step per batch. Process-boundary reviews finish before flag reviews begin. Each call has a 120-second deadline and the review a 15-minute deadline. Shutdown or budget expiry cancels and joins workers. Unfinished assessments are recorded as execution failures, with one error summary per check rather than one per assessment. Results are not resumed or cached.

There is no prohibition on consecutive chat steps or unconditional transitions.

## What the runtime needs, and where it comes from

| Property                                | Where it comes from                                                                                                                                         | Why the runtime needs it                                                                                                                                        |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Awaits the customer** (routine step)  | `awaits_customer` on a `chat_state` node. Unset = a speaking step the routine continues from (it has transitions) waits; a terminal speaking step does not. | If set, the engine **waits for the customer's reply before advancing** instead of moving on the moment it has spoken — the "ask, then act on the answer" pause. |
| **Continuous** (policy)                 | `continuous: true` on the policy.                                                                                                                           | The matcher keeps a continuous policy active across turns instead of retiring it once its action has been carried out — tone, language, standing prohibitions.  |
| **Agent intention** (policy)            | `agent_intention: true` on the policy; word the condition as "You are likely to …".                                                                         | The matcher evaluates the condition against the agent's *next* move rather than searching the past for it.                                                      |
| **Customer-dependent** (policy)         | `awaits_customer: true` on the policy.                                                                                                                      | The engine keeps the policy in play until the customer has responded to what the action asked for.                                                              |
| **Reachable follow-ups** (routine node) | Derived from the node's outgoing `transitions` and their conditions, children before parents.                                                               | The per-turn navigator chooses the next node from this table instead of re-analysing the graph every turn.                                                      |

Every condition in the routing table is composed from your transition condition and node action **exactly as you wrote them** — so write each step to stand alone ("send the failed ID verification result to the manual-review queue", not "send it to review") and list the options a condition depends on. Nothing rewrites a dangling reference for you.

`tool_instruction` is optional guidance for filling a node's tool arguments; the tools run whenever the node is reached. The validator only rejects a `tool_instruction` on a node without `tools`. Conditions for running a tool belong on the transitions that lead to its node.

## Caching: cold vs. warm boots

Routing tables are **keyed by content hash** — a routine by its id, the ordered ids of its nodes and transitions, and the engine's derivation version. At boot the engine looks each routine up by hash:

* **Hit** → the metadata loads instantly, with no model calls.
* **Miss** → the routing table is derived live (a graph walk, no model) and written back to the cache.

The cache uses the configured agent PostgreSQL database, or ephemeral memory when no database is configured. Only PostgreSQL preserves these rows across restarts.

Because the key is the content, results stay valid until the content changes, and editing one routine invalidates only that routine.

**The platform warms the cache as part of deploying content**; a cold boot derives the tables live in seconds, so a missed warm-up costs little.

### What invalidates the cache

The content hash makes the rules simple:

* **No effect** (cache stays valid): runtime upgrades, manifest tuning-knob changes, secret rotations — none change content hashes.
* **Invalidated automatically** (new hash → re-derived on next deploy): any edit to a routine's nodes / transitions.
* **Invalidated by a runtime upgrade only when the derivation changed:** the routine key also carries the engine's evaluation-algorithm version, so an upgrade that changes how the routing table is composed re-derives every routine once, while other upgrades keep the cache.

## Tuning live evaluation

Routing-table derivation has no model to tune and completes in seconds. The separate advisory review uses `llms.evaluation.judge`; it does not delay readiness. Remove the obsolete `llms.evaluation.startup` setting from existing manifests before upgrading; the schema no longer accepts it.

## Content written before schema 9.0

The 1.x line validates every routine with the rules introduced in schema 9.0, so routines written for an earlier schema must be updated before they are pinned to a 1.x version. Startup does not ask a model whether a step needs a customer reply: routines without `awaits_customer` treat every non-terminal `chat_state` node as waiting for a reply, and terminal chat nodes default to not waiting. This also changes look-ahead routing conditions, not just the immediate pause.

Audit mid-flow informational steps before moving such content to 1.x. A step that explains something and should continue without a reply needs `awaits_customer: false`:

```yaml
- id: explain
  chat_state: Explain that you will check the withdrawal status.
  awaits_customer: false
  transitions:
    - to: check_status
```

Keep `awaits_customer: true` on steps that ask a question or otherwise require an answer. This flag also exists on standalone and scoped policies, where its default is `false`. Content is not automatically rewritten. Publish the updated content, then update the agent's version pins before upgrading the runtime.

## Changing derivation logic

Engine and eval-tool share transition wording but maintain separate graph walks. Changes to wording **or traversal** (depth limits, cycle handling, forks) require bumping `EVAL_ALGORITHM_VERSION` for cache invalidation and `graph_build_version` in the routine service so already-ready graphs are rebuilt. Unchanged content can then reuse the newly published graphs on later boots. A runtime image version bump alone does not invalidate these graphs.

## Watching it

Evaluation emits one log line per routine — `Routine '<title>' evaluated: N nodes in Xs` (`N=0` means it was served from cache) — and one summary line, `[eval] startup evaluation summary: …`, with the routine and node counts, cache hits and elapsed time. See [Observability](https://docs.interactive.ai/agents/guides/observability.md#boot-time-evaluation-logs).

## See also

* [Conversation lifecycle](https://docs.interactive.ai/agents/concepts/conversation-lifecycle.md) — how a turn consumes this metadata
* [Routines](https://docs.interactive.ai/agents/concepts/routines.md) and [Policies](https://docs.interactive.ai/agents/concepts/policies.md) — the inputs being evaluated
* [Architecture](https://docs.interactive.ai/agents/concepts/architecture.md#boot-sequence) — where evaluation sits in boot


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.interactive.ai/agents/concepts/startup-evaluation.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
