Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Startup evaluation

What the engine derives about your routines at boot, which behavioural flags you author instead, and how the routing table is cached.

Context — Before an agent serves traffic, the engine derives each routine's routing table and reads the behavioural flags you authored. This page explains what the runtime needs, where each fact comes from, and how the routing table is cached. It assumes Routines (nodes, transitions) and Policies; watching it run is in Observability.

YAML examples follow manifest schema 9.10.0. Manifest and content shapes are schema-versioned and differ across runtime versions — see Versioning & compatibility.

Why evaluation exists

Your routines and policies are written in natural language. To run a turn quickly and consistently, the engine needs a few structured facts about them: does this step wait for the customer's reply? does this policy stay in force every turn? which node can come next, and when?

Those facts used to be worked out by a model at every cold boot and cached. Two boots of identical content did not agree on them, and the boot took minutes and cost money. They are now authored or derived: the flags are fields you set (with sensible defaults from the routine's shape), and the routing table is computed from your transitions — no model is needed to become ready. An advisory authoring review runs separately after the HTTP listener binds. What remains of graph "evaluation" is the routing table, derived once per routine and cached because the runtime reads it from node metadata.

Where it sits in boot

Startup evaluation is the last phase of the boot sequence, and it runs before the agent begins serving: the HTTP port binds and health checks start passing only once it settles. It is a graph walk over your routines and completes in seconds, cold or warm.

Authoring checks

Four deterministic checks gate startup before policies and routines are created:

Check
Validates

schema-valid

Existing policy and routine schemas, including inline policies, transition targets, and external-tool instructions. Invalid policies are rejected, not silently skipped.

references-resolve

Unambiguous tool references, including policy reevaluate_after and routine reevaluation tools. Existing manifest prerequisite checks still apply separately.

steps-reachable

Every step is reachable from the routine entry, enforced by the shared routine semantic validator. This checks graph structure, not natural-language feasibility.

think-ids-unique

Think-step IDs are unique across all loaded routines.

Routine-local semantics, including reachability, live in the schemas package and are shared by Pydantic validation at startup and platform semantic validation after JSON Schema validation. The agent does not implement a second graph traversal. Missing entry points, duplicate node IDs, or dangling targets are reported before reachability is attempted.

Tool resolution and cross-routine think-ID uniqueness remain agent-level checks: they require the deployed tool registry or the complete loaded routine set. Policy field validation stays in PolicySchema. Neither policy intent nor behavior-flag recommendations become schema constraints; all LLM checks remain agent-level advisory reviews.

Upgrade warning: invalid policies, unreachable steps, and cross-routine think-ID collisions can now prevent startup on content that older versions accepted. Once the updated schemas library is deployed on the platform, unreachable routines will also fail authoring validation. Semantic dispatch is by schema major: reachability applies to the current graph rules (9+), including existing schema-9 pins; the frozen pre-9 rules remain unchanged.

Correct these before upgrading. Failures identify the affected content in startup logs and the failed readiness state; these are hard gates, not advisory warnings. Routine schema failures stop config loading before MCP discovery and deployment-wide checks. Reachability-only schema failures use steps-reachable; when Pydantic combines different graph-error reasons, the diagnostic uses schema-valid and includes every combined message. Startup is fail-fast, not an exhaustive audit of every loaded resource.

After readiness and HTTP binding, each process runs three advisory checks using llms.evaluation.judge and its reasoning setting. They are enabled by default. Set agent_config.runtime.startup_review: false to opt out of LLM reviews; all four deterministic startup gates still run.

runtime.startup_review replaces runtime.authoring_review; rename existing overrides when upgrading to a release with the new name. Older published runtimes continue to use authoring_review; it is not an alias in the new schema.

The advisory checks are:

Check
Reviews

policy-process-boundary

Whether a policy contains meaningful stages with conditional progression that belong in a routine. Complexity or tool use alone is not a violation.

policy-behavior-flags

Whether continuous, agent_intention, and awaits_customer match policy intent and effective runtime behavior, including metadata overrides.

step-customer-waiting

Whether completing a chat step needs a customer reply, considering rendered instructions and transitions. Missing continuation requires author review, not an invented successor.

Reviews never change readiness, content, flags, or graphs. They report no_change, recommendation, needs_review, execution_failure, or not_applicable. Invalid or incomplete judge output is an execution failure, not proof of a content problem. Evidence quotations come from server-owned passages, and proposed flag changes are interpreted by the server.

Findings appear in readable logs and the separate Authoring validations trace. Each log message names the check, subject, outcome, and rationale; structured fields retain detail, proposed changes, evidence references, and the actual review trace_id. Recommendations and ambiguities are warnings; execution failures are errors; no-change findings are debug logs. Authoring review finished reports outcome counts and model usage.

The trace has one aggregate span per check, with its judge generations directly underneath. The root input identifies the startup trigger, advisory mode, snapshot hash, review version, checks, judge configuration, and subject counts. Each check input also lists its subjects, aspects, and batch count. Credentials and unrelated runtime configuration are excluded.

Root and check outputs include actionable findings grouped by check, separate execution_failures, outcome counts, and unfinished-review details. Findings retain rationale, subject, aspect, authored/proposed changes, and evidence references; unchanged assessments contribute counts rather than cluttering the output. Full prompts, evidence quotations, and raw assessments remain on judge generations. Model usage is reported on the root. No endpoint or persistent review store is added.

Enabled reviews run once per process start, including warm boots and every replica, so restarts incur model costs again. Up to four calls run concurrently within a check, with up to five policies per batch and one chat step per batch. Process-boundary reviews finish before flag reviews begin. Each call has a 120-second deadline and the review a 15-minute deadline. Shutdown or budget expiry cancels and joins workers. Unfinished assessments are recorded as execution failures, with one error summary per check rather than one per assessment. Results are not resumed or cached.

There is no prohibition on consecutive chat steps or unconditional transitions.

What the runtime needs, and where it comes from

Property
Where it comes from
Why the runtime needs it

Awaits the customer (routine step)

awaits_customer on a chat_state node. Unset = a speaking step the routine continues from (it has transitions) waits; a terminal speaking step does not.

If set, the engine waits for the customer's reply before advancing instead of moving on the moment it has spoken — the "ask, then act on the answer" pause.

Continuous (policy)

continuous: true on the policy.

The matcher keeps a continuous policy active across turns instead of retiring it once its action has been carried out — tone, language, standing prohibitions.

Agent intention (policy)

agent_intention: true on the policy; word the condition as "You are likely to …".

The matcher evaluates the condition against the agent's next move rather than searching the past for it.

Customer-dependent (policy)

awaits_customer: true on the policy.

The engine keeps the policy in play until the customer has responded to what the action asked for.

Reachable follow-ups (routine node)

Derived from the node's outgoing transitions and their conditions, children before parents.

The per-turn navigator chooses the next node from this table instead of re-analysing the graph every turn.

Every condition in the routing table is composed from your transition condition and node action exactly as you wrote them — so write each step to stand alone ("send the failed ID verification result to the manual-review queue", not "send it to review") and list the options a condition depends on. Nothing rewrites a dangling reference for you.

A node that calls an external tool must say how in tool_instruction; the validator rejects one that does not. Built-in tools (built-in:emit_output, built-in:initiate_human_handoff) carry their own contract and need none.

Caching: cold vs. warm boots

Routing tables are keyed by content hash — a routine by its id, the ordered ids of its nodes and transitions, and the engine's derivation version. At boot the engine looks each routine up by hash:

  • Hit → the metadata loads instantly, with no model calls.

  • Miss → the routing table is derived live (a graph walk, no model) and written back to the cache.

The cache uses the configured agent PostgreSQL database, or ephemeral memory when no database is configured. Only PostgreSQL preserves these rows across restarts.

Because the key is the content, results stay valid until the content changes, and editing one routine invalidates only that routine.

The platform warms the cache as part of deploying content; a cold boot derives the tables live in seconds, so a missed warm-up costs little.

What invalidates the cache

The content hash makes the rules simple:

  • No effect (cache stays valid): runtime upgrades, manifest tuning-knob changes, secret rotations — none change content hashes.

  • Invalidated automatically (new hash → re-derived on next deploy): any edit to a routine's nodes / transitions.

  • Invalidated by a runtime upgrade only when the derivation changed: the routine key also carries the engine's evaluation-algorithm version, so an upgrade that changes how the routing table is composed re-derives every routine once, while other upgrades keep the cache.

Tuning live evaluation

Routing-table derivation has no model to tune and completes in seconds. The separate advisory review uses llms.evaluation.judge; it does not delay readiness. Remove the obsolete llms.evaluation.startup setting from existing manifests before upgrading; the schema no longer accepts it.

Migrating existing content to schema 9.0

Startup no longer asks a model whether a step needs a customer reply. Existing routines without awaits_customer now treat every non-terminal chat_state node as waiting for a reply; terminal chat nodes default to not waiting. This also changes look-ahead routing conditions, not just the immediate pause.

Audit mid-flow informational steps before upgrading. A step that explains something and should continue without a reply needs awaits_customer: false:

Keep awaits_customer: true on steps that ask a question or otherwise require an answer. This flag also exists on standalone and scoped policies, where its default is false. Content is not automatically rewritten during migration.

Every node calling an external tool now requires a non-empty tool_instruction explaining how to call it. Built-in tools may omit it; use think: for reasoning nodes. Audit and publish any required content changes, then update the agent's version pins before upgrading the runtime.

Changing derivation logic

Engine and eval-tool share transition wording but maintain separate graph walks. Changes to wording or traversal (depth limits, cycle handling, forks) require bumping EVAL_ALGORITHM_VERSION for cache invalidation and graph_build_version in the routine service so already-ready graphs are rebuilt. Unchanged content can then reuse the newly published graphs on later boots. A runtime image version bump alone does not invalidate these graphs.

Watching it

Evaluation emits one log line per routine — Routine '<title>' evaluated: N nodes in Xs (N=0 means it was served from cache) — and one summary line, [eval] startup evaluation summary: …, with the routine and node counts, cache hits and elapsed time. See Observability.

See also

Last updated

Was this helpful?