Observability

Traces, structured logs, trace naming with your business identifiers, the retry-fallback signals, and the grep recipes that answer "what did the agent do?".

Context — Every turn an agent takes is traced (OpenTelemetry) and logged (structured JSON). This guide shows how to read both, how to make traces searchable by your identifiers, and which signals deserve alerts.

YAML examples follow manifest schema 6.1.5. Manifest and content shapes are schema-versioned and differ across runtime versions — see Versioning & compatibility.

Traces

The agent exports OpenTelemetry traces over OTLP/HTTP. By default they go to the InteractiveAI platform's traces backend (derived from interactive_platform.base_url, authenticated with the platform keys) and appear in the platform's Traces view. A custom backend is one manifest block away:

agent_config:
  traces:
    deployment_environment: production
    backend:
      url: https://otel.your-provider.com/v1/traces
      api_key: ${OTEL_API_KEY}
      api_key_scheme: bearer    # or "basic" for public:secret key pairs

deployment_environment (default production) tags every trace with deployment.environment — it's the environment filter in the trace UI, so staging and production agents with the same name stay separable.

What a trace contains

One trace per turn (conversational) or per run (autonomous). Inside it: policy matching batches, routine evaluation decisions, every model call (chat and evaluation lanes), every tool call with arguments and results, knowledge-base retrievals with the rewritten query, and the final reply. Each turn's trace also carries a metadata snapshot: the session metadata verbatim and the resolved context variables exactly as fed to the model — so "what did the agent know?" is answerable months later.

The trace's input is the customer message; its output is the turn's reply messages (or the autonomous run's typed output).

Trace naming

Traces group and name themselves off one resolved resource id, in precedence order:

1. Autonomous runs — the value of the input field named by agent_config.traces.trace_id_field. Set it to your business key:

   agent_config:
     traces:
       trace_id_field: customer_id

   A run triggered with {"input": {"customer_id": "cus_abc", ...}} then traces as {agent}-cus_abc instead of a synthetic run id.

2. Conversational sessions — session.metadata["session_key"], an optional key your integration sets when opening the session. Session ids are opaque server-generated hashes; session_key is how you attach a stable, human-meaningful identifier (a ticket id, a case number):

   sess = await client.sessions.open(
       id="zendesk-42",
       session_key="TICKET-7841",
   )

   On a session that already exists, set it after the fact instead:

   await sess.update_metadata({"session_key": "TICKET-7841"})

   Every turn of that conversation then groups under TICKET-7841 in the trace UI's session view, named {agent}-TICKET-7841, with the user-id column filterable by the same key. The key must be exactly session_key.

3. Fallback — the first 8 characters of the session id, so grouping always works.

The resolved resource id is truncated to 256 characters before it's used in any of the naming above, so an oversized business key can't blow out trace names.

The autonomous callback's trace_id field links a delivered result back to its trace directly.

Logs

Everything the process emits — agent, HTTP server, engine — is single-line JSON on stdout:

{"timestamp": "2026-06-04T10:15:02.114Z", "level": "info", "message": "Received event from session_id: sess_abc for routine_id: kyc-decision", "logger": "agent_server.endpoints.autonomous_routines", "request_id": "8f2a…", "session_id": "sess_abc", "routine_id": "kyc-decision", "run_id": "run_9f8e…"}

Conventions worth knowing when querying:

Field	Meaning
timestamp, level, message, logger	Always present.
request_id, method, path	Bound for every HTTP request.
session_id, trace_id	Bound during engine turns.
duration_ms / duration_s	Timings — milliseconds generally, seconds for evaluation-phase logs.
phase	Marks special subsystems: eval (boot-time evaluation), retry-fallback (model escalation).

The manifest's runtime.log_level (default INFO) drives verbosity; DEBUG adds per-decision detail. Health-probe and event-polling requests are excluded from access logs by design.

Boot-time evaluation logs

Cold-cache routine evaluation is the noisy phase. At INFO you get one bookend per routine:

Routine 'Book A Car' evaluated: 11 nodes in 154.2s

(0 nodes = served from cache.) At DEBUG, per-stage and per-step lines appear, all tagged phase="eval" and message-prefixed [eval]. Failures log at WARNING/ERROR regardless of level.

# everything eval-related, one agent, last 30 minutes
iai agents logs <agent> --since 30m | grep '\[eval\]'

# structured: every eval line via the phase field
iai agents logs <agent> --since 30m | jq 'select(.phase == "eval")'

Slow boots → Startup evaluation.

Retry-fallback signals

When an evaluation call exhausts its 3 attempts and escalates to evaluation_fallback (see Models), the runtime logs it with a [retry-fallback] message marker and phase="retry-fallback", including which call site escalated and both model names — kick-in and success at WARNING, both-models-exhausted at ERROR.

iai agents logs <agent> --since 30m | grep '\[retry-fallback\]'

What to alert on:

Signal	Meaning	Action
Occasional [retry-fallback] … succeeded	Normal — the safety net working	None
Sustained escalation rate	Evaluation primary struggling with your content	Simplify conditions or promote llms.evaluation
[retry-fallback] at ERROR (both exhausted)	A turn failed an internal decision	Investigate the trace; check router health
[eval] … FAILED	A routine failed boot-time evaluation	Fix the routine; the boot log names it
Retrieval warnings	Knowledge base unreachable/misbehaving — turns proceed ungrounded	Check KB health; answers degrade silently otherwise

A debugging workflow

"The agent did something odd in ticket 7841":

1. Find the session in the trace UI by TICKET-7841 (you set session_key, right?). All its turns are grouped.

2. Open the odd turn's trace. Check, in order: which policies matched (and which surprisingly didn't), which routine/node was selected, what each tool returned, what the KB retrieval contributed.

3. Check the config snapshot on the trace — it records which policy and content versions were live, so "did yesterday's content release cause this?" is a lookup, not an archaeology dig.

4. Correlate logs by the trace id (trace_id field) for anything infrastructural (timeouts, reconnects, escalations).

Symptom-indexed problems live in Troubleshooting.