> For the complete documentation index, see [llms.txt](https://docs.interactive.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.interactive.ai/agents/guides/observability.md).

# Observability

Traces, structured logs, trace naming with your business identifiers, the retry-fallback signals, and the grep recipes that answer "what did the agent do?".

> **Context** — Every turn an agent takes is traced (OpenTelemetry) and logged (structured JSON). This guide shows how to read both, how to make traces searchable by *your* identifiers, and which signals deserve alerts.
>
> YAML examples follow **manifest schema 1.4.0**. Manifest and content shapes are schema-versioned and differ across runtime versions — see [Versioning & compatibility](https://docs.interactive.ai/agents/operations/versioning.md).

## Traces

The agent exports OpenTelemetry traces over OTLP/HTTP. By default they go to the InteractiveAI platform's traces backend (derived from `interactive_platform.base_url`, authenticated with the platform keys) and appear in the platform's Traces view. A custom backend is one manifest block away:

```yaml
agent_config:
  traces:
    deployment_environment: production
    backend:
      url: https://otel.your-provider.com/v1/traces
      api_key: ${OTEL_API_KEY}
      api_key_scheme: bearer    # or "basic" for public:secret key pairs
```

`deployment_environment` (default `production`) tags every trace with `deployment.environment` — it's the environment filter in the trace UI, so staging and production agents with the same name stay separable.

### What a trace contains

One trace per turn (conversational) or per run (autonomous). Inside it: policy matching batches, routine evaluation decisions, every model call (chat and evaluation lanes), every tool call with arguments and results, knowledge-base retrievals with the rewritten query, and the final reply. Each turn's trace also carries a metadata snapshot: the session metadata verbatim and the resolved context variables exactly as fed to the model — so "what did the agent know?" is answerable months later.

The trace's **input** is the customer message; its **output** is the turn's reply messages (or the autonomous run's typed output).

### Thinking traces

Think nodes appear under one `Execute: Thinking` chain per tool phase, directly under the iteration. Concurrent think calls are siblings:

```
Iteration: 3
  Execute: Thinking
    Think: classify_withdrawal
    Think: check_eligibility
    Response: Tool announcement
```

Each generation child contains its actual prompt, model output, model, tokens, and cost. Retries and fallback remain separate real calls: a retry is named with its attempt (`Think: classify_withdrawal (attempt 2/3)`), and a call the evaluation fallback model made with `(fallback)`. A reply the step rejects, because it doesn't fit the step's `output`, marks its call as a warning that names the field and the rule (`reply rejected: passages/1/excerpt: 406 characters, maximum 400`); the `Execute: Thinking` chain then warns too, listing the steps whose replies were rejected; if no try is left, the chain fails instead. The parent input lists the routine and step IDs; its output maps step IDs to the exact values successfully saved in session state. There is no additional synthetic `Thinking` tool child. Validation or persistence failures mark the parent as failed and identify the phase and step. Results not saved (for example, calls filtered out by the execution plan) are recorded as `think.not_saved`, not included in the saved output. When a new customer message restarts the turn mid-think, the parent is not marked as failed: it records `think.cancelled` with the reason, plus `think.not_saved`.

The announcement child exists only when its model is called, including calls that return an empty message. It stays asynchronous: the parent ends after the announcement finishes, without blocking the next engine work. Announcement failure/cancellation is recorded separately and does not invalidate saved think outputs. The parent duration therefore includes announcement time, not just model inference. Ordinary tools keep their existing evaluation/execution spans; a single announcement covering a mixed think/tool batch is attached to `Execute: Thinking` without making an additional announcement call.

### Computing traces

Compute nodes appear the same way, under one `Execute: Computing` chain per tool phase, directly under the iteration:

```
Iteration: 3
  Route: check → prepare (rule 1)
  Execute: Computing
    Compute: prepare
```

Each `Compute:` child shows the values its formulas read as input and the values they worked out as output; no model call sits under it. The parent input lists the routine and step IDs, and its output maps step IDs to the values saved. A value with nothing to read (a missing field, which reads as empty) marks the child as a warning, naming it. A value empty because the step it reads failed, or absent because the step's `output` declares it optional, is listed in the child's `notes`, with no warning: a failed step's own span already shows the failure. A tool's span does the same for a formula argument whose `default()` stood in for a value with nothing to read: a warning naming it, unless the call failed, or a note for a failed step or an optional field. A formula that fails marks the child and the parent as failed, with the value and the reason. When an iteration only thinks and computes, it shows no `Evaluate: Tools` or `Execute: Tools` chains; ordinary tools in the same iteration keep theirs. `Evaluate: Tools` appears only when a model prepared arguments, a request failed or customer data was missing: a call whose arguments are all bound shows under `Execute: Tools` alone.

### Routing and parallel steps

A routine step reached with no model call shows how it was reached, in a `Route:` chain at the top of its iteration:

```
Iteration: 4
  Route: analyse → judge (rule 1)
  Parallel: judge (3 steps)
    Think: judge_identification
    Think: judge_valuation
    Think: judge_accountability
```

`Route: <step left> → <step> (<why>)` gives the reason as `rule N` (the first transition rule that held), `otherwise`, `single transition` (the step's only way on) or `entry`. Its input is the step left's transitions, as written, and the values its rules read; its output is the step chosen. A value a rule found nothing to read marks it as a warning, naming the path; a value of the wrong type for its operator, as an error. A step the model chose keeps an `Evaluate: Routine steps` chain, with the model call under it.

A parallel step's members run under one `Parallel: <step>` chain, in place of the `Execute: Thinking`, `Execute: Computing` or `Execute: Tools` chain their kind would have. When its members are of several kinds, each chain adds its kind (`Parallel: read_pdfs · tools`), and a tool member's call adds its step (`read_pdf · read_first`). A retry gives its attempt and how many members ran again (`Parallel: analyse (attempt 2 · 1 of 4 steps)`); the chain's metadata lists in `parallel.kept` the members that kept their earlier result.

### Trace naming

Traces group and name themselves off one resolved **resource id**, in precedence order:

1. **Autonomous runs** — the value of the input field named by `agent_config.traces.trace_id_field`. Set it to your business key:

   ```yaml
   agent_config:
     traces:
       trace_id_field: customer_id
   ```

   A run triggered with `{"input": {"customer_id": "cus_abc", ...}}` then traces as `{agent}-cus_abc` instead of a synthetic run id.
2. **Conversational sessions** — `session.metadata["session_key"]`, an optional key your integration sets when opening the session. Session ids are opaque server-generated hashes; `session_key` is how you attach a stable, human-meaningful identifier (a ticket id, a case number):

   ```python
   sess = await client.sessions.open(
       id="zendesk-42",
       session_key="TICKET-7841",
   )
   ```

   On a session that already exists, set it after the fact instead:

   ```python
   await sess.update_metadata({"session_key": "TICKET-7841"})
   ```

   Every turn of that conversation then groups under `TICKET-7841` in the trace UI's session view, named `{agent}-TICKET-7841`, with the user-id column filterable by the same key. The key must be exactly `session_key`.
3. **Fallback** — the first 8 characters of the session id, so grouping always works.

The resolved resource id is truncated to 256 characters before it's used in any of the naming above, so an oversized business key can't blow out trace names.

The autonomous callback's `trace_id` field links a delivered result back to its trace directly.

## Logs

Everything the process emits — agent, HTTP server, engine — is single-line JSON on stdout:

```json
{"timestamp": "2026-06-04T10:15:02.114Z", "level": "info", "message": "Received event from session_id: sess_abc for routine_id: kyc-decision", "logger": "agent.endpoints.autonomous_routines", "request_id": "8f2a…", "session_id": "sess_abc", "routine_id": "kyc-decision", "run_id": "run_9f8e…"}
```

Conventions worth knowing when querying:

| Field                                     | Meaning                                                                                       |
| ----------------------------------------- | --------------------------------------------------------------------------------------------- |
| `timestamp`, `level`, `message`, `logger` | Always present.                                                                               |
| `request_id`, `method`, `path`            | Bound for every HTTP request.                                                                 |
| `session_id`, `trace_id`                  | Bound during engine turns.                                                                    |
| `duration_ms` / `duration_s`              | Timings — milliseconds generally, **seconds** for evaluation-phase logs.                      |
| `phase`                                   | Marks special subsystems: `eval` (boot-time evaluation), `retry-fallback` (model escalation). |

The manifest's `runtime.log_level` (default `INFO`) drives verbosity; `DEBUG` adds per-decision detail. Health-probe and event-polling requests are excluded from access logs by design.

## Boot-time evaluation logs

Cold-cache routine evaluation is the noisy phase. At `INFO` you get one bookend per routine:

```
Routine 'Book A Car' evaluated: 11 nodes in 154.2s
```

(`0 nodes` = served from cache.) At `DEBUG`, per-stage and per-step lines appear, all tagged `phase="eval"` and message-prefixed `[eval]`. Failures log at WARNING/ERROR regardless of level.

```bash
# everything eval-related, one agent, last 30 minutes
iai agents logs <agent> --since 30m | grep '\[eval\]'

# structured: every eval line via the phase field
iai agents logs <agent> --since 30m | jq 'select(.phase == "eval")'
```

Slow boots → [Startup evaluation](https://docs.interactive.ai/agents/concepts/startup-evaluation.md#caching-cold-vs-warm-boots).

## Retry-fallback signals

When the reply or an evaluation call exhausts its 3 attempts and switches to a fallback model (`llms.fallback` for the reply, `llms.evaluation.fallback` for evaluation; see [Models](https://docs.interactive.ai/agents/concepts/models.md)), the runtime logs it with a `[retry-fallback]` message marker, `phase="retry-fallback"` and `lane="response"` or `lane="evaluation"`, including both model names — failed reply attempts, kick-in and success at WARNING, every model exhausted at ERROR.

```bash
iai agents logs <agent> --since 30m | grep '\[retry-fallback\]'
```

What to alert on:

| Signal                                             | Meaning                                                           | Action                                              |
| -------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------- |
| Occasional `[retry-fallback] … succeeded`          | Normal — the safety net working                                   | None                                                |
| Sustained escalation rate                          | Evaluation primary struggling with your content                   | Simplify conditions or promote `llms.evaluation`    |
| `[retry-fallback]` at ERROR (all models exhausted) | A turn failed its reply or an internal decision                   | Investigate the trace; check router health          |
| `[eval] … FAILED`                                  | A routine failed boot-time evaluation                             | Fix the routine; the boot log names it              |
| Retrieval warnings                                 | Knowledge base unreachable/misbehaving — turns proceed ungrounded | Check KB health; answers degrade silently otherwise |

## A debugging workflow

"The agent did something odd in ticket 7841":

1. **Find the session** in the trace UI by `TICKET-7841` (you set `session_key`, right?). All its turns are grouped.
2. **Open the odd turn's trace.** Check, in order: which policies matched (and which surprisingly didn't), which routine/node was selected, what each tool returned, what the KB retrieval contributed.
3. **Check the config snapshot** on the trace — it records which policy and content versions were live, so "did yesterday's content release cause this?" is a lookup, not an archaeology dig.
4. **Correlate logs** by the trace id (`trace_id` field) for anything infrastructural (timeouts, reconnects, escalations).

Symptom-indexed problems live in [Troubleshooting](https://docs.interactive.ai/agents/operations/troubleshooting.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.interactive.ai/agents/guides/observability.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
