Shared routine graph lifecycle
Immutable shared routine graphs, replica initialization, recovery, and maintenance-only cleanup of obsolete versions.
Routine graphs in PostgreSQL are shared by replicas. A ready graph is immutable: starting another replica must not replace its nodes or remove its evaluated transition metadata.
Initialization
One PostgreSQL advisory lock per database and agent covers configuration application, startup evaluation, and persistence of evaluation results. Other replicas of that agent wait; different agents can initialize independently. The lock uses a dedicated connection and is released on completion or failure. Loss of that connection cancels initialization. Transient/local stores do not need a cross-replica lock.
A graph's content-based ID includes its authored nodes, entry, conditions, scoped policies, rendered actions (including macro expansion), resolved tool bindings, prompt version, and graph build version. Bump graph_build_version in config/services/routines.py when construction or evaluation semantics change. That produces new graph/cache identities instead of modifying published data.
The root node carries graph_state:
building: private initialization state. Routine discovery excludes it.ready: evaluation finished, its metadata was persisted, and every projected executable/nonterminal node has a reachability list. Empty lists are valid for terminal steps; missing evaluation metadata is not.
An unchanged ready graph is reused without node or edge writes. Discovery remembers ready graph IDs locally, avoiding repeated root reads; it does not cache building states. Each replica still reconstructs its local policies, tool relationships, reason-step schemas, and autonomous registrations. A changed graph is built under a different ID; old replicas can continue reading the previous version during a normal rollout.
An interrupted/unmarked graph is deleted and reconstructed under the startup lock before publication. Ready graphs are never reset as part of recovery. Evaluation failure prevents readiness. Existing in-flight work is not migrated between graph versions; drain it before retiring its deployment.
First rollout to an existing database
Old runtimes do not honor the initialization lock and can still overwrite shared metadata or run timed cleanup. Do not mix those runtimes with this lifecycle. Perform the first rollout per agent database during a maintenance window:
Back up the database. Pause incoming work and scheduled/autonomous triggers.
Disable autoscaling/restarts of the old deployment, drain in-flight work, and stop every old replica sharing this agent database.
Start one new-runtime replica with the unchanged pinned configuration.
Wait for successful evaluation and readiness; run an appropriate smoke check.
Restore the desired replicas and resume work. Remove the old deployment so it cannot subsequently restart against the database.
Customer data and sessions are not deleted by graph initialization. New graph identities may require a cold evaluation even if older evaluation cache entries exist. Measure that time before scheduling the maintenance window and allow sufficient startup-probe time; readiness now waits for graph publication.
Explicit cleanup
Normal boot and scale-out no longer schedule stale-store deletion. STORE_GC_DELAY_SECONDS has been removed; setting it does not enable cleanup. Obsolete graphs stay stored but cannot activate on a replica whose local condition policies belong to a different graph version.
For cleanup, pause and drain all work and stop all other replicas first. Start one replica with the desired configuration and the additional server flag:
After successful initialization, and before serving, it invokes the existing agent-scoped store sweep. This deletes obsolete routine and glossary versions that predate this boot and are absent from the current deployment inventory. Routine deletion also removes its nodes, edges, associations, and vector entry. Customer/session data and evaluation caches are not included in this sweep.
Alternatively, once the single maintenance replica is ready, trigger the same sweep without restarting it:
No request body or startup flag is required for the endpoint. It uses this replica's deployment inventory and acquires the same agent initialization lock. It returns 200 with counts for completed store sweeps:
A store-sweep failure returns 500 with the failed store names in errors. Other stores are still swept. A failed store may already have deleted some rows; its count is omitted, not reported as zero. Inspect logs and retry during maintenance if necessary. Repeating a completed sweep is safe.
The endpoint rejects requests before readiness or without a deployment inventory (503), and rejects non-PostgreSQL storage (409). It uses existing agent bearer/cookie authentication, not a separate operator role: restrict access to trusted maintenance callers. Normal startup still performs no automatic cleanup.
Neither the CLI flag nor the endpoint discovers or stops other replicas for you. The startup lock only excludes other initializers; it cannot protect a serving old replica or in-flight job from operator-requested deletion. A row's age is not evidence that it is unused. Check cleanup logs for failures before ending maintenance.
Remove the maintenance flag from the normal deployment command before restoring replicas/autoscaling. Retain a backup if rollback matters. A rollback to an older configuration on this lifecycle can reuse its retained graph, or rebuild it if cleaned up; a rollback to the pre-lifecycle runtime requires maintenance again.
Last updated
Was this helpful?

