A 1-second submit hid a 30-minute outage

At 15:50 UTC on August 9, Deixic accepted an AI turn in about one second. The sandbox was ready 22 seconds later. The first assistant chunks reached durable storage after 105.8 seconds, and a tool proposal arrived after 146.6 seconds.

The turn never produced a terminal assistant event.

During the next 30 minutes, the runtime claimed the work 95 times. The model produced 109 response chunks and one tool proposal. Recovery eventually started, hit a snapshot fork rejection, and left the next turn attached to a failed replacement session.

From the product, this looked like one outage: Dex was stuck. In the system, it was a chain of distinct defects across scheduling, state projection, streaming, snapshot recovery, and session lineage. The user-facing failure began after both the submit endpoint and sandbox had reported success.

We reconstructed the incident with a bounded, read-only agent investigation that separated source behavior, production evidence, and release state.

The first plausible cause was real and incomplete#

Earlier that day, several requests failed before a runner session existed. Repairs aligned the prewarm and runner contracts, retained cleanup ownership across transfers, reclaimed expired submit leases, stabilized resident identity, and stopped permanent admission failures from retrying. Those changes explained the early failures, but this later turn had a ready sandbox, streamed output, and a tool proposal.

What the investigation method changed#

The investigation could inspect source, Git history, production logs, database records, and release evidence through existing read access. Its operating boundary prohibited code edits, workload restarts, job reruns, cloud changes, and deployments.

The work ran in three parallel evidence lanes:

1. Reconstruct the runtime state machine from source and recent changes.
2. Trace the production request through the worker, runner, model runtime, and sandbox provider.
3. Audit merge, image publication, deployment, and live workload evidence as separate release states.

One coordinating thread compared the three reports against the user timeline. Smaller threads tested bounded hypotheses about draft ownership, request-path divergence, runner provisioning, release admission, replay disconnects, and snapshot readiness.

This structure produced four concrete benefits:

• The state machine, live request, and release chain were inspected at the same time.
• The read-only boundary kept evidence collection separate from production changes.
• The coordinating thread rejected explanations that failed to account for the timestamped events.
• The release lane prevented merged code from being mistaken for deployed behavior.

That release lane tracked five separate facts: merge, image publication, GitOps update, applied pod digest, and successful user turn. During this incident, one runner fix was merged and its image was published and signed. The deployment pipeline rejected it because the source branch had already advanced. Production was still running the older digest.

How the failures compounded#

Failure propagation · one Deixic turn

Five runtime boundaries formed one recovery loop

Five layers of failure#

1. Retry ordering starved fresh work#

The runtime controller had one replica and processed work serially. Its batch setting admitted four rows per tick, but due retries received priority. A not-ready result scheduled another retry and ended the current drain after one claim.

The original turn had already received its first claim. After its retry loop formed, fresh thread heads stopped receiving service.

During a six-minute production window, the worker processed 38 claims across 13 older work items. Fresh canary turns received zero claims. Both canary profiles timed out after roughly 92 seconds while still queued.

The scheduler was operating according to its ordering rules. Those rules allowed a sustained retry set to consume every execution opportunity.

2. Two tables disagreed about runner readiness#

The runner stored successful provisioning in the runtime binding and emitted a provisioned event. The owning runner session row remained in requested.

The session had already appended a turn, streamed assistant chunks, and proposed a tool. The API still reported it as requested.

The worker only replaced sessions in exact terminal states. It suppressed 22 eligible replacement probes because the stale session row looked nonterminal. One state transition missing from one transaction held recovery back for almost 30 minutes.

3. Replay advanced without making progress#

The model runtime streams events over SSE. In the incident trace, the HTTP body sometimes failed after the runner had validated a safe cursor. The consumed event was intentionally suppressed, so the cursor advanced while the projected event count remained zero.

The replay contract had no field for partial transport completion. The worker received an ordinary successful response, checkpointed the cursor, and settled the claim.

The next claim refreshed a usable runner binding. That refresh cleared the replay-unavailable streak before replay produced a model chunk, tool event, waiting state, or terminal event. The loop alternated between unavailable sessions and successful pages with zero applicable progress. The 15-minute and 120-attempt liveness guards never accumulated.

4. Snapshot recovery rejected the source#

Once the session reached its idle deadline, replacement finally began. The runner fetched the snapshot successfully. The sandbox provider rejected the fork with HTTP 409 and a typed non-restorable detail.

The replacement session moved to provisioning_failed. The recovery path needed an exact, fenced fallback: automatic Dex recovery could invalidate the specific bad snapshot and create a clean session, while explicit restore requests had to remain terminal.

The first repair for this boundary exposed another defect. It published snapshots before authoritative readiness and kept a pending snapshot ID only in memory. A timeout or process restart could issue another snapshot creation request.

The replacement repair persisted the pending snapshot identity, resumed with GET-only polling after restart, published only after authoritative Ready status, checked the exact source identity, and required a server-held proof before automatic fallback could destroy state.

5. The next turn inherited the failed replacement#

The failed replacement became the restore source for the next turn. That turn repeatedly attempted to restore a session already known to have failed and received runner_not_ready.

The durable lineage needed to retain the original snapshot-bearing source until a replacement had proven itself. A newly created session was insufficient evidence.

Why one fix could never close this incident#

Queue fairness addressed admission. Publishing the session as running enabled earlier replacement. Snapshot fallback repaired recovery. Replay liveness and restore lineage still needed distinct fixes.

The customer-visible contract was also too weak. Product analytics counted work_started as an answer even though the affected turn had no terminal assistant event.

A healthy submit endpoint and a ready sandbox did not prove that the user received a completed response.

The repair plan therefore crossed several owners while keeping each state transition local to its owner:

• The runner publishes session readiness atomically with binding state and its provisioned event.
• The worker reserves service for fresh thread heads while continuing due work.
• Replay transport reports complete, partial_transport, or reset.
• Replay liveness resets only after durable, turn-applicable progress.
• Snapshot fallback uses exact tenant, session, generation, lease, source, and snapshot fences.
• Failed replacements cannot replace the original restore lineage.
• Runtime-controller replicas remain at one until worker identity and live-lease exclusion are safe for scale-out.

The infrastructure this method required#

We built Orb, an internal remote-agent runtime, using the evidence-lane and control-boundary lessons from this investigation.

Orb persists threads, transcripts, lifecycle events, usage, and audit history in SQLite. Executors reconnect with stable identities after client, network, or control-plane interruptions. Approval gates stay in the control plane. Tool access can be restricted to named operations, with start and completion evidence recorded for each call. Workspace diffs, logs, and artifacts remain attached to the thread.

These properties let us keep separate evidence lanes without tying the investigation record to one client session or one agent process. A source branch, production branch, and release branch can reconnect independently, while a coordinating thread compares their durable reports against the incident timeline.

Operating rules for the next incident#

Split evidence before splitting code#

Source behavior, production behavior, and release provenance deserve separate investigators. The three reports should meet in one causal timeline before anyone proposes a broad repair.

Make readiness specific#

This incident had sandbox readiness, binding readiness, session readiness, replay transport progress, turn-applicable progress, and terminal user response. A single ready field cannot represent those facts.

Treat recovery as a normal production path#

Snapshot creation, restart-safe polling, fork classification, invalidation, cold fallback, and lineage handoff need the same durable state and regression coverage as the initial run.

Record the entire release chain#

For every incident fix, record the source commit, immutable image digest, deploy revision, applied workload digest, and live behavior result. During an outage, “merged” answers only one of those questions.

Measure the terminal event#

The useful SLO is the time from accepted submit to a durable terminal assistant event. Queue admission, sandbox creation, first chunk, and tool proposal are intermediate milestones.