A 1-second submit hid a 30-minute outage

At 15:50 UTC on August 9, Deixic accepted an AI turn in about one second. The sandbox was ready 22 seconds later. The first assistant chunks reached durable storage after 105.8 seconds, and a tool proposal arrived after 146.6 seconds.

The turn never produced a terminal assistant event.

During the next 30 minutes, the runtime claimed the work 95 times. The model produced 109 response chunks and one tool proposal. Recovery eventually started, hit a snapshot fork rejection, and left the next turn attached to a failed replacement session.

From the product, this looked like one outage: Dex was stuck. In the system, it was a chain of distinct defects across scheduling, state projection, streaming, snapshot recovery, and session lineage, and the failure began after both the submit endpoint and sandbox had reported success.

We reconstructed the incident with a read-only agent investigation that kept source behavior, production evidence, and release state apart.

The first plausible cause was real and incomplete#

Earlier that day, several requests failed before a runner session existed. Those failures had their own causes and did not explain this turn, which had a ready sandbox, streamed output, and a tool proposal.

What the investigation method changed#

The investigation had read access to source, Git history, production logs, database records, and release evidence. It could not edit code, restart workloads, rerun jobs, change cloud resources, or deploy.

The work ran in three parallel evidence lanes:

1. Reconstruct the runtime state machine from source and recent changes.
2. Trace the production request through the worker, runner, model runtime, and sandbox provider.
3. Audit merge, image publication, deployment, and live workload evidence as separate release states.

One coordinating thread compared the three reports against the user timeline. Smaller threads tested narrow hypotheses about draft ownership, request-path divergence, runner provisioning, release admission, replay disconnects, and snapshot readiness.

This structure had four benefits:

• The state machine, live request, and release chain were inspected at the same time.
• Read-only access kept evidence collection separate from production changes.
• The coordinating thread rejected explanations that failed to account for the timestamped events.
• The release lane kept merged code from being mistaken for deployed behavior. It tracked five separate facts: merge, image publication, GitOps update, applied pod digest, and successful user turn. Each can be true while the next is false.

How the failures compounded#

Failure propagation · one Deixic turn

Five runtime boundaries formed one recovery loop

Five layers of failure#

1. Retry ordering starved fresh work#

The runtime controller had one replica and processed work serially. Its batch setting admitted four rows per tick, but due retries received priority. A not-ready result scheduled another retry and ended the current drain after one claim.

After the original turn's retry loop formed, fresh thread heads stopped receiving service. During a six-minute production window, the worker processed 38 claims across 13 older work items. Fresh turns received zero claims. Two synthetic test profiles timed out after roughly 92 seconds while still queued.

The scheduler followed its ordering rules, and those rules let a sustained retry set consume every execution slot.

2. Two tables disagreed about runner readiness#

The runner stored successful provisioning in the runtime binding and emitted a provisioned event. The owning runner session row remained in requested.

The session had already appended a turn, streamed assistant chunks, and proposed a tool, yet the API still reported it as requested.

The worker only replaced sessions in exact terminal states, so the stale row suppressed 22 eligible replacement probes. One state transition missing from one transaction held recovery back for almost 30 minutes.

3. Replay advanced without making progress#

The model runtime streams events over SSE. In the incident trace, the HTTP body sometimes failed after the runner had validated a safe cursor. The consumed event was intentionally suppressed, so the cursor advanced while the projected event count remained zero.

The replay contract had no field for partial transport completion, so the worker received an ordinary successful response, checkpointed the cursor, and settled the claim.

The next claim refreshed a usable runner binding, which cleared the replay-unavailable streak before replay produced a model chunk, tool event, waiting state, or terminal event. The loop alternated between unavailable sessions and successful pages with zero applicable progress, so the 15-minute and 120-attempt liveness guards never accumulated.

4. Snapshot recovery rejected the source#

Once the session reached its idle deadline, replacement finally began. The runner fetched the snapshot successfully, and the sandbox provider rejected the fork with HTTP 409 and a typed non-restorable detail.

The replacement session moved to provisioning_failed. Recovery needed an exact, fenced fallback: automatic Dex recovery invalidates the specific bad snapshot and creates a clean session, while explicit restore requests stay terminal.

The first repair for this boundary had its own defect. It published snapshots before authoritative readiness and kept a pending snapshot ID only in memory, so a timeout or process restart could issue another snapshot creation request. A sound design persists the pending snapshot identity, resumes with GET-only polling after restart, publishes only after authoritative Ready status, checks the exact source identity, and requires a server-held proof before automatic fallback destroys state.

5. The next turn inherited the failed replacement#

The failed replacement became the restore source for the next turn, which repeatedly tried to restore a session already known to have failed and received runner_not_ready. The durable lineage has to retain the original snapshot-bearing source until a replacement proves itself. A newly created session is not evidence.

Why one fix could never close this incident#

Queue fairness addresses admission, publishing the session as running enables earlier replacement, and snapshot fallback repairs recovery. Replay liveness and restore lineage each need a distinct fix.

The customer-visible contract was also too weak: product analytics counted work_started as an answer even though the affected turn had no terminal assistant event.

A healthy submit endpoint and a ready sandbox did not prove that the user received a completed response.

The repairs cross several owners, and each state transition stays local to its owner:

• The runner publishes session readiness atomically with binding state and its provisioned event.
• The worker reserves service for fresh thread heads while continuing due work.
• Replay transport reports complete, partial_transport, or reset.
• Replay liveness resets only after durable, turn-applicable progress.
• Snapshot fallback uses exact tenant, session, generation, lease, source, and snapshot fences.
• Failed replacements cannot replace the original restore lineage.

Operating rules for the next incident#

Split evidence before splitting code#

Give source behavior, production behavior, and release provenance separate investigators, and merge their reports into one causal timeline before anyone proposes a broad repair.

Make readiness specific#

This incident involved sandbox readiness, binding readiness, session readiness, replay transport progress, turn-applicable progress, and terminal user response. One ready field cannot represent them all.

Treat recovery as a normal production path#

Snapshot creation, restart-safe polling, fork classification, invalidation, cold fallback, and lineage handoff need the same durable state and regression coverage as the initial run.

Record the entire release chain#

For every incident fix, record the source commit, immutable image digest, deploy revision, applied workload digest, and live behavior result. "Merged" answers only the first.

Measure the terminal event#

Measure the time from accepted submit to a durable terminal assistant event. Queue admission, sandbox creation, first chunk, and tool proposal are intermediate milestones.