After the outage: making agent completion provable

In our previous post, Bounded agent outage investigation, we described an agent turn that crossed several apparently healthy milestones. The request was accepted. A sandbox became ready. The model streamed output. A tool was proposed.

The user still never received a completed answer.

The incident exposed a problem deeper than any single timeout. The system had several ways to report progress, but no sufficiently strict definition of completion. We traced the failure across scheduling, runtime state, event replay, recovery, and deployment. Since then, we have changed each of those boundaries and the way we verify them.

Started is not done#

Agent systems produce a lot of plausible intermediate signals. A request can be accepted without being scheduled. A computer can be provisioned without being attachable. A stream can advance without producing durable user-visible progress. A replacement session can exist without being safe to resume from. A fix can be merged and published without running in production.

Any one of those states can look healthy in isolation. The outage emerged because several were allowed to stand in for the final outcome.

Completion contract · one agent turn

Progress becomes completion only when the outcome is durable

The rule we now use is simple: a turn is successful only when the intended user-visible result is durably recorded. When the turn requires an approval or an action, those must resolve as well. When it creates an execution environment, cleanup must be accounted for.

That rule now shapes both the runtime and the release process.

Fairness is part of correctness#

One contributing failure was queue starvation. Retrying work could repeatedly consume the available execution opportunities, leaving fresh turns accepted but unclaimed.

The worker now reserves service for fresh thread heads while continuing to process due retries. Parking a not-ready turn no longer ends the entire drain cycle. Within a thread, later turns remain behind earlier active turns until the earlier input has been durably appended.

This is not simply latency tuning. A queue that can indefinitely starve valid work is incorrect, even when every individual retry follows its policy.

Ready now means usable#

The runtime previously had two views of session state that could disagree. The provider binding could be running while the session record still said requested.

That ambiguity affected replacement and recovery decisions. A computer could be doing real work while another part of the system treated it as never having become ready.

Provisioning now advances the provider binding, lifecycle evidence, and attachable session state behind the same authority fences. A session becomes ready only when there is a real runtime to attach to and the owning lease remains valid. A late or stale completion cannot resurrect an expired session.

“Ready” is now a contract, not an optimistic observation.

Transport progress is not turn progress#

The outage also revealed an uncomfortable edge case in streaming replay. The transport could validate a cursor and then fail before producing an event applicable to the current turn. Preserving the cursor was correct. Interpreting that replay as successful was not.

Replay now represents its completion explicitly: complete, partial transport, or reset. Runner Host owns the transport result and safe cursor. Platform Worker separately decides whether the page produced durable model output, a tool transition, a waiting state, or a terminal result for the turn.

Safe cursor progress can survive a partial stream. It cannot authorize completion by itself. A terminal-looking snapshot without a projected terminal event remains incomplete.

This separation matters for any long-running agent. Delivery progress and task progress are different facts.

Recovery keeps the last known-good lineage#

When the original session eventually became replaceable, its snapshot could not be restored. The replacement failed, but the next turn inherited that failed target and repeatedly tried to recover from it.

Recovery now preserves the original snapshot-bearing source until a replacement has actually become usable. Snapshot fallback is fenced by the exact tenant, session, generation, lease, source resource, and snapshot identity. A generic provider conflict does not silently erase recovery state.

A replacement is not part of the durable lineage merely because it was created. It earns that position by reaching a valid state.

Verification now follows the whole execution#

We also changed how these failures are tested.

The original incident became a reusable runtime-verification scenario. It exercises protocol negotiation, admission, durable append, event replay, restart behavior, snapshot readiness, cleanup custody, and release-set compatibility. Failures can be injected at each boundary, and restarts can occur after custody has been taken or replay has begun.

The scenarios enforce a small set of system properties:

• Durable effects are not duplicated across replay.
• Readiness is emitted only after the complete flow succeeds.
• Failures retain an explicit owner for cleanup.
• Incompatible protocol or recovery states fail closed.
• Local evidence never presents itself as production proof.

That last property matters. The verifier records what it can prove and what still requires a live production check. A broad green checkmark is less useful than evidence with an explicit boundary.

Production release gates now require evidence across the operating turn, runtime session, sandbox, approvals, terminal settlement, deployed artifact, and cleanup state. We do not treat a merged commit, a published image, a ready pod, or a successful model call as proof that the user-visible operation completed.

What this changes in Deixic#

This work sharpened a distinction that applies well beyond this incident.

Agent activity has several meaningful states:

accepted -> scheduled -> running -> waiting -> acted -> completed

Collapsing those into “started” or “successful” loses the information operators need. It hides whether the agent is waiting for a person, retrying infrastructure, holding a computer, applying an effect, or actually finished.

Deixic exists because production agent systems need more than a transcript. They need an action history: who requested the work, what ran, what authority it had, which model and computer it used, what approvals were requested, what evidence was produced, and whether the intended outcome actually completed.

The outage forced us to apply that model more strictly to our own runtime.

The result is fewer intermediate states that can masquerade as completion, a durable source of truth for recovery, and release gates tied to the behavior that matters to the user.

The standard going forward#

For production agent work, our definition of done is now stricter:

The requested outcome is durably visible. Required approvals and actions have resolved. The execution lineage can be reconstructed. The running artifact is the one we intended to deploy. Any computer or external resource created for the work has an accounted-for terminal state.

Everything before that is progress.

Failures will still happen. The goal is to make them bounded, attributable, recoverable, and impossible to confuse with success.