After the outage: making agent completion provable

Our previous post, Bounded agent outage investigation, described an agent turn that crossed several apparently healthy milestones: the request was accepted, a sandbox became ready, the model streamed output, a tool was proposed. The user never received a completed answer.

No single timeout caused it. The system had several ways to report progress and no strict definition of completion. We traced it across scheduling, runtime state, event replay, and recovery, then changed each boundary and how we verify it.

Started is not done#

Agent systems produce plausible intermediate states:

• A request can be accepted without being scheduled.
• A computer can be provisioned without being attachable.
• A stream can advance without producing durable user-visible progress.
• A replacement session can exist without being safe to resume from.
• A fix can be merged and published without running in production.

Each looks healthy in isolation. The outage happened because several were allowed to stand in for the final outcome.

Completion contract · one agent turn

Progress becomes completion only when the outcome is durable

Our rule: a turn is successful only when the intended user-visible result is durably recorded. Any required approval or action must resolve too, and any execution environment it created needs accounted-for cleanup. The rule shapes both runtime and release process.

Fairness is part of correctness#

One contributing failure was queue starvation: retrying work repeatedly consumed the available execution opportunities, leaving fresh turns accepted but unclaimed.

The worker now reserves service for fresh thread heads while still processing due retries, and parking a not-ready turn no longer ends the drain cycle. Within a thread, later turns wait until earlier input is durably appended.

A queue that can indefinitely starve valid work is incorrect, even when every individual retry follows its policy.

Ready now means usable#

The runtime had two views of session state that could disagree: the provider binding could be running while the session record still said requested. A computer could do real work while another part of the system treated it as never ready, which corrupted replacement and recovery decisions.

Provisioning now advances the provider binding, lifecycle evidence, and attachable session state behind the same authority fences. A session becomes ready only when a real runtime exists to attach to and the owning lease remains valid, and a late or stale completion cannot resurrect an expired session. "Ready" is a contract, not an optimistic observation.

Transport progress is not turn progress#

The outage exposed an edge case in streaming replay: the transport could validate a cursor and then fail before producing an event applicable to the current turn. Preserving the cursor was correct; interpreting that replay as successful was wrong.

Replay now reports its completion explicitly: complete, partial transport, or reset. The runner host owns the transport result and the safe cursor; the platform worker separately decides whether the page produced durable model output, a tool transition, a waiting state, or a terminal result.

Safe cursor progress survives a partial stream but cannot authorize completion. A terminal-looking snapshot without a projected terminal event remains incomplete. Delivery progress and task progress are different facts.

Recovery keeps the last known-good lineage#

When the original session became replaceable, its snapshot could not be restored. The replacement failed, and the next turn inherited that failed target and kept trying to recover from it.

Recovery now preserves the original snapshot-bearing source until a replacement is usable; a replacement joins the durable lineage by reaching a valid state, not by being created. Snapshot fallback is fenced by the exact tenant, session, generation, lease, source resource, and snapshot identity, and a generic provider conflict no longer erases recovery state.

Verification follows the whole execution#

The original incident became a reusable runtime-verification scenario covering protocol negotiation, admission, durable append, event replay, restart behavior, snapshot readiness, cleanup custody, and release-set compatibility. Failures are injected at each boundary, and restarts occur after custody has been taken or replay has begun.

The scenarios enforce five system properties:

• Durable effects are not duplicated across replay.
• Readiness is emitted only after the complete flow succeeds.
• Failures retain an explicit owner for cleanup.
• Incompatible protocol or recovery states fail closed.
• Local evidence never presents itself as production proof.

The last property matters most. The verifier records what it proves and what still needs a live production check, because evidence with an explicit limit beats a broad green checkmark.

For release, a merged commit, published image, ready pod, or successful model call is no proof that the user-visible operation completed. Evidence covers the operating turn, runtime session, sandbox, approvals, terminal settlement, deployed artifact, and cleanup state.

What this changes in Deixic#

Agent activity has several states:

accepted -> scheduled -> running -> waiting -> acted -> completed

Collapsing those into "started" or "successful" hides whether the agent is waiting for a person, retrying infrastructure, holding a computer, applying an effect, or finished.

Deixic keeps company procedures, tests, authority, and execution history outside the agent. A completion contract gives the next worker an explicit success criterion, and the result and recovery history show whether it was met. Incident-derived checks belong in the company’s reusable methods. See the company layer.

We now apply that model to our own runtime: a durable source of truth for recovery and release gates tied to the behavior the user sees.

The standard going forward#

For production agent work, done means:

• The requested outcome is durably visible.
• Required approvals and actions have resolved.
• The execution lineage can be reconstructed.
• The running artifact is the one we intended to deploy.
• Any computer or external resource created for the work has an accounted-for terminal state.

Everything before that is progress. Failures will still happen; this standard keeps them limited in scope, attributable, recoverable, and distinguishable from success.