A customer reports an unauthorized data access on May 1. The investigation runs on June 15. The audit log has the agent's action, the operator behind it, the policy decision, and the outcome. The investigator has two questions: "what happened," and "why was this allowed at the time — and would it be allowed today." The first question is reconstructable from the audit row alone. The second question requires replay: re-running the action's evaluation against the policy that was in force on May 1, with the inputs the agent saw, to determine whether the policy gate would still emit the same decision.

Replay is the investigation primitive that turns an audit log from a record of what happened into a tool for answering counterfactual questions. It is also audit lag paid down in advance: the answer to a "why was this allowed" question that hasn't been asked yet is already sitting in the policy store, waiting on the row that names the version. It is also a regulatory primitive: the EU AI Act's Article 12 obligation to enable post-market monitoring and to identify risk situations practically requires the ability to reconstruct events. A runtime that produces the audit row but cannot replay the evaluation is missing the second half of the investigation surface.

What "replay" means concretely#

Three distinct operations live under the word replay; teams often conflate them, with operational consequences.

Action replay. Given an audit row, reconstruct the agent's action: which tool was called, with what arguments, against which resource, under which identity. This is what the row schema supports directly; no replay engine is required.
Decision replay. Given the audit row and the policy version recorded on it, re-evaluate the policy gate's decision. The output should match the original decision field. This is what proves the audit row is internally consistent and what allows the investigator to ask "if the policy had been different at the time, would the gate have emitted a different decision?"
Run replay. Given the audit row, the agent's reasoning record, and the run's input context, reconstruct the chain of actions the agent took. This is the heaviest operation; it depends on the agent's reasoning being deterministically reproducible, which is not always possible.

Decision replay is the operation that matters most for security investigations and for the EU AI Act's compliance surface. Run replay is the operation that matters for model risk validation and product debugging. Action replay is the operation that's always available, because the audit row is the only artifact required.

Decision replay in detail#

The mechanics: the audit row records the inputs to the policy gate at the time of evaluation. The five inputs covered in Policy at action time — identity, resource, arguments, run context, policy version — are all on the row, with the policy version as a foreign key into the policy store. The replay engine fetches the policy by version, constructs the input bundle from the row, and runs the policy.

The output should be identical to the row's decision field. If it isn't, one of three things has happened:

The policy store has lost a version (the policy that was in force at the time has been overwritten without preserving the old version). This is a runtime bug; the policy store has to be versioned.
The audit row is missing an input the policy depends on. This is a schema bug; the row needs another field.
The policy depends on external state that was different at the time. This is the unresolvable case in pure replay; the state has to be reconstructed from other sources or the policy has to be rewritten to compile that state into the policy version at load time.

The first two are problems the runtime should not have. The third is a design choice: policies that depend on time-varying external state are harder to replay than policies that depend only on the row's inputs. The discipline covered in the policy post — compile state into the policy version, avoid I/O on the evaluation path — is also what makes decision replay tractable.

Counterfactual decision replay#

Once decision replay works, the natural extension is counterfactual: what would the decision have been under a different policy? Two use cases:

Investigation. Given an action that turned out to be wrong, what policy change would have prevented it? The investigator drafts the proposed change, applies it as a new policy version, runs decision replay against the proposed version, and observes whether the decision differs. The change moves forward if the replay shows the desired effect on the historical action.
Policy review. Before deploying a new policy version, what fraction of historical actions would have changed decision under it? The replay engine runs the new policy against the last 30 days of audit rows and reports how many allowed actions would have become denied or require-approval. The team reviews the changes; an unexpectedly large change set is a signal that the policy has a bug.

Both use cases depend on the audit log carrying enough input data to drive the policy under arbitrary versions. The minimal row that supports decision replay is the row covered in What an agent audit row should carry. The reasoning trace is not required.

Stochastic vs deterministic replay#

Run replay — reconstructing what the agent did and why — runs into the stochasticity question. The model decision at each step depends on the model's sampling, which is not deterministic by default. Two agents given the same inputs may produce different action chains.

The dimensions of stochasticity to address:

Temperature and sampling. If the model is run at non-zero temperature, the same inputs produce different outputs across runs. Run replay against the live model is not deterministic.
Model version. The model that ran the action on May 1 may not be the model deployed today. If the provider has updated the underlying model, replays against the current model are not faithful.
Prompt assembly. Subtle differences in how the prompt was constructed (whitespace, order of retrieved chunks, timestamp inclusion) change the model's output.

Three replay strategies that handle stochasticity differently:

Deterministic seed replay. The runtime records the model's sampling seed; the replay engine uses the same seed against the same model version. This works when the seed is reproducible and the model version is still available; both are runtime-recording decisions that the original action's emitter has to make.
Distribution replay. The replay runs the action many times and reports the distribution of outcomes. Useful for "is this kind of action consistently allowed" but doesn't reconstruct the specific event.
Trace replay. The runtime records the model's output verbatim, and the replay re-runs the policy gate's decisions against the recorded outputs. This trades model fidelity for decision fidelity; the agent's actions are exactly what they were, but the model wasn't re-evaluated.

Trace replay is what most investigation use cases actually need. The question is rarely "what would the model do if we re-ran it"; the question is "given what the model did, was the decision the right one."

The reasoning store#

Run replay requires more than the audit log. It requires the agent's reasoning record: the prompt, the model's response, the retrieved context, the tool outputs. This is the structured reasoning record covered in the audit-row post — kept in a separate store, not in the SIEM-shaped audit log.

The reasoning store's purpose is exactly this replay use case. Its access controls are tighter than the audit log's because it contains content the agent ingested from customer data and third-party APIs. Its retention is shorter than the audit log's because the storage cost is higher and the use case is more specialized.

A useful default partition: the audit log retains for years; the reasoning store retains for 30 to 90 days. Investigations that need both have to land within the reasoning store's window; investigations that come later have the audit log alone, which is enough for decision replay but not for run replay.

EU AI Act Article 12 connection#

Article 12 of the EU AI Act requires high-risk AI systems to enable post-market monitoring and to identify situations that may present risk. Both require replay. Post-market monitoring asks the runtime to surface patterns across many actions; risk identification asks the runtime to reconstruct individual events when they happen.

The replay primitives this post describes are the operational implementation of the Article 12 surface. The audit row is the record-keeping requirement; decision replay is the monitoring capability; counterfactual decision replay is the risk-identification capability that lets the deployer assess proposed mitigations.

A runtime that ships the row schema but not the replay engine is meeting the letter of Article 12 partially. A runtime that ships both is the more defensible posture under conformity assessment.

Model risk validation use case#

Beyond regulation, decision replay is the operational tool that model risk teams use to validate an agent runtime before deploying it on a new workload.

The validation pattern: the model risk team writes a test suite of synthetic actions — actions the agent might plausibly attempt, including out-of-scope ones. The team runs decision replay over the suite against the production policy, expecting specific outcomes. The replay engine reports which expected outcomes matched and which didn't. Mismatches are either bugs in the policy or bugs in the expectations; either is something the team needs to resolve before deploying.

This pattern is the analogue of running a unit test suite against a piece of business logic. The replay engine is the test harness; the policy is the code under test; the synthetic actions are the test cases.

Where the replay engine sits#

Architecturally, the replay engine is adjacent to the policy gate but doesn't sit on the action path. The policy gate runs on every action; the replay engine runs on demand, against historical actions, and against synthetic actions. The two share the policy evaluator (the same code that decides decisions live also decides them on replay) but have different I/O profiles.

The replay engine reads from the audit store and the policy store. It produces reports rather than affecting live agent actions. Its observability is separate from the runtime's; replay activity is itself an action that should be audited, because the replay engine's queries against the audit store are themselves access to sensitive data.

Closing#

A runtime that produces a structured audit row and a separate reasoning record, with versioned policies and a working replay engine, supports three concrete checks: reconstructing what an agent did from the row alone, re-running a policy decision against the version that was active at the time, and testing a proposed policy change against 30 days of historical actions before it ships. The audit log supplies the first; the replay engine supplies the other two.

Deixic ships the row schema, the policy versioning, and the replay surface from the same control plane. See What you can do for the runtime surfaces; the replay engine reads from the same audit log it depends on.