Running a lot of software engineering through coding agents taught me that “tests passed” should be a receipt, and a sentence is not enough.

A test result is evidence for the exact code it ran against.

Agents were doing something reasonable:

Run the tests.
Tests pass.
Write “validated” in the PR.
Make another change.
Push.

The validation statement was no hallucination: the tests passed, just not against the code now under review.

We hit this in Deixic when a batch of agent-cut PRs produced broad CI failures, mostly from stale validation: the agent had run the right command against an earlier tree or claimed validation with nothing binding the result to the current tree.

So we stopped treating validation as prose. In our monorepo, a verification run now produces a receipt like:

{
  "scope": "maestro",
  "tree": "59305411b7c83fd49ad8654c2e5e0fcbd0d59dbf",
  "command": "cargo clippy --workspace --all-targets --locked -- -D warnings",
  "exit_code": 0,
  "host": "developer@dev-desktop",
  "output_sha256": "..."
}

The important field is tree, literally:

git rev-parse HEAD:products/maestro

Our check accepts the receipt only if the current subtree still hashes to the same value. So this sequence:

tests pass
    ↓
receipt(tree=abc123)
    ↓
agent edits code
    ↓
tree=def456

produces this instead of "tests passed":

your evidence is stale; rerun the tests

A regression test in scripts/ci/test-verify-receipt.py covers this:

def test_check_fails_when_the_subtree_moved_after_recording(self):
    run_receipt(self.repo, "record", "--scope", "maestro")

    self.touch_maestro("// changed after the receipt was written\n")
    self.commit("maestro change")

    check = run_receipt(self.repo, "check", "--scope", "maestro")

    self.assertEqual(check.returncode, 1)
    self.assertIn("does not match", check.stderr)

We also refuse to record a receipt against dirty source:

def test_record_refuses_a_dirty_subtree(self):
    self.touch_maestro("// uncommitted\n")

    record = run_receipt(self.repo, "record", "--scope", "maestro")

    self.assertEqual(record.returncode, 2)
    self.assertFalse(self.receipt_file("maestro").exists())

Otherwise the evidence attaches to a Git tree that is not the code you tested.

Failure is evidence too:

def test_failing_command_is_recorded_and_check_fails(self):
    record = run_receipt(
        self.repo,
        "record",
        "--scope",
        "maestro",
        "--command-override",
        "printf 'boom\\n'; exit 7",
    )

    receipt = json.loads(...)
    self.assertEqual(receipt["exit_code"], 7)

That one matters more than it sounds. In one full workspace test run, a single test failed and passed when rerun by itself. An agent could turn that into “tests passed.” We kept the failed receipt and wrote:

The failed receipt is retained; it is not replaced with a passing receipt from the isolated retry.

An isolated retry proves only that the isolated retry passed.

That principle has spread. Our repo instructions require agents to treat these as different proof states:

local tests
PR checks
merge
release/GitOps revision
applied rollout
live behavior

A passing unit test does not prove deployment, a successful deployment does not prove the customer-visible behavior works, a screenshot from fixtures does not prove live data, and a model saying it completed something does not make it completed.

We apply the same idea to PR review. Our PR contract is path-aware: if an agent changes a state machine, “tests passed” isn't enough, and the PR states the invariant, transitions, replay behavior, and evidence:

<!-- pr-contract:state-machine -->
status: passed
invariant: one terminal transition is persisted per turn
transitions: pending to running to terminal is monotonic
replay: replaying the lease returns the original receipt
evidence: python3 -m unittest tests/test_runtime_replay.py

Boundary changes require a negative case, cross-repo changes require synchronization evidence, and blocked verification has to say it's blocked instead of quietly becoming “passed.”

At high agent throughput, prose is no longer a sufficient control plane for engineering truth.

“Tested.”
“Reviewed.”
“Fixed.”
“Deployed.”
“Working.”

Those should be references to evidence with explicit scope and invalidation semantics. The implementation can be simple; the useful pattern is:

claim
  → artifact hash
  → command
  → result
  → evidence
  → expiration condition

Bind the claim to the thing it describes, and when that thing changes, make the claim stop being true automatically. Coding agents make software faster to produce, which makes knowing exactly what has been proven more important.