How we turned our coding agent into our own bugbot

We had a coding agent before we had a reviewer, and we were partway into building the reviewer as a separate system when the overlap became obvious.

The coding agent reads a repository, searches for symbols, traces callers, inspects tests, calls tools, and runs against a specific revision. A reviewer needs all of that. What differs is the question: the coding agent asks how to make a change work; the reviewer asks how it can fail.

Changing the question is a prompt change. The rest of this post is what it took to make the answer good enough for every pull request.

We own the harness#

The model that writes a change and the model that reviews it do not have to come from the same provider.

A recent change in our application repository was implemented through Codex and reviewed by Gemini 2.5 Pro. Both ran on our existing agentic-coding runtime, with the same repository context, tool use, structured output, and controlled execution. The model is a configuration value, so changing it does not mean rebuilding the harness.

What the naive pipeline misses#

The first version anyone builds:

git diff
   ↓
LLM
   ↓
GitHub comment

It breaks in five ways:

• The model does not know which revision it is reading, or which callers matter.
• Large diffs exceed the context window.
• A reviewer rewarded for finding bugs starts inventing them.
• Each push reviews the same code again from scratch, so the bot repeats false positives a maintainer already rejected.
• The output reads as authoritative even when coverage was poor.

Ours:

                    GitHub PR
                        │
                        ▼
               pin base + exact head
                        │
                 fingerprint change
                        │
          ┌─────────────┴─────────────┐
          │                           │
     full review?              reuse / incremental /
                               integration review?
          │                           │
          └─────────────┬─────────────┘
                        ▼
                  investigation
              diff + callers + tests
              exact-head source reads
                        │
                        ▼
                  general reviewer
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
        security lens      product/data lens
              └─────────┬─────────┘
                        ▼
              evidence validation
                        │
                  P0/P1 verifier
                        │
                        ▼
               one GitHub verdict
                 + durable receipt
                        │
                        ▼
               maintainer feedback
                        │
                        ▼
              lessons + regressions
                        │
                        ▼
                 replay + eval

Force the reviewer to investigate before it judges#

Before the first model call, the wrapper extracts the declarations the change adds (functions, types, routes, exported constants) and searches the exact reviewed commit for their callers and references. The model also gets read-only tools: pinned file reads, path search, and git grep. Reads resolve against the immutable pull-request head instead of whatever is on main when the model asks.

If the wrapper could not complete that investigation, the model has to search or read source itself. A verdict with no investigation behind it is rejected and retried.

We extracted a shared codec used by one of our synthetic canaries, and everything passed inside the application checkout. An adversarial pass followed the deployment path into the Kubernetes manifests and found that production runs the canary from a flat ConfigMap-mounted directory. The new ./lib/... import worked in the repository and would have failed once re-vendored into production.

The repair added a test asserting that every relative import the canary uses has a shape the production CronJob can mount.

Specialist passes#

One general-purpose pass was not enough. An authorization change needs a reviewer hunting tenant confusion and privilege bypass; a migration needs expand/contract safety and replay; a UI change needs owner-state and placeholder-state checks.

After the general review, the runtime dispatches specialist passes based on the paths that changed. We run three lenses:

• Security: incorrect authorization, tenant leaks, SSRF, secret exposure, approval bypass, and workflow permission mistakes.
• Data: unsafe migrations, missing tenant scope, unstable persisted IDs, cross-service writes, transaction mistakes, and double counting.
• Product: client-side reconstruction of owner state, string-matched authorization, controls that do not round-trip to their owner, and placeholder state presented as production state.

All three run against the same pinned revision under one total budget. No pass votes on another's finding, since a second opinion from a model that did not investigate adds nothing.

The reviewer is allowed to say "I don't know"#

One pull request in the set was close to four megabytes of diff, which no reviewer reads in full. The review states its coverage: about 180 KB of the change reached the review context, across general, security, and product analysis.

On that review, both specialist passes produced malformed source citations, and we rejected them. The final review records those results as invalid and leaves their conclusions out of the pull-request comment.

A reviewer that can only return pass or fail reports success when it had no coverage, so the states are:

reviewed
partially reviewed
finding verified
finding unresolved
specialist failed
evidence invalid
budget exhausted

The comment names which of those applied.

High-severity findings get an adversary#

P0 and P1 findings go to a second pass whose job is to disprove the finding. The verifier looks for counterexamples, existing guards, callers the first pass missed, and tests that disprove the alleged failure. It can remove a finding but cannot add an unverified location.

For small Python and Rust cases we also attempt an executable proof in an isolated environment.

Untrusted pull-request code does not execute inside the credential-bearing reviewer. The trusted base owns the reviewer and its credentials, candidate source is treated as data, and execution happens through a separate sandboxed proof path.

Reuse prior review results#

A full review of the four-megabyte pull request costs more than most updates warrant, so the runtime reuses prior results when the fingerprint allows.

The fingerprint covers the complete patch, source tree, review implementation, model, provider, guidance, selected lessons, and context configuration, and the review mode follows from it:

• An identical fingerprint reuses the prior review with no model request.
• An unchanged patch whose upstream dependencies moved gets an integration review.
• A compatible update reviews only changed files plus files containing previous findings.
• An ambiguous change falls back to a full review, since the cheaper guess is the one that misses a bug.

On the Codex change above, an 8.4 KB diff was reviewed with a security specialist in 18.3 seconds of model execution, at an estimated $0.0716, using about 45.5K input tokens and 1.5K output tokens. Each review records cost, coverage, model calls, tokens, and mode.

Maintainer verdicts feed scoped lessons#

A normal code-review bot is stateless. It comments, someone fixes or ignores it, and the next pull request starts from zero, false positives included. We already build eval systems, so the reviewer got an eval loop.

Every finding gets a stable ID tied to the exact revision, source location, prompt, and selected lessons. A maintainer marks it:

confirmed
false-positive
duplicate
accepted-risk
unresolved

A maintainer can also report a missed bug. Only an explicit mark records an outcome; merges, silence, and bot reactions record nothing.

Confirmed findings and false positives can propose repository-specific lessons. Those lessons do not enter the production prompt directly, because a reviewer that rewrites its own instructions from its own output degrades. They go into Git, scoped to relevant paths, and they expire unless renewed. A pull request cannot modify the lesson file that instructs its own review. Candidate lessons replay against frozen cases before promotion.

Our bug database is our Git history#

We had another source of ground truth: bugs we shipped and later fixed.

Fix pull requests declare the regression they close:

Fixes-Regression-Of: #1234

The review system mines those edges. For each usable regression it freezes the original offending pull request at its original head, before the fix existed, and derives candidate expected locations from code that pull request introduced and the fix later removed. The cases split deterministically into learning and held-out sets, and the reviewer is never told which later pull request fixed the bug.

So a proposed review change gets a testable question: would this reviewer have caught bugs we shipped, before we knew the answer?

Presumed-clean controls penalize a reviewer that flags every pull request.

Two limits. Hitting a line that a later fix changed does not prove the model understood the bug, and a small historical corpus does not establish population-wide recall.

Sometimes the reviewer finds the second bug#

One production failure came from a missing database permission for governed tool binding reads.

The obvious repair granted the runtime the necessary SELECT and scoped the main binding read under PostgreSQL row-level security. The review then followed the neighboring path and found a second defect: visible_channels also read the binding table without installing tenant context. That path now uses its own scoped transaction, released before downstream authorization performs further reads, and the regression test verifies that an ordinary channel stays visible while another actor's binding is filtered out.

We want this sequence from an engineering agent:

find failure
    ↓
trace causal path
    ↓
find adjacent failure
    ↓
repair
    ↓
encode invariant as test
    ↓
review new revision
    ↓
remember outcome

We did not train a model#

We did not fine-tune a code-review model, and the weights learn nothing from our corrections. The repository, the Git history, and the adjudication record hold the lessons, and failures in the reviewer itself become tests.

Because that knowledge sits behind our own runtime, changing the model keeps all of it. A newer model gets promoted on evidence: run the old and new configurations against the same frozen cases, compare findings, false positives, failures, latency, and spend, then check the held-out set before changing the default.

What this still does not do#

• Large changes force partial coverage.
• Models still emit invalid evidence.
• Some bugs need production context the reviewer does not have.
• Explicitly reported misses are not enough to estimate true recall, since we only hear about the ones someone reports.

Most of the system above exists because of those limits.