We had a coding agent before we had a reviewer, and we were partway into building the reviewer as a separate system when the overlap became obvious.

The coding agent reads a repository, searches for symbols, traces callers, inspects tests, calls tools, works across a large codebase, and runs against a specific revision. A reviewer needs all of that. What differs is the question. The coding agent asks how to make a change work; the reviewer asks how the change can fail.

Changing the question is a prompt change. The rest of this post is what it took to make the answer good enough to put on every pull request.

We own the harness#

The model that writes a change and the model that reviews it do not have to come from the same provider.

A recent change in our application repository was implemented through Codex and reviewed by Gemini 2.5 Pro. Both ran on the runtime we already use for agentic coding, with the same repository context, investigation, tool use, structured output, and controlled execution.

The model is a configuration value inside that runtime, which is why we built the reviewer as a second operating mode of the coding agent rather than a separate product. Changing the model does not mean rebuilding the harness around it.

What the naive pipeline misses#

The first version anyone builds looks like this:

git diff
   ↓
LLM
   ↓
GitHub comment

It breaks quickly, and for several reasons at once. The model does not know which revision it is reading, or which callers matter. Large diffs exceed the context window. A reviewer rewarded for finding bugs starts inventing them. Pushing to a pull request reviews the same code again from scratch, so the bot repeats false positives a maintainer already rejected. And the output reads as authoritative even when coverage was poor.

Ours has more steps:

                    GitHub PR
                        │
                        ▼
               pin base + exact head
                        │
                 fingerprint change
                        │
          ┌─────────────┴─────────────┐
          │                           │
     full review?              reuse / incremental /
                               integration review?
          │                           │
          └─────────────┬─────────────┘
                        ▼
                  investigation
              diff + callers + tests
              exact-head source reads
                        │
                        ▼
                  general reviewer
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
        security lens      product/data lens
              └─────────┬─────────┘
                        ▼
              evidence validation
                        │
                  P0/P1 verifier
                        │
                        ▼
               one GitHub verdict
                 + durable receipt
                        │
                        ▼
               maintainer feedback
                        │
                        ▼
              lessons + regressions
                        │
                        ▼
                 replay + eval

Force the reviewer to investigate before it judges#

A reviewer that reads the excerpts it was handed and returns a verdict is guessing. For code changes we do not allow it.

Before the first model call, the wrapper extracts the declarations the change adds — functions, types, routes, exported constants — and searches the exact reviewed commit for their callers and references. The model also gets read-only tools: pinned file reads, path search, and git grep. Reads resolve against the immutable pull-request head rather than whatever is on main when the model asks.

If the wrapper could not complete that initial investigation, the model has to search or read source itself. A verdict with no investigation behind it is rejected and retried.

A recent example. We extracted a shared codec used by one of our synthetic canaries, and everything passed inside the application checkout. An adversarial pass followed the deployment path into the Kubernetes manifests and found that production runs the canary from a flat ConfigMap-mounted directory. The new ./lib/... import worked in the repository and would have failed the moment the code was re-vendored into production.

The repair did not stop at the import path. We added a test asserting that every relative import the canary uses has a shape the production CronJob can mount.

Specialist passes#

One general-purpose pass was not enough. An authorization change needs a reviewer hunting tenant confusion and privilege bypass. A migration needs expand/contract safety and replay. A UI change needs owner-state and placeholder-state checks. A single prompt holding all of that at once is weaker at each of them.

After the general review, the runtime dispatches specialist passes based on the paths that changed. We run three lenses today.

The security lens looks for incorrect authorization, tenant leaks, SSRF, secret exposure, approval bypass, and workflow permission mistakes. The data lens looks for unsafe migrations, missing tenant scope, unstable persisted IDs, cross-service writes, transaction mistakes, and double counting. The product lens looks for client-side reconstruction of owner state, string-matched authorization, controls that do not round-trip to their owner, and placeholder state presented as production state.

All three run against the same pinned revision under one total budget, with different prompts and different jobs. No pass votes on another pass's finding, since a second opinion from a model that did not investigate adds nothing.

The reviewer is allowed to say "I don't know"#

One pull request in the set was close to four megabytes of diff, which no reviewer reads in full. The review states its coverage: about 180 KB of the change reached the review context, across general, security, and product analysis.

On that same review, both specialist passes produced malformed source citations, and we rejected them. The final review records that those specialist results were invalid rather than folding their conclusions into the pull-request comment.

This is a schema problem before it is a modeling problem. A reviewer that can only return pass or fail will report success when it had no coverage. So the states are:

reviewed
partially reviewed
finding verified
finding unresolved
specialist failed
evidence invalid
budget exhausted

The comment names which of those applied.

High-severity findings get an adversary#

P0 and P1 findings go to a second pass whose job is to disprove the finding. The verifier looks for counterexamples, existing guards, callers the first pass missed, and tests that disprove the alleged failure. It can remove a finding. It cannot invent a new unverified location.

For small Python and Rust cases we additionally attempt an executable proof in an isolated environment.

Untrusted pull-request code does not execute inside the credential-bearing reviewer. The trusted base owns the reviewer and its credentials, and candidate source is treated as data. Execution happens through a separate sandboxed proof path.

Reuse prior review results#

A full review of the four-megabyte pull request above costs more than most updates to it warrant, so the runtime reuses prior results where the fingerprint allows.

The fingerprint covers the complete patch, source tree, review implementation, model, provider, guidance, selected lessons, and context configuration, and the mode follows from it. An identical review is reused with no model request. A pull request whose patch is unchanged but whose upstream dependencies moved gets an integration review. A compatible update reviews only changed files plus files containing previous findings. Ambiguous changes fall back to a full review, since the cheaper guess is the one that misses a bug.

One pull request changed after its first review, took an incremental review, and carried the previously reviewed files forward.

On the Codex change above, an 8.4 KB diff was reviewed with a security specialist in 18.3 seconds of model execution, at an estimated $0.0716, using about 45.5K input tokens and 1.5K output tokens. Cost, coverage, model calls, tokens, and review mode are recorded per review, which is what makes them optimizable.

Then we made the bugbot learn#

A normal code-review bot is stateless. It comments, someone fixes or ignores it, and the next pull request starts from zero, including the false positives. We already build eval systems, so the reviewer got an eval loop.

Every finding gets a stable ID tied to the exact revision, source location, prompt, and selected lessons. A maintainer marks it:

confirmed
false-positive
duplicate
accepted-risk
unresolved

A maintainer can also report a bug the reviewer missed. Only an explicit mark records an outcome; merges, silence, and bot reactions record nothing.

Confirmed findings and false positives can propose repository-specific lessons. Those lessons do not enter the production prompt directly, because a reviewer that rewrites its own instructions from its own output degrades. They go into Git, scoped to relevant paths, and they expire unless renewed. A pull request cannot modify the lesson file that instructs its own review. Candidate lessons replay against frozen cases before promotion.

Our bug database is our Git history#

We had another source of ground truth already: bugs we shipped and later fixed.

Fix pull requests declare the regression they close:

Fixes-Regression-Of: #1234

The review system mines those edges. For each usable regression it freezes the original offending pull request at its original head, before the fix existed, and derives candidate expected locations from code that pull request introduced and the fix later removed. The cases split deterministically into learning and held-out sets, and the reviewer is never told which later pull request fixed the bug.

That makes a proposed review change answerable. Instead of asking whether a new prompt looks stronger, we ask whether this reviewer would have caught bugs we actually shipped, before we knew the answer.

We separately track presumed-clean controls, so a reviewer that flags every pull request scores worse.

Two limits. Hitting a line that a later fix changed does not prove the model understood the bug, and a small historical corpus does not establish population-wide recall.

Sometimes the reviewer finds the second bug#

One production failure came from a missing database permission for governed tool binding reads.

The obvious repair granted the runtime the necessary SELECT and scoped the main binding read under PostgreSQL row-level security.

The review then followed the neighboring path and found a second defect: visible_channels also read the binding table without installing tenant context.

So the repair was extended. That path now uses its own scoped transaction and releases it before downstream authorization performs further reads, and the regression test verifies that an ordinary channel stays visible while another actor's binding is filtered out.

That sequence is what we want from an engineering agent:

find failure
    ↓
trace causal path
    ↓
find adjacent failure
    ↓
repair
    ↓
encode invariant as test
    ↓
review new revision
    ↓
remember outcome

We didn't train a bugbot#

We did not fine-tune a code-review model. The underlying weights learn nothing from our corrections. What holds the learning is the repository, the Git history, and the adjudication record, and failures in the reviewer itself become tests.

That trade is deliberate. Because the intelligence sits behind our own runtime, changing the model keeps all of it. A newer model gets promoted on evidence: run the old and new configurations against the same frozen cases, compare findings, false positives, failures, latency, and spend, then check the held-out set before changing the default.

What this still does not do#

Large changes force partial coverage. Models still emit invalid evidence. Some bugs need production context the reviewer does not have. Explicitly reported misses are not enough to estimate true recall, since we only hear about the ones someone reports. Most of the system above exists because of those four limits.

At this point "coding agent" and "code reviewer" are two jobs run by one system, and an adjudicated finding outlives the comment it came from: it becomes a scoped lesson and a frozen eval case.