On 2026-07-16, someone flipped one runner-routing variable by hand onto a saturated pool, and pull-request CI in our platform monorepo stalled for 18 hours. A factory this automated has single points of failure a person can trip with one unreviewed change; this one cost 18 hours.

The volume that makes this dangerous, measured from the GitHub API and git log on 2026-07-19: the evalops org merged 1,338 pull requests in seven days, 581 of them in evalops/deploy, the GitOps repo holding production desired state. That one repo ran 18,421 Actions workflow runs on 18 July. Of the 841 commits reaching its main that week, 52% were bot-authored or sync-titled — before counting the LLM-authored commits attributed to humans, which the repo's own AGENTS.md says is the usual case.

At that ratio, per-diff human review has stopped being the control; nobody reads 581 diffs a week. This post is about what substitutes for it — the mechanisms that let a mostly machine-authored merge stream run production, and what they cost to learn. We covered the authoring side in the kernel rewrite; this is the operating side.

One shape per change class#

The highest-leverage substitute: find your highest-volume machine change class and give it one machine-checkable shape. Auto-merge what matches; everything else waits for a person.

In deploy, that class is image promotion across ~45 services. Its shape is a single contract file, contracts/runtime-image-sync/production.yaml, which pins everything checkable: the PR title must match ^chore:\s+sync platform runtime images to sha-[a-f0-9]{40}$, desired state is a lock file of immutable sha256: digests plus source commits, auto-merge fires only from an allowlisted set of chore/*-image-sync branch prefixes, and the bots authenticate with short-lived GitHub App tokens over OIDC. The contract opens with an instruction to the agents that edit the repo: this file is the only place promotion behavior may live; do not add gate or rollback logic to a workflow.

The payoff: a change class that landed 388 times last week has one shape, and human review is spent only on what falls outside it.

An incident is closed when the failure is machine-rejected#

The repo carries 116 workflows, 284 preflight tests, an OPA policy tree, and a compiled Go validator. Almost none of that was designed up front; each piece was added after the incident it now prevents.

Three worth copying:

A July schema-migration ordering bug was fixed in two PRs and locked in the same week with a third titled test(resilience): enforce schema-migration gate.
Changes that move production cost — replicas, machine_type, disk_size — trip a gate that requires approval, and the gate checks the approver's GitHub permission level before accepting it.
The 18-hour stall got the same treatment. The hand-flipped variable left no reviewed record, so the fix moved runner routing into a config file in deploy: every routing change is now a pull request with an author, a diff, and a reviewer.

One more control: in a repo where agents author most changes, the AGENTS.md prose is a working control document, because the agents load it before editing.

Automation earns trust on a schedule#

Autonomy in this system is a ladder, and position on it is explicit.

At the bottom, systems with the largest blast radius get none: GitOps applications labeled tier-0 infrastructure run with self-heal off and retries at zero, so live-state drift pages a human instead of reconciling in a loop. In the middle, automation acts only inside identity-checked envelopes: the workflow that auto-reverts a failed promotion acts only when the bad commit is still at origin/main, has a single parent, and was authored by the exact sync-bot identity — and it runs trusted main's code, never the failed run's. At the top, autonomy is granted by numbers: the new ephemeral runner fleet stayed on manual self-heal until it survived a seven-day acceptance gate — at least 100 canary jobs, ≥99% success, pod-startup p95 under 60 seconds — after which a separate guarded commit flipped the flag.

The transferable rule is the last one: before automation gets write access to production state, write its acceptance gate as numbers.

Runners are a security boundary#

At this volume, CI runners spend the day executing machine-authored change, and the stall shows what merely starving them costs. The controls treat them accordingly: runner pods mount no Kubernetes service-account tokens and sit behind default-deny egress; the trusted runner group is scoped to exactly two repos; and the runner image itself is built on GitHub-hosted runners, because building it on the fleet it exists to fix would be circular. Routing is one reviewed change: 70 of the platform's 80 CI jobs select runners through six repository variables, so re-pointing the fleet is a single PR in deploy instead of 32 workflow edits.

Five policies to write before the volume arrives#

Reading the org's own guidance, the written rules cover incidents after the fact better than decisions before them — the same gap between official path and operational path that the operating-model post treats as the whole product. If agent volume is heading toward your repos, these are the five we found missing, each already overdue by one measurable fact:

Which runner class a new CI job belongs on. Ours lives in a Python script and tribal knowledge.
A per-job compute budget, with metering to enforce it. Tool setup ran 19 times per main-branch run — about 180 runner-seconds — before per-run metering existed to notice.
One retry convention. Absent one, agents improvise: a single publish workflow accumulated 82 hand-rolled retry loops with inconsistent backoff.
Who may re-run a required gate, and when a re-run masks a real failure. A re-run can turn a red required check green with no new commit and no new reviewer.
Timeout tiers. Our job timeouts span 5 to 90 minutes with no written rule about which jobs earn which.

What remains as the control#

In the kernel-rewrite week, 90% of the platform's merged PRs came from agent-named branches; this week, 52% of deploy's commits were bot-authored or sync-titled. Once your numbers look like that, the controls are the ones this post walked through: identity checks on the automation, policy evaluated at action time against a contract, approval bounded by verified role, and a reviewed record of every promotion and routing change. On 18 July those held across 18,421 runs.