Operations · · 5 min read
Discovery lag: how long before you find out what your agents did?
Agents act at machine speed; teams find out at meeting speed. Naming the gap, measuring it, and the one mechanism that makes it negative.
A team posted on Hacker News about running seventeen agents in production: "We had no idea what they were spending. The UI for all of that spending was our credit card statement. Once a month. After the fact." (HN)
Call the gap they're describing discovery lag: the time between an agent doing something and a human finding out. That team's discovery lag for spend was thirty days.
Thirty days is the comfortable end of the range. Another founder on the same forum left an agent running, walked away for twenty minutes, and came back to a bill he didn't expect — twenty minutes of lag, thirty-two dollars. The Replit incident last year sat somewhere in between: an agent that had been instructed not to touch production deleted a database during a code freeze, and discovery happened whenever a human next looked. And the worst case showed up in a thread about agent monitoring, describing a post-mortem where "the deviation was visible in hindsight from the logs, but no system caught it in real time." That's infinite discovery lag with perfect logging. A log line nobody reads has the same discovery lag as no log line at all. That is the difference between observability and discovery; buying more of the first buys none of the second.
The ratio that broke#
Discovery lag isn't new. Organizations size their rhythms around it: code review before merge, standup every morning, the books closed every month. Those rhythms were the discovery system, and they were adequate for one reason — work arrived at human speed, so the ratio of "how fast things happen" to "how fast we find out" stayed near one.
Agents broke the ratio. A fleet acts hundreds of times an hour, around the clock, while discovery still happens at meeting speed. Nothing categorically new goes wrong — overspends, bad writes, and scope creep all predate agents. It goes wrong faster than you find out, and the mistake compounds inside the lag. A runaway loop discovered at twenty minutes costs thirty-two dollars; discovered at the billing cycle, it's a budget line with a name nobody recognizes. A wrong write discovered at review time is a diff comment; discovered after three other agents have built on top of it, it's an incident with a family tree.
Security teams already have a name for this number: mean time to detect. No one would run a SOC whose MTTD for intruders was "one billing cycle." But that is the standing MTTD most teams have for their own agents — actors holding more privileges than any intruder, acting at machine speed, detected by invoice. One commenter put the audit version of it directly: "You can't confidently tell a compliance officer what your synthetic workforce is doing." (HN) That's discovery lag measured in quarters.
Three regimes#
Every mechanism anyone has proposed for this problem lands in one of three regimes.
After the fact — lag greater than zero. Dashboards, log search, invoices, the weekly review. Compression helps, and it helps linearly: finding out in an hour beats finding out in a month by roughly the ratio of the damage curves. But the mistake has already happened, and something built on it in the meantime.
As it happens — lag of zero. Live status, spend that updates as tokens burn, alerts. Better — with one assumption baked in: a human is watching. At 2am, during the exact hours agents are most useful, nobody is watching.
Before it happens — lag less than zero. The gate. An action that needs a human waits until a human has seen it, so discovery precedes execution. Negative lag is categorically different from compressed lag: it is the only regime in which the mistake doesn't happen.
Negative lag has a strict structural requirement, and it's where most current setups quietly fail: the gate has to exist at an enforcement boundary, somewhere the agent is unable to proceed rather than instructed not to. A system prompt only impersonates a gate: positive lag wearing a costume. The Replit agent had been told not to touch production. ("The infrastructure for agents to act is scaling way faster than the infrastructure for anyone to verify what they did," as one commenter put it when agents gained the ability to buy domains and deploy. "We don't let humans open bank accounts without audit trails." — HN)
Sorting actions into regimes#
You can't put everything behind a gate. Negative lag spends a scarce resource — a human's attention — and spending it on reversible actions is how you train operators to click through approvals. The sort key is reversibility:
- Irreversible or expensive to reverse — production writes, deletions, external sends, spend past a threshold: negative lag. It waits for a yes.
- Reversible — the other ninety-something percent: zero lag. Visible while it runs, pausable when it looks wrong.
- The past — every action, both kinds: bounded lag. Each one keeps a record of who ran it, what it touched, and who approved it, so any question about history costs minutes.
Written down, that's a policy — one you can hand to a compliance officer.
Measure it#
Three numbers tell you where you stand:
- Median action-to-seen time. From an agent completing an action to a human having actually seen it — on a screen someone opens. A log someone could theoretically grep doesn't count.
- Worst-case lag for the irreversible class. The target is less than or equal to zero. That's not an aspiration; it's the definition of the class being gated.
- Audit lag. Wall-clock time to answer "who approved this write?" — from question to evidence.
If you can't produce these numbers, you still have them; they're just set to the defaults. The default for spend is one billing cycle. The default for the irreversible class is however long until someone looks. The default for audit is an engineer's afternoon, per question.
The seventeen-agent team didn't have an agent quality problem — the work was fine. Their problem was that everything the fleet did reached them a month later, on a statement, in aggregate. Ask what your discovery lag is, number by number. If any answer contains the word "invoice," start there.