← Blog

Operations · · 5 min read

Discovery lag: how long before you find out what your agents did?

Agents act at machine speed; teams find out at meeting speed. Naming the gap, measuring it, and the one mechanism that makes it negative.

Jonathan Haas

A team posted on Hacker News about running seventeen agents in production: "We had no idea what they were spending. The UI for all of that spending was our credit card statement. Once a month. After the fact." (HN)

Call this gap discovery lag: the time between an agent doing something and a human finding out. That team's lag for spend was thirty days.

Thirty days is the comfortable end:

  • Another founder on the same forum left an agent running, walked away for twenty minutes, and came back to a bill he didn't expect: twenty minutes of lag, thirty-two dollars.
  • In the Replit incident last year, an agent instructed not to touch production deleted a database during a code freeze, and discovery happened whenever a human next looked.
  • The worst case came from a thread about agent monitoring, describing a post-mortem where "the deviation was visible in hindsight from the logs, but no system caught it in real time."

That last case is infinite discovery lag with perfect logging. A log line nobody reads has the same lag as no log line, so more observability does not shorten discovery.

The ratio that broke#

Organizations size their rhythms around discovery lag: code review before merge, standup every morning, the books closed every month. Those rhythms were the discovery system, and they were adequate because work arrived at human speed, so the ratio of how fast things happen to how fast we find out stayed near one.

Agents broke the ratio. A fleet acts hundreds of times an hour, around the clock, while discovery still happens at meeting speed. Overspends, bad writes, and scope creep predate agents; now they happen faster than anyone finds out, and the mistake compounds inside the lag:

  • A runaway loop discovered at twenty minutes costs thirty-two dollars. Discovered at the billing cycle, it is a budget line nobody recognizes.
  • A wrong write discovered at review is a diff comment. Discovered after three other agents built on it, it is an incident with a family tree.

Security teams call this number mean time to detect. No one would run a SOC whose MTTD for intruders was one billing cycle, yet that is the standing MTTD most teams have for their own agents, which hold more privileges than any intruder, act at machine speed, and get detected by invoice. One commenter put the audit version of it directly: "You can't confidently tell a compliance officer what your synthetic workforce is doing." (HN) That is discovery lag measured in quarters.

Three regimes#

Every proposed mechanism lands in one of three regimes.

Dex, waiting in the queueLag is the time between something happening and anyone knowing.

After the fact (lag greater than zero). Dashboards, log search, invoices, the weekly review. Compressing the lag helps linearly: finding out in an hour beats finding out in a month by roughly the ratio of the damage curves. The mistake has still happened, and something has been built on it.

As it happens (lag of zero). Live status, spend that updates as tokens burn, alerts. This is better, but it assumes a human is watching. At 2am, when agents are most useful, nobody is.

Before it happens (lag less than zero). The gate. An action that needs a human waits until a human has seen it, so discovery precedes execution. Negative lag lets the reviewer stop the proposed mistake before it runs, which compressed lag never does. A human can still approve a bad action.

Negative lag has a strict structural requirement, where most setups fail: the gate has to sit at an enforcement boundary, somewhere the agent is unable to proceed. A system prompt that instructs the agent not to proceed leaves the lag positive. The Replit agent had been told not to touch production. ("The infrastructure for agents to act is scaling way faster than the infrastructure for anyone to verify what they did," as one commenter put it when agents gained the ability to buy domains and deploy. "We don't let humans open bank accounts without audit trails." — HN)

Sorting actions into regimes#

You can't put everything behind a gate. Negative lag spends a human's attention, and spending it on reversible actions trains operators to click through approvals. The sort key is reversibility:

  • Irreversible or expensive to reverse (production writes, deletions, external sends, spend past a threshold): negative lag. It waits for a yes.
  • Reversible (recoverable changes): zero lag. Visible while it runs, pausable when it looks wrong.
  • The past (every action, both kinds): each one keeps a record of who ran it, what it touched, and who approved it, so a reviewer can follow the recorded identity, decision, and result.

That policy, written down, goes to a compliance officer.

Measure it#

Three numbers:

  1. Median action-to-seen time. From an agent completing an action to a human having seen it on a screen someone opens. A log someone could grep in theory does not count.
  2. Worst-case lag for the irreversible class. The target is zero or less.
  3. Audit lag. Wall-clock time to answer "who approved this write?", from question to evidence.

If you can't produce these numbers, you still have them, set to the defaults: one billing cycle for spend, however long until someone looks for the irreversible class, an engineer's afternoon per question for audit.

The seventeen-agent team's work was fine. Everything the fleet did reached them a month later, on a statement, in aggregate. Ask what your discovery lag is, number by number, and if any answer contains the word "invoice," start there.