AI Throughput Is Not AI Progress

AI can increase the amount of work a team produces before it increases the amount of progress the team makes.

That distinction matters after OpenAI's September 6 research-acceleration disclosure. The company reports that, by mid-August, its research organisation was using 3.1 agent-workdays for every human workday. Researchers were contributing code faster and running more experiments. Yet OpenAI also says these measures are easy to collect and difficult to interpret, and that people still choose priorities, judge results, and decide whether to scale, pause, or deploy.

This is not only a frontier-lab issue. Analytics, product, and engineering teams are beginning to run several agents at once: one explores data, another writes code, another tests variants, and another drafts the summary. The visible queue fills with notebooks, pull requests, charts, and recommendations.

The scarce resource moves downstream.

When generation becomes cheap, disciplined judgment becomes the capacity constraint.

The operational risk is review debt: a growing inventory of plausible work that has not received enough challenge, reproduction, synthesis, or accountable decision-making.

More activity can hide a slower decision loop

Output metrics reward the part of the system that agents make easiest to scale. Experiments completed, lines of code, analyses produced, and agent-hours consumed all show activity. None proves that the organisation learned something reliable or made a better decision.

OpenAI's report makes the gap unusually visible. It says that more than half of successful agent tasks estimated at four to eight hours still involved at least one human intervention. It also notes that as some tasks become easier to automate, the least automatable work takes a larger share of human attention and becomes the next bottleneck.

That pattern is familiar in enterprise work. If an agent creates ten pricing analyses overnight but one commercial lead can review only two, the team does not have ten insights. It has two reviewed decisions and eight liabilities in a queue. Those eight items may become stale, conflict with one another, or attract superficial approval because the backlog feels urgent.

Adding another agent makes that queue grow faster. The remedy is to design the full decision loop, not only the generation step.

Put an evidence gate between output and decision

Every agent-generated experiment or analysis should enter a small, explicit flow. The aim is not to make all work heavy. It is to match review effort to consequence and prevent unreviewed volume from masquerading as progress.

The gate should be proportional. A reversible internal chart may need a peer check. A credit-policy change needs independent validation, data lineage, approval, and rollback evidence. The key is that the required evidence is set before the agent produces a persuasive answer.

Do not ask the same agent that created the result to be the only reviewer. It can run checks and critique its own work, but those are generated artifacts too. Independent review may come from a person, a deterministic test, a separately controlled system, or a combination appropriate to the risk.

Manage the queue, not just the agents

Review debt becomes controllable when it is visible. Start with four operating rules.

• Limit work in progress. Set a maximum number of unreviewed items for each team or decision type. When the limit is reached, agents help reproduce, test, document, or retire existing work instead of starting more.
• Reserve review capacity. Plan reviewer time alongside compute and agent budgets. If generation capacity doubles, decide where the additional validation and decision capacity will come from.
• Expire undecided work. Give each result a review-by date. Re-run or discard work whose data, code, assumptions, or business context are no longer current.
• Route by consequence. Low-risk work can use fast evidence gates. High-impact or hard-to-reverse decisions receive independent review and stronger proof.

This also changes how teams use parallel agents. Concurrency is valuable when agents explore genuinely different hypotheses or perform independent checks. It is wasteful when five agents produce near-duplicate answers that one reviewer must reconcile. Ask each run to reduce a named uncertainty, not merely to create another candidate.

Epoch AI's task-level framework for AI research is useful beyond research labs. It separates the work into deciding, designing, building, running, analysing, and communicating, and warns against measuring only what is easiest to observe. Apply the same decomposition to an enterprise workflow. If agents accelerate build and run while analyse and decide remain fixed, fund the bottleneck instead of celebrating the busiest stages.

Measure decisions with evidence

A compact operating dashboard can expose whether throughput is helping:

• median and 95th-percentile time from generated output to a recorded decision;
• number and age of items waiting for review;
• share of decisions with complete inputs, versions, tests, and named ownership;
• independent reproduction rate for results above the risk threshold;
• reversal rate after approval, with the reason for each reversal;
• ratio of experiments started to decisions made.

Keep model cost, agent-hours, and output count as capacity measures, not outcome measures. Pair them with the decision metrics above and one domain result: incident reduction, forecast improvement, conversion lift, cycle-time change, or another measure tied to the work.

A rising experiment-to-decision ratio is an early warning. It can mean the team is exploring well, but if queue age and evidence gaps rise with it, the organisation is manufacturing review debt.

Scale judgment deliberately

The wrong reaction is to slow every agent until human review catches up. The better move is to automate the mechanical parts of evidence creation while keeping accountable judgment explicit.

Agents can attach data hashes, run reproducibility checks, compare against baselines, execute negative tests, and assemble review packets. Deterministic gates can reject missing fields or failed tests before a person sees the work. Reviewers can then focus on assumptions, trade-offs, and whether the evidence supports action.

The headline metric is no longer how much work the agents produced. It is how many consequential decisions moved through a complete evidence loop—and how often those decisions held up.

AI throughput is real capacity. Progress begins when that capacity becomes trusted learning and an accountable choice.

Continue reading

AI Data Agents Need an Investigation Contract

When an AI agent can explore data faster than a team can review it, define the question, evidence, budget, and stopping rule before the first query.

Your AI Agent Registry Needs an Expiry Date

A searchable agent catalog can spread stale trust. Add admission evidence, review-by dates, and retirement before reuse scales.