Coding agents are among the most economically consequential deployments of language models anyone has made, which makes the numbers used to rank them load-bearing. Terminal-Bench became one of the reference points for that ranking. Version 2.0 went out after what its authors describe as extensive human and model-assisted review and red-teaming — the version that was supposed to be trustworthy.

A new audit found issues in roughly 30% of it.

The paper is "Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks" (arXiv 2611.08417), by E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré. The repairs ship as Terminal-Bench 2.1.

Debt, not bugs

The authors give the failure mode a name: benchmark maintenance debt — the growing gap between what a benchmark scores and the capability it claims to measure.

Two mechanisms produce it, and only one is a bug. The first: as models improve, they find valid solutions that existing verifiers reject. The task is fine, the model is fine, the checker is wrong — and the model is marked down for being better than the test anticipated. The second: the evaluation infrastructure moves underneath the tasks. A time limit, a container image, a dependency version changes, and an assumption baked into a test no longer holds.

Neither leaves a stack trace. A benchmark that rots this way still runs, still returns numbers, and still gets quoted.

What the repairs changed

Fixing the 30% was not cosmetic. On the repaired benchmark, measured accuracy rose by more than ten percentage points — a swing larger than the gap between many adjacent model releases. Leaderboard positions moved by up to three ranks.

The overall ordering, though, held: the correlation between old and new rankings is Spearman rho = 0.958. The reassuring reading is that the ladder is broadly right. The uncomfortable one is that individual rungs are not, and a rung is what a buyer selects on.

The clock is part of the ruler

The paper's second finding travels further than benchmarks. Resource allocation is not a neutral harness detail; it is part of the measurement specification.

Extending the time limit closed a 20-point harness gap on GPT 5.4 and shrank a 30-point gap on GPT 5.4 mini to five points. Same model, same tasks, different wall clock — and a gap that looked like a model difference was mostly a budget difference. Most published agent evaluations fix a single turn budget across models of very different speeds, then report the resulting spread as capability.

Maintenance as a workflow

The proposed process is unglamorous and mostly about not trusting automation with the last word: community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. The human gate matters. A benchmark repaired automatically by the models being benchmarked is a closed loop with a number at the end of it.

There is a wider point in the timing. Terminal-Bench has since moved through further major versions, and every one of them inherits the same obligation: the repairs that produced 2.1 were needed on a benchmark that had already been validated, and nothing about the next version makes that stop being true.

For anyone quoting a coding-agent score — in a launch post, a procurement memo, a leaderboard screenshot — the practical question is two words longer than it used to be: which version, and when was it last validated? A benchmark is software. Software is never finished, and a number with no maintenance log behind it is a claim, not a measurement.