Artificial Analysis has rebuilt its Intelligence Index, and the most consequential change is not which model is on top. It is that most of the test is now invisible.

Version 5.0, published this week, combines nine evaluations into a single score weighted across four equally sized categories: knowledge work, coding, science and general knowledge. Two benchmarks are new — Terminal-Bench-Science 0.1, seventy agentic research tasks written and reviewed by domain experts across the life, physical, earth, mathematical and engineering sciences, and FrontierCode 2.0, Cognition's private agentic coding benchmark of 132 tasks in real open-source repositories, graded on whether a maintainer would actually merge the agent's change. Three were upgraded: HLE-Diamond, a refined 1,000-question version of Humanity's Last Exam split evenly between reasoning and knowledge, AA-Omniscience v1.1, and the hard subset of CritPt v2. Three were dropped: GDPval-AA v2.1, SciCode and AA-LCR v1.1.

The structural shift is the secrecy. As heise reports, roughly 60 percent of the tasks in the index are no longer public — the first time the majority has been closed. The motivation is uncontroversial: public benchmarks leak into training data, and labs optimise against them until the score measures memorisation and benchmark-shaped reasoning rather than capability. A model that has seen the test is not being tested.

The cost is equally real, and the index's own users pay it. A public benchmark can be audited. When researchers found broken tasks, ambiguous answers and leaked solutions in HumanEval, GSM8K and MMLU, those benchmarks got better because the failures were visible to everyone. A private task set cannot be checked by outsiders, so a mislabelled answer, a task with two valid solutions, or a verifier that rewards the wrong behaviour simply moves a headline number with nobody able to notice. Artificial Analysis says all evaluations are run independently by the company — a meaningful claim about the process, and not the same thing as external review. Reproducibility takes the same hit: you cannot re-run a task you cannot see.

There is a second, quieter shift. Several component benchmarks now originate with vendors — FrontierCode from Cognition, GDP.pdf from Surge AI, AutomationBench from Zapier's environment — and are implemented by Artificial Analysis inside a composite that steers purchasing decisions. That is a reasonable way to get hard, realistic tasks. It also means control over what "frontier capability" means is distributed across companies with their own interest in the answer.

The leaderboard as of publication: Gemini 4 Carbon (High) at 59, Claude Opus 5.5 at 58 (Max) and 57 (Xhigh). Heise notes Opus 5.5 led after the rebuild until Gemini 4 Carbon's release overtook it, that Grok 4.7 lost points, and that Kimi K3.1 Pro fell — leaving the best open model twelve points behind the frontier, where the previous version had the gap at three.

That last number will be quoted far more than it deserves. A rebuild that swaps three tests, upgrades three others and leaves the weights alone still changes what the score means; a widening gap under a new test mix is partly a statement about the new mix. It may be a real capability gap, or it may be that the new tests — particularly the agentic coding and science ones — reward whatever the current frontier models were trained for. Separating those two needs the old and new suites run side by side on the same models, published together. Which is exactly the check that churn and secrecy make harder.

The practical takeaway for anyone citing the index: name the version. Scores from v4.x and v5.0 are not comparable, and a chart that plots them on one line is wrong. For procurement, component benchmarks matter more than the composite — a model that is mid-table overall and strong on Terminal-Bench and FrontierCode is a better coding agent than the reverse — and the cost-per-task and token-usage data published alongside the index remain its most underused part.

Sources