OpenAI published a notable blog post today arguing that its GPT-5.6 Sol model outperforms Anthropic's Claude Opus 5 on ARC-AGI-3 — a new interactive reasoning benchmark — but only when two API settings are enabled that are not part of the standard evaluation harness.
ARC-AGI-3, the latest version of the ARC Prize competition, is designed to measure how well AI agents learn and reason in unfamiliar environments. The benchmark uses 2D puzzle games where agents must figure out game mechanics without explicit instructions. The benchmark has been praised for being more demanding than static text or image classification tasks.
Under the official harness — which uses a generic setup without model-specific optimizations — GPT-5.6 Sol scored just 13.3% on the public set, while Claude Opus 5 holds the top position at 30.2%. But OpenAI discovered that the official harness was discarding the model's private reasoning after each action, forcing it to re-learn the game from scratch every turn.
When OpenAI enabled two API settings — retained reasoning (keeping the model's thinking visible across actions) and compaction (summarizing conversation history instead of truncating it) — GPT-5.6 Sol's score jumped to 38.3% with 6x fewer output tokens. The company argues these settings match how the model is actually deployed in ChatGPT and Codex.
'Benchmarks rarely measure AI models in isolation,' OpenAI wrote. 'They also measure less visible choices about API settings, harness design, and prompting.' The company acknowledged that ARC-AGI-3 used a deliberately simple harness to make model comparisons fairer, but argued that harness design can obscure real-world capabilities.
The response from the AI research community was mixed. Some praised OpenAI for exposing how benchmark methodology can distort results, while others pointed out that enabling model-specific settings for one model but not another would undermine fair comparison. ARC Prize organizers have not yet responded to OpenAI's claims.
The episode highlights a broader tension in AI evaluation: as models become more capable, the gap between standardized benchmarks and real-world performance continues to widen — raising questions about what these scores actually measure.
