The Allen Institute for AI has released StateBench, a benchmark for state tracking across long documents: how well a language model keeps hold of what has happened to the entities in a text as their state changes. It sounds narrow, but it is the substrate of tasks as different as reading code, where variables are assigned and reassigned, and following a story in which a character hands an object to someone else.
StateBench grades difficulty in four levels that map onto kinds of assignment. At the first, a value never changes, like a birth date. At the second, values are reassigned, like a thermostat where the latest setting governs. At the third, each update builds on the current value, x = x + 5, like a running score. At the fourth, values are assigned by reference, x = y, so one variable takes another's current value: the S₅ shell game of following successive swaps among five cups. The grading is not merely pedagogical; it follows Krohn-Rhodes theory, an algebraic decomposition of state-tracking problems into primitive operations. To compare levels fairly, StateBench derives all four variants from the same underlying sequence, with identical variables, values and assignment positions, changing only the type of assignment.
On pretrained open models, including Olmo 3 as a pure transformer and Olmo Hybrid with Gated DeltaNet layers, the pattern was unambiguous. Almost all did well on the first level, lost accuracy on the second as inputs grew, and sat at chance on the fourth. Format mattered too: small hybrids scored near the ceiling on first-level problems phrased RULER-style, and near chance on the same problem framed as spreadsheet edits.
Four frontier models, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, reproduced the pattern at larger scale. All four were solid on the first two levels. On the third and fourth, accuracy fell sharply with length, dropping to 13% or lower on the longest fourth-level tasks. Ai2 offers sensitivity as the likely factor: at these levels many steps can affect the final answer, so a single missed update is enough to get it wrong.
The most revealing section is the training experiment. Ai2 trained 275-million-parameter models from scratch directly on StateBench, on sequences of at most 256 assignments, then tested them out to 64,000, or 250 times the training length. Gated DeltaNet and hybrid models generalised almost perfectly across tens of thousands of assignments. Transformers learned the easier levels, degraded on long sequences, and never learned the fourth level at all, even with direct training. For recurrent and hybrid architectures the gap is training; for transformers it is the architecture.
What remains open is substantial. A new proof shows Gated DeltaNet with a more constrained memory update can solve all aperiodic tasks and certain periodic ones under fixed-precision arithmetic, but its exact position in the hierarchy is unresolved. StateBench also assumes a fixed number of variables, while real tasks introduce new entities as they unfold. For anyone building coding agents, the 13% is best read as a measurement of the assignment-by-reference work that long agent sessions actually consist of: a variable bound to another variable, then rebound, thousands of steps deep.
