Ai2's StateBench shows frontier models lose track of swapped values in long documents

On November 3, 2026, Ai2 released StateBench, a benchmark for long-context state tracking: following entities whose values change across a long document. Problems come in four levels - r-trivial (values set once), aperiodic (reassignments, latest value counts), soluble (updates build on the current value, such as running totals) and general finite (one variable takes another’s value, as in swaps, linked to the S5 problem). Data and code are on Hugging Face and GitHub with a tech report.

Frontier models (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna) did well on r-trivial and aperiodic problems, but accuracy fell sharply with length on soluble and general finite problems, reaching 13 percent or lower on the longest general finite ones. Pretrained open 7B models, including Olmo 3 and Olmo Hybrid, stayed at chance on general finite problems. Small hybrid and Gated DeltaNet models near ceiling on RULER-style retrieval dropped to near chance when the same problems were framed as spreadsheet edits.

In controlled training from scratch at 275 million parameters, trained on up to 256 assignments and tested on up to 64,000, Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, while transformers degraded on long sequences and never learned general finite problems. A Gated DeltaNet variant with a more flexible memory update stayed near perfect on the hardest level out to roughly 10 times its training length.

Why it matters: long-context scores are often measured with needle-in-a-haystack retrieval, and StateBench shows that retrieving a fact and tracking how it changes are different skills, with architecture choices mattering for the second.

What it does not show: the framework assumes a fixed set of variables while real documents introduce new entities; the theoretical placement of the best Gated DeltaNet variant is unresolved; the effects of numerical precision and of mixing it with attention are open; and the blog gives no total problem count.

Sources

Last verified November 9, 2026