StateBench Tests How Language Models Track State Across Long Documents

What happened

Hugging Face introduces StateBench, a benchmark that measures four levels of state tracking, from retrieving an unchanged value to following chains of references and updates. In tests, frontier models’ accuracy on the hardest problems fell to 13% or lower on the longest inputs. Models trained directly on the tasks showed different results: recurrent and hybrid models generalized much farther than transformers, while transformers did not learn the hardest level.

Read next

• More changes in today’s briefingOpenAI’s updated Model Spec explained through its five core values

The story so far

Hugging Face introduced StateBench, a benchmark for measuring how well language models track changing information across long documents. On the hardest tasks with the longest inputs, tested frontier models scored 13% accuracy or less. StateBench tests four levels of tracking, from retrieving a value that has not changed to following chains of references and updates. In experiments where models were trained directly on these tasks, recurrent and hybrid models generalized much farther than transformers; the transformers did not learn the hardest level.