Hugging Face Introduces StateBench for Long-Context State Tracking

Hugging Face introduced StateBench, a benchmark for measuring how well language models track changing information across long documents. On the hardest tasks with the longest inputs, tested frontier models scored 13% accuracy or less. StateBench tests four levels of tracking, from retrieving a value that has not changed to following chains of references and updates. In experiments where models were trained directly on these tasks, recurrent and hybrid models generalized much farther than transformers; the transformers did not learn the hardest level.

Generated by AI from reports · Updated 44 minutes ago

There is not enough continuous observation data to plot a trend yet.