Stanford Study Finds 44 Percent of LLM Findings Have Expired

What happened

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models released between July 2024 and September 2026. They found that 44 percent of the claims had expired, 34 percent had weakened significantly, and the average lifespan among findings with an estimated expiration date was 15.7 months. Results varied widely across models, leading the researchers to recommend documenting model versions, dates, prompts, and analysis code.

Read next

• More changes in today’s briefingOpenAI’s updated Model Spec explained through its five core values

The story so far

Stanford researchers Owen Queen and James Zou found that 44 percent of 50 widely cited claims about large language models had expired when tested against 20 models released from July 2024 through September 2026. Another 34 percent had weakened significantly, and findings with an estimated expiration date lasted an average of 15.7 months. The results varied widely across models. Queen and Zou recommended that researchers document model versions, test dates, prompts, and analysis code so others can assess how findings hold up over time.