Nathan Lambert warns benchmark announcements can be overtaken before model release

Google released Gemini 4 Argon globally for coding, knowledge work, enterprise workflows and cyber defense, with access through the Gemini API and several Google products. Introductory API pricing is $2 per million input tokens and $10 per million output tokens, with cached input discounted 95%; Artificial Analysis reports rates double after November 30. At AI Engineer NYC, about 20,000 people joined keynotes on AI and finance, including a discussion of benchmarking models like Argon by Vals AI’s CEO.

AI labs are buying defunct startups’ code and workplace archives through companies such as SimpleClosure and Sunset to train models and build environments where AI agents practice tasks. SimpleClosure’s Asset Hub has handled over 150 deals, while Google bought Spirit Airlines’ decades of data for $10 million this past August. The market raises questions about employee privacy, consent, and pricing, and no company has released tests comparing training results from real-world archives with alternatives.

Meta released Muse Wonder and the more capable Wonder Full AI model, which the company says performed near Fable 5.1, Opus 5.5 and GPT-6 Astra on several capability tests, and slightly better on some agent and knowledge-work benchmarks. Meta highlighted Active Memory, which retains relevant background across conversations, and Active Focus, which keeps working toward complex goals and adapts when new information or obstacles arise.

If you announce a model's benchmarks without releasing it (weights, product, api, etc "later"), feels like almost a sure thing that someone releases a better model in your category before you get it out.