Nathan Lambert Says Open Models’ Demise Has Been Greatly Exaggerated

What happened

Open models' demise has been greatly exaggerated, but the previous AA index did flatter them. As an industry we need more specific intelligence indices, as LLMs are used in vastly different tasks. My meta-review on the benchmarking industry.

Read next

• More changes in today’s briefingGoogle DeepMind announces Genie 4 for real-time world simulation with native audio

The story so far

Nathan Lambert defended the AAv5 index and open models, arguing that claims of open models’ demise were greatly exaggerated while acknowledging that the previous AA index had flattered them. Lambert said benchmarking is messier, more opinionated, and trickier than it is generally given credit for. He called for more specific intelligence indices because LLMs are used for vastly different tasks.