Nathan Lambert Defends the AAv5 Index and Open Models

What happened

I feel a need to defend the AAv5 index and open models at the same time. Benchmarking is messier, more opinionated, and trickier than most people give credit to.

Read next

• More changes in today’s briefingGoogle DeepMind announces Genie 4 for real-time world simulation with native audio

The story so far

Nathan Lambert defended the AAv5 index and open models, arguing that claims of open models’ demise were greatly exaggerated while acknowledging that the previous AA index had flattered them. Lambert said benchmarking is messier, more opinionated, and trickier than it is generally given credit for. He called for more specific intelligence indices because LLMs are used for vastly different tasks.