Nathan Lambert says real-world usability matters more than small AA score differences

What happened

At the top end of models, real-world usability matters a lot more than +/- 2 points on AA. This takes a while to understand for a model, where AA is a sort of potential, but some models hit and some miss (e.g. opus 5 vs 5.5). Seems like 3.1 may be closer to the miss side. https://twitter.com/fawnunwrap/status/2113693298800074801

Read next

• More changes in today’s briefingKimi K3.1 Pro ranks fourth on Artificial Analysis, approaching Claude Opus 5.5

The story so far

Nathan Lambert said real-world usability matters much more than a two-point difference in either direction on AA scores among top models, and suggested that 3.1 may fall on the disappointing side in practice. Lambert described AA as a measure of potential, saying it takes time to understand how that potential translates into a model’s usefulness. He cited Opus 5 versus 5.5 as an example of models that can hit or miss in actual use.