Skip to content
K = 0.731

Nathan Lambert argues real-world usability outweighs small AA score differences among top models

1 reports 1 reporting sources 2 hours agoNew

Understand the event

AI overview

Nathan Lambert said real-world usability matters much more than a two-point difference in either direction on AA scores among top models, and suggested that 3.1 may fall on the disappointing side in practice. Lambert described AA as a measure of potential, saying it takes time to understand how that potential translates into a model’s usefulness. He cited Opus 5 versus 5.5 as an example of models that can hit or miss in actual use.

Generated by AI from reports · Updated 1 hours ago

Latest development10/24 20:14 (UTC+8)
Nathan Lambert says real-world usability matters more than small AA score differencesNathan Lambert · X

Read next

More changes in today’s briefingKimi K3.1 Pro ranks fourth on Artificial Analysis, approaching Claude Opus 5.5

Report timeline

Follow the reports to explore different sides of the event.

10/24
  1. Nathan Lambert · X
    Nathan Lambert says real-world usability matters more than small AA score differences

    At the top end of models, real-world usability matters a lot more than +/- 2 points on AA. This takes a while to understand for a model, where AA is a sort of potential, but some models hit and some miss (e.g. opus 5 vs 5.5). Seems like 3.1 may be closer to the miss side. https://twitter.com/fawnunwrap/status/2113693298800074801

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.