Ethan Mollick Says Harnesses and Smart Models Could End Famous Hard Benchmarks

What happened

Harnesses + smart models = the end of the last famous hard benchmarks. Models can now work for extremely long times (bye, METR Long Tasks), they can output professional documents with correct math (so long, GDPval) and they can solve complex novel puzzles (farewell, ARC-AGI-3). https://twitter.com/arcprize/status/2110768288523251762

The story so far

Ethan Mollick argued that smart models paired with harnesses undermine three prominent hard AI benchmarks: METR Long Tasks, GDPval and ARC-AGI-3. He said models can now work for extremely long periods, produce professional documents with correct math and solve complex novel puzzles—capabilities he linked to the respective benchmarks. Mollick characterized those capabilities as signaling “the end of the last famous hard benchmarks.” His conclusion concerns what models can accomplish with harnesses, rather than model performance alone.

Keep reading

Trending now

1. 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
2. 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
3. 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
4. 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights