Ethan Mollick Says Harnesses and Smart Models Undermine Three Hard AI Benchmarks

Ethan Mollick argued that smart models paired with harnesses undermine three prominent hard AI benchmarks: METR Long Tasks, GDPval and ARC-AGI-3. He said models can now work for extremely long periods, produce professional documents with correct math and solve complex novel puzzles—capabilities he linked to the respective benchmarks. Mollick characterized those capabilities as signaling “the end of the last famous hard benchmarks.” His conclusion concerns what models can accomplish with harnesses, rather than model performance alone.

Generated by AI from reports · Updated 1 day ago