What happened
Epoch AI tested GPT-6.2 Astra and other recent models on open-ended Epoch Automation Reports tasks, with an Epoch employee scoring their outputs against internal quality standards. All models fell short of the 100% employee standard: Fable 5.2 averaged 82%, while GPT-6.2 Astra ranked fourth at 65%, despite sharing the top ECI score of 174 with Fable 5.2.
Can the most capable LLMs match humans at real work? Not yet, at least not on the kind of work we do at Epoch.
We tested GPT-6.2 Astra and other recent models on Epoch Automation Reports, our benchmark of open-ended Epoch work tasks, ranging from making Epoch-style diagrams to designing basic research experiments.
An Epoch employee scores the models’ outputs based on our internal quality standards. A score of 100% represents the standard expected of an Epoch employee.
All models failed to meet the Epoch employee standard. GPT-6.2 Astra ranks fourth (averaging 65%), despite sharing the top ECI score of 174 with Fable 5.2.
Fable 5.2 scored highest, averaging 82%.
On one task, both Fable 5.2 and Opus 5.5 matched the level of a contractor. More on that below.
The story so far
OpenAI released the GPT-6.2 Astra and Sol models, which are available now in ChatGPT Work, Codex, and the API. Astra, positioned as the company’s new flagship model, offers reduced pricing with API rates 20% lower than its predecessor at $8 per million input tokens and $40 per million output tokens, while Sol maintains the existing $2 and $10 pricing structure. OpenAI also expanded access to its Ultrafast mode to include Pro 200 and Pro 500 subscription tiers. The company reports that GPT-6.2 Astra sets new highs on benchmarks including Agents' Last Exam, BenchCAD 2.1, and Terminal-Bench Science 0.1 at lower task costs. However, external evaluations show mixed results. Epoch AI reported that GPT-6.2 Astra scored 65% on internal automation quality standards, trailing Fable 5.2. Additionally, independent testing by Simon Willison indicated that the model struggled with animation tasks compared to other frontier models. OpenAI also implemented tighter cybersecurity safeguards for the new models and plans to provide access through its Daybreak platform for approved defenders, though the company noted that these safeguards may increase refusal rates for some security research tasks.
How the story developed · 4
View full event- 11/18 06:16 (UTC+8)
Epoch AI Finds GPT-6.2 Astra Falls Short of Employee Standard on Work Tasks This report
- 11/18 06:15 (UTC+8)Tibor Blaho Tests GPT-6.2 Astra and Sol on Daily WorkTibor Blaho · X
- 11/18 06:15 (UTC+8)Perplexity Shares GPT-6.2 Astra and Sol WANDR EvalsAravind Srinivas · X
- 11/18 02:36 (UTC+8)OpenAI Introduces GPT-6.2 Astra and SolOpenAI News
