Google and Mercor Introduce an 85-Task Engineering Benchmark

1 reports 1 reporting sources Updated 3 hours ago

Understand the event

AI overview

Google and Mercor introduced Physical-Engineering's Last Exam (PLE), a benchmark of 85 engineering tasks for evaluating AI agents. On 56 hidden tests, Claude Opus 5.5 with Claude Code scored 48.3%. Across co-creation tasks, it satisfied 91% of individual constraints on average, but only 41.3% of design submissions met every hard safety and feasibility requirement. The results point to a gap between meeting constraints one at a time and delivering a viable design that satisfies them all.

Generated by AI from reports · Updated 1 hours ago

Latest development11/5 13:51 (UTC+8)

Google and Mercor’s PLE Exposes Constraint Bottlenecks in Engineering Agents Leiphone

Models in this story

Claude Opus 5.5#1 overall$4 input / $20 output per million tokens

API prices of all models

Report timeline

Follow the reports to explore different sides of the event.

11/5

13:51
Leiphone
Google and Mercor’s PLE Exposes Constraint Bottlenecks in Engineering Agents
Google and Mercor’s Physical-Engineering's Last Exam (PLE) evaluates 85 engineering tasks and finds a gap between local accuracy and viable design delivery. Claude Opus 5.5 with Claude Code scored 48.3% on 56 hidden tests; across co-creation tasks, it met 91% of individual constraints on average, while only 41.3% of design submissions met all hard safety and feasibility requirements. The article examines constraint coupling, design state management, and verification as challenges for engineering agents.

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.