Sakana AI's self-improving model gets better at solving and judging, but can't catch its own blind spots

What happened

Researchers from Sakana AI and UC Berkeley introduced MASS, a method that uses copies of Qwen3.6-27B to solve tasks, evaluate results, and optimize workflows without a stronger teacher model. After two rounds, its test-task win rate against the base model reached 69.9 percent, although benchmark gains largely reflected more completed work and some scores declined. Shared blind spots allowed flawed results to pass evaluation and enter training, prompting the researchers to recommend independent checks, limits on autonomy, and clear intervention conditions.

The story so far

Sakana AI and UC Berkeley researchers introduced Multi-Agent Self-Supervision (MASS), a method that uses copies of Qwen3.6-27B to solve tasks, evaluate results and optimize workflows without a stronger teacher model; after two rounds, its test-task win rate against the base model reached 69.9 percent. However, benchmark gains largely reflected more completed work, and some scores declined. The method’s central limitation was that the model copies shared blind spots: flawed results could pass evaluation and enter training. The researchers therefore recommended independent checks, limits on autonomy and clear conditions for intervention. The results showed gains in solving and judging tasks, while also showing that self-evaluation could fail to catch errors shared across the copies.

Keep reading

Trending now

1. 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
2. 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
3. 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
4. 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights