Taste-Bench tests AI agents’ judgment on decisions with long-term consequences

What happened

Researchers introduced Taste-Bench, a publicly available benchmark with 502 questions testing whether models can choose better paths in coding and AI research tasks before seeing later outcomes. Across 14 models, GPT-5.6 Sol led at 59.7 percent under a scoring rule requiring correct answers in both option orders, where random guessing scores 25 percent. Training a Qwen3.6-27B student improved its accuracy from 30 to 47.9 percent, and providing its advice to a separate agent raised success from 14.6 to 33.7 percent on 41 held-out SWE-bench Pro tasks.

The story so far

Researchers introduced Taste-Bench, a publicly available benchmark testing AI models’ choices before they see later outcomes; GPT-5.6 Sol led the 14 models evaluated with a score of 59.7 percent. The benchmark contains 502 questions about decisions in coding and AI research. Its scoring rule requires a model to answer correctly with the options presented in both orders, making the random-guessing baseline 25 percent. The researchers also trained a Qwen3.6-27B student model, raising its accuracy from 30 percent to 47.9 percent. When a separate agent received advice from that student, its success rate rose from 14.6 percent to 33.7 percent on 41 held-out SWE-bench Pro tasks.

Keep reading

Trending now

1. 1DeepSeek Releases V4.1-Flash-1017, Now Available on Ollama Cloud With US Hosting
2. 2Google DeepMind Says Four Gemini 4 Argon Agents Beat Minecraft Without Human Help
3. 3Meta Removes Business Terms Language That Raised Website Scraping and AI Training Concerns
4. 4Anthropic Says Claude Reconstructed Five NIST Elliptic-Curve Seeds Using a Timestamp-Initialized Generator