Ai2 Releases Benchmark Showing LLM Research Novelty Judgments Depend on Evaluation Setup

Ai2 released Old Ideas, Novel Problems, a study and benchmark for evaluating how reliably large language models judge the novelty of scientific research ideas, and found that results were sensitive to the evaluation setup. The release includes data, code, and a reusable evaluation pipeline. In the study, changing a prompt reduced GPT-5.4’s accuracy from 93.5% to 40.9% on pairs of human and AI-generated ideas. Those accuracy scores depend on labels that assume the generated ideas are less novel. Ai2 also found that retrieval and additional reasoning did not consistently improve results, while two specialized evaluators performed worse than a simple prompted Claude Opus 4.6 baseline.

Generated by AI from reports · Updated 53 minutes ago

There is not enough continuous observation data to plot a trend yet.