Ai2 study finds LLM judgments of research novelty are sensitive to evaluation setup
Can we trust LLMs to judge whether a research idea is novel?
- Who
- Ai2
- What
- released the Old Ideas, Novel Problems study, benchmark, data, code, and pipeline for evaluating LLM research novelty judges
- Quote
Today we’re releasing Old Ideas, Novel Problems , a new study of how reliably large language models can evaluate the novelty of scientific research ideas.
What happened
Ai2 released Old Ideas, Novel Problems, a study with a benchmark, data, code, and a reusable pipeline for evaluating research novelty judges. Changing a prompt reduced GPT-5.4’s accuracy from 93.5% to 40.9% on human versus AI-generated idea pairs, whose labels assume generated ideas are less novel. Retrieval and additional reasoning did not consistently improve results, and two specialized evaluators underperformed a simple prompted Claude Opus 4.6 baseline.
Why it matters
The study shows why evaluating novelty judges on human papers alone can miss prompt sensitivity on AI-generated ideas, offering a concrete basis for choosing evaluation data that matches intended use.
Read next
See the full storyAi2 Releases Benchmark Showing LLM Research Novelty Judgments Depend on Evaluation SetupThe story so far
Ai2 released Old Ideas, Novel Problems, a study and benchmark for evaluating how reliably large language models judge the novelty of scientific research ideas, and found that results were sensitive to the evaluation setup. The release includes data, code, and a reusable evaluation pipeline. In the study, changing a prompt reduced GPT-5.4’s accuracy from 93.5% to 40.9% on pairs of human and AI-generated ideas. Those accuracy scores depend on labels that assume the generated ideas are less novel. Ai2 also found that retrieval and additional reasoning did not consistently improve results, while two specialized evaluators performed worse than a simple prompted Claude Opus 4.6 baseline.
Keep up with AI, one email a day
Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-10-27