🔍 Read the full analysis: How To Evaluate Research Novelty With LLMs on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers tested six large language models as judges of scientific idea novelty and found that results could vary substantially with evaluation choices. In one pairwise test, small prompt changes shifted accuracy by more than 50 percentage points; retrieval and additional reasoning did not consistently improve performance. The team released a paper, benchmark data, code and a pipeline, while cautioning that some benchmark labels are imperfect.
Researchers at the Allen Institute for AI (Ai2) report that six large language models gave unstable judgments about whether scientific ideas were novel, with small changes to evaluation prompts shifting accuracy by more than 50 percentage points in one experiment, as detailed in the original analysis. The team released a benchmark, data, code and an automated pipeline, offering researchers tools to test whether LLMs can reliably screen ideas before further scientific work.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, along with Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The models assessed novelty through both absolute ratings and pairwise comparisons. The researchers report that changing a few words in a pairwise prompt produced a swing of more than 50 percentage points in accuracy. Adding literature retrieval or giving the models more reasoning did not consistently improve their judgments.
For one benchmark, the team selected 154 ideas from accepted ICLR 2026 papers in the top-rated 10% of their research areas. Reviewers had to praise an idea’s novelty, with no reviewer disputing it. The lower-novelty group contained 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised novelty. An annotator agreed with the direction of the extracted reviewer signal in 47 of 50 checks; the authors say the three disagreements did not change review classifications.
A second setting paired the 154 human-authored ideas with 154 ideas from a basic language-model generator. The authors treated generated ideas as less novel but describe those labels as weak, since each generated idea was not independently verified. In a separate check of 30 pairs—weighted toward examples the strongest judge had missed—an expert preferred the human-authored idea each time and judged none of the generated ideas novel. The authors also report that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Unstable Scores Matter
Research groups may use an LLM judge to sort candidate ideas before committing time to literature searches, experiments or expert review. If small prompt edits produce substantially different scores, a result may reflect the chosen evaluation setup as much as the idea’s originality. That makes scores harder to compare across projects and complicates decisions about which proposals deserve attention.
The findings also bear on tools designed to generate hypotheses or synthesize research. Such systems need a credible way to assess whether a proposed idea adds something beyond existing work. The study does not show that every LLM novelty assessment fails; it indicates that a strong result on one prompt or benchmark is not, by itself, evidence of dependable judgment. The released materials give other researchers a way to examine that reliability rather than relying on a single score.
Products Worth Considering
Sonos Era 100 SL Compact Microphone-Free Speaker, Black
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
As an affiliate, we earn on qualifying purchases.
How Ai2 Tested Novelty Judgments
A novelty judge is a model asked to assess whether an idea is new relative to prior research, either by rating it alone or comparing it with another idea. The Ai2 authors say there is no standard recipe: evaluations can differ in model, prompt, comparison format, reasoning budget and access to retrieved literature. They argue that judges are often tested on human-written papers even when the intended application is screening AI-generated proposals.
The benchmark uses reviewer comments as evidence about relative originality, but the study cautions against treating acceptance or rejection alone as a novelty label. Papers can be accepted or rejected for reasons other than originality. The authors also say some findings about disagreement across evaluation setups do not depend on the benchmark labels, though the supplied study summary does not detail how broadly that result holds across settings.
“Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”
— Ai2 study authors
Limits of the Benchmark Labels
The benchmark provides evidence about relative novelty, not a definitive measure of originality. Its human-authored examples were selected using reviewer comments and ranking criteria, which may not capture every relevant prior work or expert judgment. In the human-versus-generated setting, the generated ideas were labeled less novel without individual verification across the full set. The expert check covered 30 pairs, not every benchmark example.
The supplied material does not provide full results for every model, prompt, reasoning budget or evaluation format, or state the release date. It also does not establish whether the findings generalize to other academic fields, different idea-generation systems or judges with broader literature access. The evidence supports concern about sensitivity in the tested setups, but does not identify a single method that reliably resolves it.
Further Tests Across Research Fields
Ai2 has made the paper, benchmark data, code and automated pipeline available for researchers to build additional evaluation sets and compare approaches. Further testing can examine whether the reported instability appears with ideas from other disciplines, different generators and judge designs, and whether any combination of prompting, retrieval and model configuration produces more consistent results.
The release description gives no timetable for a follow-up study. Until additional evaluations clarify how well these findings travel beyond the tested examples, researchers using novelty scores can treat them as one input rather than a settled measure of originality. What remains open is whether changes to evaluation design can make LLM judgments dependable across different ideas and research settings.
Key Questions
What did the Ai2 researchers find?
They report that six LLMs’ judgments of scientific idea novelty varied across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.
Did literature retrieval or extra reasoning improve the judgments?
The authors report that adding literature retrieval or giving models more reasoning did not consistently improve their novelty judgments.
How reliable are the benchmark’s novelty labels?
The labels rely partly on reviewer comments and selection criteria, so they are evidence of relative novelty rather than a definitive standard. The authors describe the labels for generated ideas as weak because those ideas were not individually verified.
What has Ai2 released?
Ai2 released the study paper, benchmark data, code and an automated pipeline intended to help researchers create and evaluate additional test sets.
Primary source: Hugging Face · via ThorstenMeyerAI.com
