Old Ideas, Novel Problems

The Instability of LLM-Based Novelty Evaluation

Noy Sternlicht1,2, Simra Shahid3, Peter Jansen2,4, Daniel S. Weld2,5, Pao Siangliulue2, Tom Hope1,2

1Hebrew University of Jerusalem, 2Allen Institute for AI, 3Microsoft, 4University of Arizona, 5University of Washington

Paper Code Dataset Cite

Can we trust LLM novelty judges?

The same idea pair is judged under three prompt variations: which was judged novel, which is novel, and which was judged more novel. A novelty judge scores each, and GPT-5.4 pairwise accuracy under each is 93.5, 40.9 and 59.7 on Human+Generated, versus 83.1, 76.0 and 80.5 on Human-Only.
Seemingly small changes, such as minor prompt variations, can wildly swing a judge's performance. Judges can grow markedly more brittle once LLM-generated ideas enter the data (Human+Generated), a failure that does not show up on the human-written papers they are typically tested on (Human-Only).

Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to LLMs. These judges are typically built ad hoc and validated, if at all, on human-written papers rather than on the generated ideas they are meant to score.

So, how do novelty judges perform? Not well.

We run a systematic controlled study of novelty evaluation design choices (prompt wording, retrieval, reasoning effort, idea format, and verdict aggregation), varying one at a time across six judges.

Small choices have large consequences. Simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical pairs it is shown, shifting pairwise accuracy by more than 50 points and occasionally pushing it below chance. The same change can help one judge and hurt another.

Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about the reported novelty gains of automated ideation systems, and call for more robust novelty evaluation methods.

Benchmark

Benchmark pipeline: an LLM extracts novelty signals from OpenReview submissions and reviews, filters them into validated novel and lower-novelty human-authored ideas, adds weakly labeled lower-novelty ideas from a vanilla LLM, and combines them into two data setups, Human-Only and Human+Generated.
Benchmark construction. We mine novelty labels automatically from ICLR 2026 reviews, keeping only submissions whose reviewers agree on the originality of the contribution, and complement them with weakly labeled lower-novelty ideas from a non-SOTA LLM ideator. The result is an open-source novelty evaluation dataset with two data setups, Human-Only and Human+Generated. Tap to enlarge.

Data statistics

Setup Ideas Eval. instances
D+ D− Pairwise Pointwise
Human-Only 154145ICLR 154299
Human+Generated 154154LLM 154308

D+ holds the novel ideas and D− the lower-novelty ones.

Both setups share the same D+ (154 ICLR-validated novel ideas) and differ in D−: validated lower-novelty ICLR ideas in Human-Only, LLM-generated ideas in Human+Generated.

Examples of extracted novelty signals

Praised as novel
It is the first work I have seen that attempts to constrain the diffusion process within the quotient space Quotient-Space Diffusion Models
The paper identifies and clearly defines a completely new, critical, and timely research problem Image Quality Assessment for Embodied AI
Criticized as not novel
The proposal is extremely close to NEAT (Zhong et al., 2025). Both methods replace LoRA's linear update with a nonlinear mapping… NoLoRA: Nonlinear Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
The contribution is primarily procedural integration of existing components (YOLOE + CLIP + GPT-4V + SAM2)… Consistent 3D Object Detection with Active LLM Reasoning

Controlled Study

Controlled study design: two judgment formats (pairwise, pointwise) crossed with six evaluation dimensions (judge instructions, related work access, reasoning effort, idea format, idea source, verdict aggregation) and six judge models from Anthropic and OpenAI, run one configuration at a time and reported on the Human-Only and Human+Generated setups.
Study design. Two judgment formats × six evaluation dimensions × six judges, varied one at a time and reported on both data setups. Tap to enlarge.
ChangeDescriptionProtocol
Judge promptreference: a novelty criterion defining what counts as novel
P1No guardrailsWeaker criterion, no warning against trivial combinationsboth
P2No criterionDrops the novelty definition entirelyboth
P3"Which was judged novel?"A prior review found exactly one idea novel; predict whichpairwise
P4"Which is novel?"P3 without the presumption of prior reviewpairwise
P5"Which was judged more novel?"A prior review found one idea more novel; predict whichpairwise
Evaluation setupreference: no retrieval, high reasoning, free-text abstracts, aggregated verdicts
+ RetrievalJudges also receive 5 abstracts of related workboth
Low reasoningReasoning effort lowered from high to lowboth
Idea formatIdeas rewritten as a two-field plan (purpose, mechanism)both
No aggregationA single call instead of three (pairwise: one per presentation order)both
GeneratorLower-novelty ideas regenerated by other modelsboth
Judges gpt-5.1gpt-5.2gpt-5.4 claude-sonnet-4-5claude-opus-4-5claude-opus-4-6

Results

Judges are highly unstable under configuration changes

On pairwise Human+Generated, asking which idea a prior review judged novel (P3) improves every judge, by more than 20 points in some cases. Yet P4, a close variant of the same prompt that makes no reference to prior human judgment, makes gpt-5.4 collapse below chance. Meanwhile, the fixes prior work commonly reaches for have a limited effect: lowering reasoning effort from high to low costs at most 7 points and is usually neutral, and retrieval improves only some judges, by at most 16 points.

Heatmap of pairwise strict accuracy change from the reference configuration for six judges across all controlled changes.
Pairwise judging, strict accuracy (ties count as failures). Cells report the change from the reference configuration, with the absolute value in parentheses. Tap to enlarge.

Changes do not move judges together

The same change might be neutral or even beneficial for one judge, but destructive for another. This is seen most clearly in pointwise judging: removing the guardrails from the novelty definition (P1) adds 15.7 points for claude-sonnet-4-5 on Human-Only, yet costs claude-opus-4-6 6.7. A configuration validated on one judge therefore carries no guarantee for the next. In line with prior work, pairwise judging is generally more stable than pointwise judging.

Heatmap of pointwise macro-F1 change from the reference configuration for six judges across the controlled changes, on Human-Only and Human+Generated.
Pointwise judging, macro-F1. Cells report the change from the reference configuration, with the absolute value in parentheses. claude-sonnet-4-5 exposes no reasoning_effort parameter, hence the blank entry. Tap to enlarge.

Judges are more uncertain on generated ideas

A pairwise judge ties when its aggregated verdicts favor neither idea. Depending on the configuration, even relatively strong judges like claude-opus-4-6 can tie on 35% of the pairs, and weaker ones like claude-sonnet-4-5 on more than half. Every judge ties more often on Human+Generated than on Human-Only, roughly twice as often for claude-opus-4-5 and claude-sonnet-4-5.

Dot plot of pairwise tie rates per judge, one dot per controlled configuration, for Human-Only and Human+Generated. Every judge's median tie rate is higher on Human+Generated.
Tie rates across controlled configurations. Each point is one configuration of the pairwise ablation for a given judge, with the vertical bar marking the median per setting. Tap to enlarge.

Stronger retrieval does not help

We replace our retrieval pipeline with a strong agentic RAG judge: gpt-5.6-sol at xhigh reasoning effort, searching the literature iteratively with no cap on searches. It reaches 0.78 macro-F1, 11 points behind claude-opus-4-6 with the simple five-abstract pipeline, and within noise of claude-opus-4-5 and gpt-5.4, despite costing 10× more per decision.

Bar chart of pointwise macro-F1 on Human+Generated for a strong agentic RAG judge versus judges with standard five-abstract retrieval.
Pointwise macro-F1 on Human+Generated, for the strong agentic RAG judge against judges that see five pre-retrieved abstracts. Tap to enlarge.

Dedicated novelty judges do not pay off

We compare prompted judges to two dedicated novelty judges, Idea Novelty Checker (Shahid et al., 2025) and AI-Scientist (Yamada et al., 2025). In every setting and under both backbones (claude-opus-4-6 and gpt-5.4), our cheapest prompted configuration outscores both by 8 to 29 macro-F1 points, while costing up to 32× less. The Pareto front consists entirely of simple prompted judges.

Cost versus pointwise macro-F1 for claude-opus-4-6 prompted configurations and two dedicated novelty judges, on Human-Only and Human+Generated.
Cost vs. pointwise macro-F1 for claude-opus-4-6, on Human-Only (left) and Human+Generated (right). The Pareto front (line) contains only prompted judges. Error bars: 95% CIs from a paired bootstrap. Tap to enlarge.

BibTeX

@misc{sternlicht2026oldideasnovelproblems,
      title={Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation},
      author={Noy Sternlicht and Simra Shahid and Peter Jansen and Daniel S. Weld and Pao Siangliulue and Tom Hope},
      year={2026},
      eprint={2610.02022},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.02022},
}