PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

summary

Video file (mp4)

The gist

Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper.

In short

PaperAudit-Bench creates a benchmark to test how well automated peer review systems detect subtle errors in scientific papers. It uses a dataset of 220 papers from top AI conferences and a framework with three detection modes: Fast, Standard, and Deep. The findings show that structured error detection significantly improves review quality by forcing models to focus on evidence-based critiques.

Key concepts

PaperAuditDataset
This is a dataset of 220 research papers from top AI conferences (ICLR, ICML, NeurIPS 2025). Each paper is structured with section labels and contains about 15 injected errors. It's designed to be difficult by being long and having errors spread across different parts of the paper.
PaperAudit-Review Framework
This framework integrates error detection with review generation through three modes: Fast, Standard (using global memory for cross-section reasoning), and Deep (using multi-agent planning). The goal is to systematically shift reviewers toward more critical evaluations grounded in identifiable technical issues.
Lightweight Detectors Training
The system trains compact detectors using Supervised Fine-Tuning (SFT) on structured data, followed by Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO). This allows for training smaller models that achieve competitive detection performance while maintaining better efficiency and output discipline.
Evidence-Aware Review Generation
The review workflow uses the results from error detection to synthesize assessments. It moves beyond simple summarization by incorporating explicit error reports and novelty analysis, refining the baseline review under strict constraints to ensure critiques are strictly grounded in manuscript evidence.

Terminology used across episodes

This episode discusses

The paper

PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review · Read on arXiv

Institute of Automation, Chinese Academy of Sciences. · University of Chinese Academy of Sciences. · Wenge Technology

Large language models (LLMs) can generate fluent peer reviews but often miss sparse, subtle, and document-level scientific errors. We introduce PaperAudit-Bench, a benchmark for evaluating paper-level error detection and error-aware automated review. It comprises (1) PaperAudit-Dataset, 220 high-quality AI/ML papers containing more than 2,200 controlled synthetic errors across eight categories, and (2) PaperAudit-Review, an evidence-aware framework that links structured error detection to review generation. Experiments across models, error categories, paper sections, and analysis depths show that sparse long-context auditing remains challenging for current LLMs. Conditioning review generation on explicit, manuscript-grounded error evidence produces more conservative and discriminative assessments that align more closely with human judgments than representative automated-review baselines. The dataset also supports post-training compact detectors through SFT and RL; Qwen3-14B achieves 55.8% F1 at Detection@1 under a fixed held-out protocol. A PubPeer transfer study and deployment in ScienceOne, a scientific foundation-model platform led by the Chinese Academy of Sciences, further illustrate practical applicability. Code and Dataset Page: https://github.com/TU2021/PaperAudit ScienceOne Manuscript Review: https://www.scienceone.cn/lit/#/assistant

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review".

Jane: Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The title of the paper is PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review, and the authors are Songjun Tu, Yiwen Ma, Jiahao Lin, Qichao Zhang, Xiangyuan Lan, Junfeng Li, Nan Xu, and Linjing Li.

Jane: That sounds like a very specific system for testing the critical assessment abilities of AI review tools. It suggests they aren't just looking for surface-level mistakes in the writing.

Lu: The key idea is that existing models often generate fluent reviews but miss substantive issues that are hard to find when they are distributed across different parts of a paper, and this benchmark is designed to expose those weaknesses under long context settings.

Meng: So it's testing if an AI can spot something like a logical flaw in section one and then connect it to missing data in section three, which is what the authors call cross-section reasoning.

Lalam: The implication here is that we need benchmarks that specifically look for these kinds of distributed errors because they are where the real difficulty lies for current LLM review systems.

The paper's summary: Tom: The PaperAudit-Bench introduces two main things: first, the PaperAuditDataset, which is this error dataset covering both individual section errors and those needing cross-section reasoning under long context settings.

Jane: And the second part is the PaperAudit-Review framework, which integrates structured error detection with evidence-aware review generation to help support that critical assessment process.

Lu: They are using this dataset, which is built from papers accepted at top AI conferences like ICLR and ICML in two thousand twenty-five and they have about fifteen injected errors per paper on average.

Meng: The framework itself has three detection modes: Fast Mode for a quick scan, Standard Mode with explicit section awareness and a shared global memory, and Deep Mode which uses planning and multi-agent detection for specialized analysis.

Lalam: And the results show that using structured detection helps shift AI reviewers toward more critical evaluations grounded in identifiable technical issues, particularly with respect to technical quality and argumentative rigor.

The paper's improvements: Tom: They're not just presenting this idea; they are showing how they can train lightweight detectors using supervised fine-tuning and reinforcement learning on the PaperAuditDataset instead of needing massive large language model detectors.

Jane: Training these compact detectors involves SFT on structured data to map corrupted content to error descriptions, followed by RL using Group Relative Policy Optimization with a multi-dimensional reward function that captures accuracy, conciseness, and penalties for abnormal behavior.

Lu: This process yields compact detectors that have competitive detection performance while also improving output discipline compared to larger models.

Meng: From a deployment standpoint, having a lightweight detector is good because it means lower computational costs when you run the error detection part of the review pipeline.

Lalam: So the improvement lies in making these detectors efficient and disciplined, allowing us to integrate explicit error detection into the review workflow without requiring enormous model resources.

Conclusion: Tom: To wrap up, PaperAudit-Bench is presented as a unified benchmark for evaluating paper-level error detection and its interaction with automated peer review. The experiments show that structured detection changes review behavior in a more conservative and evidence-grounded manner.

Jane: So the main point is that this setup helps foster more critical and evidence-grounded automated peer review by using controlled benchmarks to see how these systems perform when they have to reason across the entire paper.

Lu: The work provides a reference point for future studies on designing error-aware automated reviewing systems, specifically focusing on how to expose distributed errors effectively.

Meng: I think the shift toward explicit error detection means that an AI reviewer is less likely to just be polite and more likely to actually challenge the technical claims based on detected flaws.

Lalam: I agree, and this whole PaperAudit-Bench setup gives researchers a concrete way to design these systems better moving forward.

More episodes

← Home