Auditing AI peer reviewers: a benchmark on real scientific papers
cs.CL, astro-ph.IM, cs.AI, cs.DL
Submitted: 2026-11-12
Updated: 2026-11-12
Code: https://github.com/inigozubeldia/skepthical-benchmark
Terminology
Sources
- AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
- TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review
- MARG: Multi-Agent Review Generation for Scientific Papers
- Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework
- AIRA_2: Overcoming Bottlenecks in AI Research Agents
- LLM-REVal: Can We Trust LLM Reviewers Yet?
- Can large language models provide useful feedback on research papers? A large-scale empirical analysis
- ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors
- Agent Laboratory: Using LLM Agents as Research Assistants
- PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- The Denario project: Deep knowledge AI agents for scientific discovery
- FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
- Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- ReviewRL: Towards Automated Scientific Review with RL
- Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data