How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices
cs.AI
Submitted: 2026-10-08
Updated: 2026-10-08
Terminology
Sources
- OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
- SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
- SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model
- From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
- AI4Research: A Survey of Artificial Intelligence for Scientific Research
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
- OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources
- Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework
- GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation
- CiteBench: A benchmark for Scientific Citation Text Generation
- MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation
- Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
- ResearchGym: Evaluating Language Model Agents on Real-World AI Research
- BLADE: Benchmarking Language Model Agents for Data-Driven Science
- SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systems
- IdeaBench: Benchmarking Large Language Models for Research Idea Generation
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems