Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists
cs.AI
Submitted: 2026-10-04
Updated: 2026-10-04
Sources
- Holistic Evaluation of Language Models
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research
- BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology
- Reasoning Models Don't Always Say What They Think
- BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
- Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
- scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology
- R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science
- Measuring Massive Multitask Language Understanding
- Humanity's Last Exam
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
- PaperBench: Evaluating AI's Ability to Replicate AI Research
- scBench: Evaluating AI Agents on Single-Cell RNA-seq Analysis
- Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
- Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems