ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces
cs.CL
Submitted: 2026-11-10
Updated: 2026-11-10
Project page: https://arebench.com
Terminology
Sources
- SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
- K-Bench: measuring model performance on real scientific agent requests
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
- OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
- ResearchGym: Evaluating Language Model Agents on Real-World AI Research
- REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- EXP-Bench: Can AI Conduct AI Research Experiments?
- RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows
- CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
- Coding-Agent Benchmarks Should Match Their Users' Task Flows
- PaperBench: Evaluating AI's Ability to Replicate AI Research
- Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
- ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
- RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
- SWE-Together: Evaluating Coding Agents in Interactive User Sessions
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data