ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration
Guo Chen, Ziwen Li, Reed Li, Yu Lu, Haibo Shi, Bingbing Xu, Junjie Huang
cs.AI
Submitted: 2026-08-11
Updated: 2026-08-12
Code: https://github.com/WinstonCHEN1/ForestBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods.
Terminology
Abstract
Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods. Outcome-only benchmarks discard collaborations, whereas LLM-as-Judge evaluation requires additional, model-dependent inference and can vary with the LLM and rubric. We introduce a generalizable evaluation framework that maps native MAS traces into a shared space of unified collaboration graphs, enabling different methods to be evaluated under the same representation, reference set, and metric panel. Candidate graphs are compared with a query-specific reference forest. Each forest is a benchmark-provided collection of verified-success graphs: it records diverse ways in which representative MAS methods can complete the task, rather than prescribing a unique optimal process. Instantiating the framework as ForestBench, we filter 844 collaboration-necessary queries from seven public datasets, precompute ten successful target-conditioned reference graphs per query, and evaluate six representative MAS frameworks. Controlled backbone, reference-construction, and perturbation studies test the stability and scope of evaluation. Once the benchmark forests are built, ForestBench scores a trace in milliseconds without further LLM inference, providing a reusable structural basis for comparing diverse MAS collaboration traces.
Sources
- Evaluating Large Language Models Trained on Code
- The Trace Is the State: Exact Credit Assignment for LLM Agent Teams
- Training Verifiers to Solve Math Word Problems
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- LLM Multi-Agent Systems: Challenges and Open Problems
- Measuring Massive Multitask Language Understanding
- MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents
- Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization
- Debating with More Persuasive LLMs Leads to More Truthful Answers
- Adaptive Graph Pruning for Multi-Agent Communication
- CORE: Full-Path Evaluation of LLM Agents Beyond Final State
- MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
- Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents
- Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents
- FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data
- MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems