MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
summary
The gist
MultiAgentBench introduces a comprehensive benchmark designed to evaluate Large Language Model (LLM)-based multi-agent systems across diverse, interactive scenarios…
In short
MultiAgentBench is a benchmark testing how Large Language Models (LLMs) work together in complex, interactive tasks involving collaboration and competition. It uses the MARBLE framework to evaluate agent coordination across six scenarios like coding and social simulations. The results show that balancing high task performance with strong coordination skills leads to better overall success.
Key concepts
- MultiAgentBench
- A comprehensive benchmark designed to rigorously test LLM multi-agent systems. It moves beyond single-agent tests to measure how agents coordinate and compete effectively in diverse, interactive scenarios.
- MARBLE Framework
- The Multi-agent cooRdination Backbone with LLM Engine. This is the coordination system built on interconnected modules: a Coordination Engine for different protocols, a Cognitive Module for social intelligence, an Agent Graph Module to map relationships, and memory/communication/action modules.
- Coordination Score (CS)
- An aggregate metric used to assess collaboration quality. It combines various KPIs like task progress, structured planning scores, communication scores (Cscore), and a competition score that specifically measures performance in conflicting-goal tasks.
- Coordination Protocols
- Four distinct methods the MARBLE framework uses to structure agent interaction: star, tree, graph, and chain. These protocols dictate how agents are organized and how they communicate or interact with each other during a task.
Terminology used across episodes
This episode discusses
- MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents · Paper Radio
- GPT-4 Technical Report
- RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
- GameGPT: Multi-agent Collaborative Framework for Game Development
- Evaluating Large Language Models Trained on Code
- The Llama 3 Herd of Models · Paper Radio
- MindAgent: Emergent Gaming Interaction
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
- Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization
- MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
- Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
- Needs, Passions and Loot Boxes -- Exploring Reasons for Problem Behaviour in Relation to Loot Box Engagement
- AgentBench: Evaluating LLMs as Agents
- GAIA: a benchmark for General AI Assistants
- Generative Agents: Interactive Simulacra of Human Behavior
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Metrics for quantifying isotropy in high dimensional unsupervised clustering tasks in a materials context
- Gemini: A Family of Highly Capable Multimodal Models
- Diffusion Models Are Real-Time Game Engines
The paper
MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents · Read on arXiv
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian
University of Illinois Urbana-Champaign
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents".
Paper summary: Tom: So, to wrap up this paper "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents", it’s about creating a benchmark that looks beyond single agent tests. The authors introduce MultiAgentBench along with the MARBLE framework which tests how LLM based systems handle collaboration and competition across six diverse interactive scenarios.
Jane: They are using milestone-based key performance indicators, structured planning scores, communication scores Cscore, and a competition score to assess quality. This framework supports various coordination protocols like star, chain, tree, and graph topologies.
Lu: The paper highlights that the cognitive planning feature improves milestone achievement rates by three percent in some settings <ref:2503.01935#pg2>. They also show that graph-based protocols perform best among the tested coordination protocols in research scenarios.
Meng: The implication for us is that when we build multi-agent systems, we can’t just focus on making one agent super smart; we need to rigorously test how they talk to each other and plan their actions under different structural constraints.
Lalam: This paper provides a comprehensive tool for evaluating LLM based multi-agent systems by capturing coordination quality through structured planning, communication scores, and competition-driven assessments. It shows that reasoning alone isn't enough in complex adversarial settings where trust matters as much as pure logic.
Conclusion: Tom: So we've been looking at this MultiAgentBench paper about how these large language models work together, or compete, in complex tasks.
Jane: Exactly, and the title itself is quite descriptive—"Evaluating the Collaboration and Competition of LLM agents." It tells you right away this isn't just about one smart model doing one thing.
Lu: The authors are using a framework called MARBLE to set up these scenarios, which includes different ways agents can coordinate, like star or graph structures.
Meng: From my side, I’m looking at how this setup translates to real deployment; it seems like a lot of moving parts for any production system.
Lalam: I'm seeing that they are using several metrics—things like task completion scores and communication scores—to measure not just if the job got done, but *how* they did it together.
Tom: Which is interesting because most evaluations just look at the final answer, but this one tries to break down the actual interaction quality.
Jane: Right, so what's the big picture here? It seems they're trying to figure out what makes a multi-agent system actually good at things like teamwork and handling disagreements.
Lu: They found that different coordination methods work better for specific jobs; for instance, graph structures seemed particularly good when agents were doing research tasks.
Meng: I’m still skeptical about the generalizability of these results; if it only works well on specific types of tasks, how do we apply this to a completely different kind of workload?
Tom: It suggests that for building robust AI systems, we have to stop treating agents as isolated brains and start focusing heavily on how they communicate and plan their next move.
Jane: That’s the core idea—that coordination quality matters just as much as the individual intelligence of each agent.
Lalam: And what this means is that when we deploy these models, we need structured ways to test if those agents can actually collaborate effectively under pressure.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game