MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents".
Paper summary: Tom: So, to wrap up this paper "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents", it’s about creating a benchmark that looks beyond single agent tests. The authors introduce MultiAgentBench along with the MARBLE framework which tests how LLM based systems handle collaboration and competition across six diverse interactive scenarios.
Jane: They are using milestone-based key performance indicators, structured planning scores, communication scores Cscore, and a competition score to assess quality. This framework supports various coordination protocols like star, chain, tree, and graph topologies.
Lu: The paper highlights that the cognitive planning feature improves milestone achievement rates by three percent in some settings <ref:2503.01935#pg2>. They also show that graph-based protocols perform best among the tested coordination protocols in research scenarios.
Meng: The implication for us is that when we build multi-agent systems, we can’t just focus on making one agent super smart; we need to rigorously test how they talk to each other and plan their actions under different structural constraints.
Lalam: This paper provides a comprehensive tool for evaluating LLM based multi-agent systems by capturing coordination quality through structured planning, communication scores, and competition-driven assessments. It shows that reasoning alone isn't enough in complex adversarial settings where trust matters as much as pure logic.
Conclusion: Tom: So we've been looking at this MultiAgentBench paper about how these large language models work together, or compete, in complex tasks.
Jane: Exactly, and the title itself is quite descriptive—"Evaluating the Collaboration and Competition of LLM agents." It tells you right away this isn't just about one smart model doing one thing.
Lu: The authors are using a framework called MARBLE to set up these scenarios, which includes different ways agents can coordinate, like star or graph structures.
Meng: From my side, I’m looking at how this setup translates to real deployment; it seems like a lot of moving parts for any production system.
Lalam: I'm seeing that they are using several metrics—things like task completion scores and communication scores—to measure not just if the job got done, but *how* they did it together.
Tom: Which is interesting because most evaluations just look at the final answer, but this one tries to break down the actual interaction quality.
Jane: Right, so what's the big picture here? It seems they're trying to figure out what makes a multi-agent system actually good at things like teamwork and handling disagreements.
Lu: They found that different coordination methods work better for specific jobs; for instance, graph structures seemed particularly good when agents were doing research tasks.
Meng: I’m still skeptical about the generalizability of these results; if it only works well on specific types of tasks, how do we apply this to a completely different kind of workload?
Tom: It suggests that for building robust AI systems, we have to stop treating agents as isolated brains and start focusing heavily on how they communicate and plan their next move.
Jane: That’s the core idea—that coordination quality matters just as much as the individual intelligence of each agent.
Lalam: And what this means is that when we deploy these models, we need structured ways to test if those agents can actually collaborate effectively under pressure.
Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Zhe Wang, Zhenhailong Wang, Cheng Qian
University of Illinois Urbana-Champaign
cs.MA, cs.AI, cs.CL, cs.CY
Submitted: 2025-03-03
Updated: 2026-11-18
Code: https://github.com/MultiagentBench/MARBLE
Importance score: 100/100
The gist: MultiAgentBench introduces a comprehensive benchmark designed to evaluate Large Language Model (LLM)-based multi-agent systems across diverse, interactive scenarios…
Key concepts
- MultiAgentBench
- A comprehensive benchmark designed to rigorously test LLM multi-agent systems. It moves beyond single-agent tests to measure how agents coordinate and compete effectively in diverse, interactive scenarios.
- MARBLE Framework
- The Multi-agent cooRdination Backbone with LLM Engine. This is the coordination system built on interconnected modules: a Coordination Engine for different protocols, a Cognitive Module for social intelligence, an Agent Graph Module to map relationships, and memory/communication/action modules.
- Coordination Score (CS)
- An aggregate metric used to assess collaboration quality. It combines various KPIs like task progress, structured planning scores, communication scores (Cscore), and a competition score that specifically measures performance in conflicting-goal tasks.
- Coordination Protocols
- Four distinct methods the MARBLE framework uses to structure agent interaction: star, tree, graph, and chain. These protocols dictate how agents are organized and how they communicate or interact with each other during a task.
Terminology
Summary
MultiAgentBench introduces a comprehensive benchmark designed to evaluate Large Language Model (LLM)-based multi-agent systems across diverse, interactive scenarios, moving beyond single-agent or narrow domain evaluations to capture the dynamics of multi-agent coordination and competition. p. 0 The framework is supported by the MARBLE (Multi-agent cooRdination Backbone with LLM Engine) framework, which itself is a robust multi-agent coordination system built on interconnected modules: a Coordination Engine orchestrating four distinct protocols (star, tree, graph, and chain); a Cognitive Module for maintaining internal state and social intelligence; an Agent Graph Module to structure agent relationships; a Memory Module for storing shared and individual memories (including RAG); a Communication Module to manage external interactions; and an Action Module to execute plans. p. 1
The benchmark covers six diverse interactive scenarios, including collaborative coding, gaming (Minecraft building), research tasks, social simulations (Werewolf and Bargaining), and database anomaly investigation. p. 8 Scenarios are curated either by adapting prior multi-agent tasks or generating LLM-based tasks with human verification, focusing on agents with mutual goals in task-oriented settings and conflicting goals in social simulations. p. 3
MultiAgentBench employs novel metrics to assess not only task completion but also the quality of collaboration and competition. p. 0 These include milestone-based Key Performance Indicators (KPIs) that track progress and individual contributions, structured planning scores, communication scores (Cscore), and a dedicated competition score capturing conflicting-goal tasks, internal performance metrics, and competitive aspects in planning and communication. p. 1 The overall Coordination Score (CS) aggregates these metrics. p. 5
Experimental results indicate that while intrinsic model capabilities are primary drivers of task performance, coordination plays a significant role; models balancing high Task Scores (TS) with moderate-to-high Coordination Scores demonstrate more robust performance. p. 5 Specific findings highlight that graph-based protocols excel in research scenarios, and cognitive evolving planning improves milestone achievement rates by 3%. p. 0 Case studies reveal emergent social behaviors, such as the importance of explicit cooperative actions in Werewolf environments, where success hinges on trust and coordination among villagers. p. 1 Furthermore, the analysis of decision-making failures in negotiations demonstrates that a lack of mutual trust and cooperative spirit severely impacts outcomes, suggesting that reasoning alone is insufficient in complex adversarial settings.
The MARBLE framework supports various coordination protocols (star, chain, tree, graph) and innovative strategies like group discussion and cognitive planning. p. 0 The benchmark curation also involves evaluating different model capabilities; for instance, gpt-4o-mini reached the average highest task score across scenarios. p. 5 Performance metrics vary by scenario: in Minecraft simulations, Llama3. p. 81-70B achieved a higher completion ratio (0. p. 43754) and a positive net score (0. p. 04511) in full-run simulations compared to gpt-4o-mini (ratio around 0. p. 525). In bargaining scenarios, gpt-4o mini achieved the highest Bargaining Score (3. p. 6710). The paper concludes that MultiAgentBench and MARBLE provide a comprehensive tool for rigorously evaluating LLM multi-agent systems by capturing coordination quality through structured planning, communication scores, and competition-driven assessments. p. 1
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve task execution through cognitive planning by implementing a
cognitive self-evolving planning method
whichmirrors human learning by generating expected outcomes and progress for each task, storing these in memory, and then comparing actual performance against these expectations in subsequent iterations.
-
Enhance coordination quality by utilizing graph-based coordination protocols such as the graph structure, as experiments revealed that this
performs the best among coordination protocols in the research scenario.
-
Improve strategic decision-making in adversarial settings by focusing on villager-centric scoring rationale, which emphasizes
cooperative roles, such as Witch and Guard,
to capturehow they can harness cooperative roles (e.g., Witch, Guard) and voting strategies more effectively.
-
Enhance negotiation performance by leveraging models that excel in seller negotiations, as the text notes that
Seller scores are consistently higher than Buyer scores across all models.
-
Improve complex problem-solving in technical domains by employing a framework where agents can leverage tools like code editors, search engines, and debuggers within a structured environment to
uncover the root cause behind the anomaly.
Sources
- GPT-4 Technical Report
- RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
- GameGPT: Multi-agent Collaborative Framework for Game Development
- Evaluating Large Language Models Trained on Code
- The Llama 3 Herd of Models
- MindAgent: Emergent Gaming Interaction
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
- Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization
- MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
- Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
- Needs, Passions and Loot Boxes -- Exploring Reasons for Problem Behaviour in Relation to Loot Box Engagement
- AgentBench: Evaluating LLMs as Agents
- GAIA: a benchmark for General AI Assistants
- Generative Agents: Interactive Simulacra of Human Behavior
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Metrics for quantifying isotropy in high dimensional unsupervised clustering tasks in a materials context
- Gemini: A Family of Highly Capable Multimodal Models
- Diffusion Models Are Real-Time Game Engines
Related papers
- Iterative Negotiation and Oversight: A Case Study in Decentralized Air Traffic Management
- Towards using Large Language Models for Risk Negotiation and Consensus-building in One Health contexts
- A-MapReduce: Executing Wide Search via Agentic MapReduce
- AgentCue: Statistical Summaries for Guiding Decentralized Multi-Agent Self-Organization
- Events as Triggers for Behavioral Diversity in Multi-Agent Reinforcement Learning
- Highway Congestion Reduction through Reinforcement Learning Based Eulerian Headway Control