Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
cs.AI, cs.MA
Submitted: 2026-09-26
Updated: 2026-09-26
Comments: Submitted to ICLR 2027
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multi-agent debate (MAD) reportedly improves reasoning and factuality over single-model inference, but prior work treats agents as symmetric peers, leaving open what drives the gains.
Abstract
Multi-agent debate (MAD) reportedly improves reasoning and factuality over single-model inference, but prior work treats agents as symmetric peers, leaving open what drives the gains. We test the hypothesis that cognitive diversity among agents is the driver, in the setting where the question is still measurable: small open-weight models with benchmark headroom. Across 23 models from eleven vendor families, five tasks, and 5,500+ debate and control runs, we vary diversity along three axes - personas, sampling temperature, and model identity - pairing every debate configuration with a generation-budget-matched majority-vote control. The hypothesis is rejected on every axis. Debate beats single-agent inference (3--7 points where tasks have headroom) but at matched budget conditions it ties or even loses to self-consistency sampling at 1.6 times the wall-clock and 3.4 times the token cost. Persona prompting reduces accuracy and a dose-response experiment over each model's full combinatorial persona space shows the cost is a persona tax, not a diversity tax: redundant personas hurt most, while maximally-diverse teams recover part of the loss. Furthermore, mixed-model teams lose to majority votes over their own rosters, with accuracy tracking member capability rather than heterogeneity, and nearly all of debate's benefit comes from the first exchange of answers. We further identify a pervasive measurement hazard in which debate transcripts silently overflow serving context windows, whose correction alone moves our debate-versus-sampling comparison from-1.8 points to parity. Our results recast reported MAD gains as an ensemble-sampling effect and provide the budget-matched, contamination-checked baseline bar that future debate mechanisms should be required to clear.
Sources
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- Gemma: Open Models Based on Gemini Research and Technology
- Measuring Massive Multitask Language Understanding
- Debating with More Persuasive LLMs Leads to More Truthful Answers
- Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- More Agents Is All You Need
- DynaDebate: Breaking Homogeneity in Multi-Agent Debate with Dynamic Path Generation
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
- Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate
- Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems