Misinformation Propagation in Benign Multi-Agent Systems

arXiv:2606.16710 · cs.MA, cs.CL · Submitted 2026-06-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Misinformation Propagation in Benign Multi-Agent Systems".

Jane: The gist The reliability of multi-agent systems becomes uncertain when some agents operate under incorrect information, and misinformation can degrade single-agent performance while persisting across multi-agent debate,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at a paper called "Misinformation Propagation in Benign Multi-Agent Systems". The main idea here is that multi-agent systems, where LLM agents talk to each other to solve problems, can get unreliable when some of those agents are fed the wrong information. This study looks at this risk across reasoning, knowledge tasks and even moral judgment.

Jane: That sounds like it could be a problem in real life where these systems are used for things like medical advice or legal analysis. So what exactly is the core claim of this paper? What's the big picture they're trying to show us?

Tom: They set up an experiment injecting intent-based misinformation into both single agent and multi-agent setups. The key finding is that this misinformation can degrade performance for a single agent, but it also shows strong persistence across multi-agent debate. Agents often keep answers introduced by agents who were given the wrong info.

Lu: It’s interesting because they specifically look at benign systems, meaning the agents aren't trying to deceive each other on purpose; it’s just context that gets messy. The paper investigates how group composition and decision protocols influence this robustness.

Meng: From a deployment standpoint, that persistence is concerning. If an agent gets fed bad info and keeps using it in a debate, how do we even deploy that system knowing the information might be corrupted? What does it cost to run these kinds of debates at scale?

Lalam: I see the paper focuses on how context matters across different tasks like commonsense reasoning, knowledge QA and moral judgment. They use a dataset called MINT, which has ten thousand two hundred seventy-eight LLM-written misinformation passages across nine intent-based categories.

Tom: The results show that a relevant false passage costs a single agent between zero point one two and zero point four three accuracy. But irrelevant or content-free passages only cost at most zero point zero six accuracy for the single agent.

Jane: That's a significant drop, right? So, if we look at the multi-agent side, how does that compare to the single agent performance degradation they found? Is it less bad when there are multiple agents involved?

Tom: It is less bad. The degradation in multi-agent setups was between negative two point two and negative ten point three percent. That contrasts with the single agent setups where you saw a drop between negative twelve point nine and negative seventeen point two percent.

Lu: They attribute that robustness to the self-refinement loop and divergent thinking that happens during iterative multi-agent debate. It suggests that the interaction itself provides a buffer against simple misinformation leakage.

Meng: So, if we have one of three misinformed agents in a debate, they aren't doing much better than just picking the majority sample over equally exposed independent samples. That’s not great for reliability when you need high accuracy.

Paper summary: Lalam: They also tested two decision-making protocols: consensus and voting. When using voting, interaction helped recover up to zero point two four accuracy under the misinformed majority, but it dropped to zero point two nine under consensus decision-making.

Tom: And a one-sentence unreliability warning helps recover up to zero point one eight accuracy when agents use voting, but that recovery drops significantly under consensus decision-making, down to zero point two eight. It also recovers up to zero point one eight accuracy when the context is actually true and agents are voting.

Jane: It sounds like the structure of how they make decisions really matters for how much misinformation hurts the final outcome. So what's the big picture here regarding system design?

Tom: The conclusion is that single LLM agents are vulnerable to contextual misinformation across reasoning, knowledge, and ethical alignment tasks. But multi-agent debate partially mitigates this effect when there are enough uninformed agents around to counter those misleading signals.

Lu: Robustness depends not just on the model itself but heavily on the group composition and the decision protocol they choose. Voting gives strong absolute accuracy but it can be more sensitive to misinformed peer pressure, whereas consensus is more stable under misinformation exposure.

Meng: So for someone actually deploying this, it sounds like you'd lean toward a consensus-based protocol if you anticipate some level of context contamination during the debate. That makes sense from a risk management viewpoint.

Lalam: And they point out that they tested two open-weight models, Llama three point three and GLM four point seven, and they used machine-generated misinformation for their MINT dataset to test their claims. The paper notes that generated misinformation might differ from humanwritten or adversarially optimized versions of the same type of text.

Jane: That's an important caveat about the data source because it affects how we interpret the results on Llama three point three and GLM four point seven specifically, right?

Tom: Exactly. And they also noted that their multi-agent setups used fixed debate structures and only two decision-making protocols, consensus and voting in their primary comparisons. That limits the general applicability of those specific findings to more complex or varied system designs.

Lu: The paper is doing a controlled analysis of how context affects benign multi-agent debate, but they admit that this bounds the scope of their conclusions. They aren't testing every possible way misinformation could enter an agent system.

Meng: So what’s the real world implication? It means if we build these systems for high stakes, we need to be careful about how we structure the team and how they decide things, not just relying on the base model quality alone.

Jane: So to wrap up, this paper is showing that multi-agent debate isn't a perfect shield against misinformation; it helps a little bit but only when the group composition and decision protocols are handled correctly. It’s about designing systems that are resilient to context errors across their entire interaction structure.

Conclusion: Tom: So, this paper is about how misinformation spreads when multiple AI agents are debating things out loud.

Jane: Right, so we saw that even in these benign setups, some agents get fed wrong information and it sticks during the debate itself.

Lu: The authors are looking at this specifically to see how group composition and the way they decide things changes how much of a problem it is.

Meng: So what does this mean for us when we try to deploy these multi-agent systems in real applications? What's the actual risk there?

Lalam: Essentially, single agents are pretty fragile against context errors across reasoning and ethics tasks, but the debate structure offers some protection.

Tom: The main point is that robustness isn't just about the model; it’s about how many uninformed agents you have and what decision protocol you use.

Jane: So if a team of agents is debating, how does their way of making a final call—like voting versus consensus—affect whether they catch that bad information?

Lalam: Voting gives strong absolute accuracy, but it can be sensitive to those misinformed peer pressures we talked about earlier.

Lu: Consensus decision-making seems more stable when there's some misinformation in the mix, which is a surprising finding.

Tom: It means if you need reliability in a multi-agent system, you have to design the structure carefully around how those agents interact and decide on an answer.

Jane: It shows that building these systems isn't just about training a smart model; it’s about designing resilient communication patterns for the agents involved.

Meng: So what we really need to watch is those decision protocols, because they seem to be a bigger factor than just having more agents in the room.

Jonas Becker, Jan Philip Wahle, Terry Ruas, Bela Gipp

University of Göttingen · LKA NRW

cs.MA, cs.CL

Submitted: 2026-06-15

Updated: 2026-10-14

Comments: 64 pages, 12 figures, 31 tables

Code: https://github.com/karpathy/autoresearch

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist The reliability of multi-agent systems becomes uncertain when some agents operate under incorrect information, and misinformation can degrade single-agent performance while persisting across

Key concepts

Multi-Agent Debate (MAD): This involves multiple LLM agents interacting sequentially to solve a problem through discussion and reasoning. Agents exchange intermediate steps, challenge each other's ideas, and collectively work toward a final decision. The study examines how misinformation impacts this collaborative process.
Misinformation Relevance: This refers to whether the injected false information is actually related to the task at hand or completely irrelevant noise. The experiment tested both task-relevant and irrelevant passages to see if the impact on agent performance differed based on the quality of the misleading context provided.
Consensus vs. Voting Protocols: These are different ways a group of agents can reach a final decision. Consensus means the last agent decides based on all prior inputs, while voting involves agents conducting a majority vote at the end. The study found that consensus-based decision-making is more stable when misinformation is present.
Robustness to Misinformation: This measures how well a multi-agent system maintains its performance when some agents are exposed to incorrect information. The results indicate that MAD can mitigate performance degradation compared to single agents, but the level of protection depends on the group's structure and how they decide.

Terminology

Multi-Agent System — 4× in this paper · explained in 7 episodes
Robustness — 2× in this paper · explained in 5 episodes
Agent Interactions — 1× in this paper
Alignment — 1× in this paper

Summary

The gist The reliability of multi-agent systems becomes uncertain when some agents operate under incorrect information, and misinformation can degrade single-agent performance while persisting across multi-agent debate, although group composition and decision protocols influence robustness > ref:2606.16710

Introduction

Multi-agent systems (MAS) based on large language models (LLMs) can solve complex problems through debate and task decomposition > ref:2606.16710 Collaborative structures have been proposed as a way to improve reasoning quality, enable task specialization, and increase robustness compared to single-agent systems > ref:2606.16710 MAS often rely on agent-user interactions and on multiple agents exchanging intermediate reasoning steps, challenging each other, and collectively arriving at a final decision > ref:2606.16710 The reliability of MAS becomes uncertain when some agents operate under incorrect information > ref:2606.16710 Inaccurate knowledge may arise from several sources, such as retrieval augmented generation, web search, hallucinations generated by the underlying model, misinterpreted information by an agent, noisy or manipulated external information sources, or adversarial agents within the system > ref:2606.16710 The consequences of misinformation can be particularly concerning in high-stakes scenarios > ref:2606.16710 For example, in a healthcare decision-support setting, a misinterpreted detail from one agent could lead to an incorrect diagnosis or treatment recommendation > ref:2606.16710 Prior work has focused on the effects of misinformation in single LLMs or in malicious settings where an adversarial or manipulated agent persuades agents in a debate > ref:2606.16710 In this work, we investigate how contextual misinformation influences the outcome of fully benign multi-agent debate (MAD) > ref:2606.16710 Our experiment setup injects misinformation into LLM agents and assesses its effects on individual-agent behavior and multiagent collaborative reasoning > ref:2606.16710 We release all code, prompts, and data > ref:2606.16710 Alongside our investigations, we publicly release MINT (Misinformation INTents), an LLM-generated dataset comprising 10,278 misinformation texts across nine intent-based categories > ref:2606.16710 We address the following research questions: RQ1 How does exposure to misinformation affect the downstream task performance of a single LLM agent? (Section 4.1) > ref:2606.16710 RQ2 How susceptible are LLM agents to misinformation during multi-agent interaction? (Section 4.2) > ref:2606.16710 RQ3 How do group composition and decision-making protocols influence robustness to misinformation? (Section 4.3) > ref:2606.16710 Our work shows that fully benign MAD can be vulnerable to contextual misinformation, but the extent of this vulnerability depends on decision-making and group composition > ref:2606.16710

Methodology

Our methodology isolates how contextual misinformation affects individual and collective reasoning by comparing single-agent prompting with multiagent debate (MAD) under uninformed and misinformed conditions > ref:2606.16710 We vary three factors: misinformation relevance (i.e., task-relevant and irrelevant misinformation), group composition (i.e., number of misinformed agents), and decision protocol (i.e., comparison of consensus with voting decision-making) > ref:2606.16710 Misinformation is generated for each example using nine intent-based categories and appended directly to the prompt > ref:2606.16710 We experiment with single-agent and multi-agent setups under uninformed and misinformed conditions > ref:2606.16710 Single-Agent evaluation involves exposing a single LLM to relevant or irrelevant misinformation in the prompt, measuring performance across reasoning, knowledge, and ethical alignment tasks > ref:2606.16710 Multi-Agent experiments run with three LLM agents for a total of five turns, inspired by the work of Du et al. (2024) and Becker et al. (2025) > ref:2606.16710 Misinformation is appended to a number of benign agents depending on the configuration tested > ref:2606.16710 We also compare two decision-making protocols, consensus and voting > ref:2606.16710 Consensus means that the last agent in the chain can decide the final solution based on previous agents’ contributions and reasoning, while voting means that agents conduct a majority voting at the end of the debate based on previous agents’ contributions and reasoning > ref:2606.16710

Results

Exposure to misinformation generally affects task performance, but the degradation is smaller in multi-agent setups (-2.2% to-10.3%) than in single-agent setups (-12.9% to-17.2%) > ref:2606.16710 This indicates that MAD is more robust to misinformation, which could be attributed to the self-refinement loop and divergent thinking in iterative MAD > ref:2606.16710 Misinformation persistence shows strong persistence of answers from misinformed agents on CWQ and WinoGrande, where the average deltas are-10.4% and-7.7%, respectively > ref:2606.16710 Framing and rumors are comparatively less persistent, whereas hoaxes and unconstrained misinformation are among the most persistent categories > ref:2606.16710 Consensus-based decision-making is beneficial when the underlying model shows sensitivity to misinformed peers > ref:2606.16710

Conclusion

Single LLM agents are vulnerable to contextual misinformation across reasoning, knowledge, and ethical alignment tasks, and such misinformation can persist during benign multi-agent debate > ref:2606.16710 While misinformation degrades task performance, multi-agent debate partially mitigates this effect, especially when enough uninformed agents are present to counter misleading signals > ref:2606.16710 Robustness against misinformation depends not only on the model, but also on group composition and decision protocol > ref:2606.16710 Voting achieves strong absolute accuracy but can be more sensitive to misinformed peer pressure, whereas consensus is more stable under misinformation exposure > ref:2606.16710

Limitations

Our study is designed as a controlled analysis of how contextual misinformation affects benign multi-agent debate, but it also bounds the scope of our conclusions > ref:2606.16710 First, we evaluate two open-weight models, Llama-3.3 and GLM-4.7 > ref:2606.16710 Second, MINT uses machine-generated misinformation, but generated misinformation may differ from humanwritten or adversarially optimized misinformation > ref:2606.16710 Third, our multi-agent setups use fixed debate structures and two decision-making protocols > ref:2606.16710

MINT Dataset Construction

The MINT dataset builds on existing corpora, including CWQ, Ethics, and WinoGrande by augmenting their samples with sample-specific, intent-based misinformation texts > ref:2606.16710 We prompt Llama-3.3-70B-Instruct to generate statements according to nine intent-based categories derived from prior work on misinformation typologies > ref:2606.16710 We add an “irrelevant true information” control by randomly sampling a sentence-bounded passage of similar length from the English Wikipedia dump of 2023-11 > ref:2606.16710 To validate the machine-generated MINT dataset, we conduct a human audit where three annotators label each of the 385 misinformation texts > ref:2606.16710

Appendix

The appendix lists the prompts used in the experiments, including single-agent and multi-agent prompts > ref:2606.16710 The MINT Generation section details Stage 1 (False-fact generation prompts) and Stage 2 (Strategy-conditioned misinformation prompts) > ref:2606.16710 The Multi-Agent Setups section shows the pseudocode for Voting and Consensus decision-making algorithms > ref:2606.16710 The Human Annotation section describes the quality assessment of machine-generated misinformation texts across two dimensions: faithfulness to the false fact and faithfulness to the claim type > ref:2606.16710

References

The references include works on LLMs in MAS, misinformation in LLMs, and related adversarial settings > ref:2606.16710 The references also include studies on peer pressure experiments and self-refinement mechanisms > ref:2606.16710 The paper includes citations for frameworks like MALLM and specific model evaluations like Llama-3.3 and GLM-4.7 > ref:2606.

Improvements for AI systems

textbfContextual Misinformation Resilience in MAS Reasoning: Implementing Group Composition and Decision Protocols for Robustness Improvements (RQ3). Improving AI systems by explicitly testing how group composition and decision-making protocols influence robustness to misinformation allows developers to design MAS that are resilient based on the findings, such as understanding that Consensus can be more stable than voting under peer pressure.
textbfModel-Specific Robustness Tuning: Tailoring Decision Protocols Based on Underlying Model Sensitivity. By analyzing how different models respond, systems can implement decision protocols tailored to model characteristics; for instance, if a model shows sensitivity to misinformed peers, consensus-based decision-making can be prioritized over voting for better stability.
textbfExplicit Misinformation Propagation Mitigation: Developing Self-Correcting Debate Loops. Systems can be improved by designing debate protocols that leverage the self-correction mechanism found in MAD, where error correction in MAD is not gradual, but depends on whether correct information is represented by a majority, enabling rapid stabilization against misinformation spread.
textbfIncreased Task Sensitivity Awareness: Prioritizing Robustness for Knowledge and Alignment Tasks. Since WinoGrande, which measures commonsense reasoning, is less affected than CWQ and Ethics, AI systems should be improved by focusing heightened misinformation defense mechanisms on tasks identified as more sensitive to misleading context, such as ethical alignment and complex QA.
textbfAdaptive Misinformation Filtering: Differentiating Misinformation Relevance During Contextual Input. Systems can be enhanced by implementing mechanisms that assess misinformation relevance (i.e., task-relevant and irrelevant misinformation) before processing, ensuring that only highly relevant contexts trigger performance degradation or require specific mitigation strategies.

Abstract

Multi-agent systems (MAS) are credited with solving complex problems, yet their agents routinely act on context that may be false. Prior work shows single LLMs are swayed by false context, and MAS can be compromised by adversarial agents persuading peers. We study the benign case: agents follow one protocol, none intends to deceive, but some read a misleading passage. We study it with MINT, 10,278 LLM-written misinformation passages in eight styles, testing three open-weight models on commonsense reasoning, knowledge QA, and moral judgment. A relevant false passage costs a single agent 0.12-0.43 accuracy; irrelevant or content-free passages typically cost at most 0.06. Debate looks robust because uninformed agents outnumber misinformed ones. With one of three misinformed, it is no better than choosing the majority over equally exposed independent samples. Interaction helps only with a misinformed majority, recovering up to 0.24 accuracy over voting (0.29 under consensus). A one-sentence unreliability warning recovers up to 0.18 accuracy under voting (0.28 under consensus), at no significant cost under voting when context is true. Robustness should be judged against exposure-matched baselines, with prompt-level skepticism as cheap first defense.