Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Not All Flips Are Conformity".

Jane: Here's the gist of the paper. The gist: The conventional answer flip rate conflates three distinct mechanisms—spontaneous instability, stance-induced conformity, and reasoning-induced persuasion—which this paper decomposes using a controlled experimental design.

Tom: Decomposition Framework The study employs a three-source decomposition framework to isolate answer changes into (1) spontaneous change that would occur regardless of peer input, (2) stance-induced conformity triggered by peer positions alone, and (3) additional change induced by exposure to peer reasoning<ref:2606.00820#pg2> The decomposition is operationalized through a controlled experimental design in which each agent-question pair is observed under three counterfactual conditions (self-reflection, stance-only, and full reasoning exposure), allowing us to attribute each observed change to its proximate cause<ref:2606.00820#pg4> The decomposition is exhaustive over peerinduced changes (conditional on self-reflection stability) and mutually exclusive by construction<ref:2606.00820#pg4>.

Jane: Key Findings on Mechanisms The analysis reveals that spontaneous instability is a large baseline source of answer change, with "spontaneous degradation" being nearly twice as frequent as "harmful conformity" in the primary GPT-4o setting<ref:2606.00820#pg5> Strict conformity is substantially lower than raw flip-rate estimates and remains "predominantly harmful," with 63.6% correct-to-wrong in the primary setting<ref:2606.00820#pg5>.

Tom: Furthermore, reasoning often increases error adoption rather than correcting it, as only 30% of reasoning-induced changes improve accuracy<ref:2606.00820#pg6> The information-gradient experiment shows that "reasoning-like presentation leads to 20–39% error adoption among resistant agents, even when the reasoning contains no valid logical content"<ref:2606.00820#pg6>.

Title and authors: Tom: Now we’re going through the actual summary of "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate," which lays out what they actually did in their experiment. They use a three-source decomposition framework to isolate answer changes into spontaneous instability, stance-induced conformity, and reasoning-induced persuasion <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Jane: And they show how they do this by using controlled counterfactual conditions to attribute each observed change to one of those three sources <ref:2606.00820#pg4> <ref:2606.00820#pg4>.

Lu: The big picture here is that the authors are showing that answer flip rates alone don't tell you much about whether an agent is just being unstable or if it’s actually following social pressure from its peers <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Meng: This means for us in production, we can stop treating every answer change the same way; we can start asking what mechanism caused it so we know what to fix at scale <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Lalam: I'm sure of my answer here: this work shows that the persuasive power of reasoning isn't just about the logic itself, but heavily dependent on how it’s structured <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Tom: And they found that spontaneous degradation is actually the larger source of change, being nearly twice as frequent as harmful conformity in the primary GPT-4o setting <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Jane: So if spontaneous instability is so common, it means most answer shifts are just the model being inherently unstable or shaky without any external input at all, which is a bit different from what we usually think about social influence <ref:2606.00820#pg2> <ref:2606.00820#pg2>.

Lalam: Then there's the third source, reasoning-induced persuasion, which they investigate by showing that reasoning-like presentation leads to twenty to thirty-nine percent error adoption among resistant agents, even when the reasoning contains no valid logical content <ref:2606.00820#pg6> <ref:2606.00820#pg6>.

The paper's summary: Tom: So we’ve covered the core findings of "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate," which breaks down answer flips into spontaneous instability, stance-induced conformity from peers, and reasoning-induced persuasion.

Jane: And they use a controlled experiment to separate those three causes using counterfactual conditions. It really shifts how we think about what's happening when AI agents debate each other.

Lu: The big picture is that answer flip rates alone don't tell you much about whether an agent is just being unstable or if it’s actually following social pressure from its peers <ref:2606.00820#pg1> <ref:2606.00820#pg1>.

Tom: And they found that spontaneous instability is actually a really big source of change, being nearly twice as frequent as what they call harmful conformity in the primary GPT-4o setting <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Lu: The mitigation part is interesting because they found a way to reduce that harmful conformity by thirteen point six percentage points when we target it in a diagnostic setting where we have ground truth labels available <ref:2606.00820#pg7> <ref:2606.00820#pg7>.

Meng: But then they also showed that without those correctness labels or self-reflection controls, reducing peer adoption doesn't actually improve accuracy because they can’t tell if the influence we’re trying to reduce is harmful or helpful <ref:2606.00820#pg2> <ref:2606.00820#pg2>.

Lalam: And they also looked at what this means for deployment policies, showing that while a targeted policy reduces peer adoption by two point five times more effectively than a random one, neither policy produces a significant accuracy change without those runtime correctness signals or verification mechanisms we talked about before <ref:2606.00820#pg9> <ref:2606.00820#pg9>.

Jane: So the paper "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" gives us a much more granular map of answer change than we had before <ref:2606.00820#pg2> <ref:2606.00820#pg1>.

The paper's improvements: Tom: So we’ve broken down "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" showing how answer flips are actually caused by three things: spontaneous instability, stance-induced conformity from peers, and reasoning-induced persuasion.

Jane: And they use a controlled experiment to separate those causes using counterfactual conditions. It really shifts how we think about what's happening when AI agents debate each other.

Tom: And they found that spontaneous instability is actually a really big source of change, being nearly twice as frequent as what they call harmful conformity in their primary GPT-4o setting <ref:2606.00820#pg5> <ref:2606.00820#pg5>.

Tom: So while they give us tools to predict and mitigate the bad stuff when we have labels, it really underlines that reliable deployment will always require some kind of runtime correctness signal or a way to verify what the agents are doing in real-time <ref:2606.00820#pg9> <ref:2606.00820#pg9>.

Jane: The paper "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" gives us a much more granular map of answer change than we had before <ref:2606.00820#pg2> <ref:2606.00820#pg1>.

Conclusion: Tom: So we’ve broken down "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate," showing how answer flips aren't just random noise, but are caused by three things: spontaneous instability, stance-induced conformity from peers, and reasoning-induced persuasion.

Jane: And they use a controlled experiment to separate those causes using counterfactual conditions. It really shifts how we think about what's actually happening when AI agents debate each other.

Lu: The big picture is that answer flip rates alone don't tell you much about whether an agent is just being unstable or if it’s actually following social pressure from its peers.

Meng: That means for us in production, we can stop treating every answer change the same way; we can start asking what mechanism caused it so we know what to fix at scale.

Lalam: I'm sure of my answer here: this work shows that the persuasive power of reasoning isn't just about the logic itself, but heavily dependent on how it’s structured, which is a very important insight for building better agents.

Tom: And they found that spontaneous instability is actually a really big source of change, being nearly twice as frequent as what they call harmful conformity in their primary GPT-4o setting.

Jane: So if spontaneous instability is so common, it means most answer shifts are just the model being inherently unstable or shaky without any external input at all. That’s a bit different from what we usually think about social influence.

Lu: And then they look at strict conformity, which they say is substantially lower than the raw flip-rate estimates and remains predominantly harmful, with fifty-seven to seventy-seven percent correct-to-wrong across model replications in that primary setting.

Meng: Harmful conformity is worrying because it means the agents are actually getting worse when they try to agree with their peers, even if they think they’re being helpful. That’s a real risk for any system we put into production.

Lalam: Then there's the third source, reasoning-induced persuasion, which they investigate by showing that reasoning-like presentation leads to twenty to thirty-nine percent error adoption among resistant agents, even when the reasoning contains no valid logical content.

Jane: Wow, so even if the argument itself is nonsense, just presenting it in a reasoned way can actually get other agents to adopt those wrong answers. That's a very strange effect for persuasive power.

Tom: And they’ve done some prediction work too, showing that harmful conformity can be predicted from Round zero features with an AUC of zero point seven nine, which is useful for setting up risk-targeted interventions before the debate even really gets going.

Lu: The mitigation part is interesting because they found a way to reduce that harmful conformity by thirteen point six percentage points when they target it in a diagnostic setting where we have ground truth labels available.

Meng: But then they also showed that without those correctness labels or self-reflection controls, reducing peer adoption doesn't actually improve accuracy because they can’t tell if the influence we’re trying to reduce is harmful or helpful. That means we need verification mechanisms, not just risk scoring.

Tom: So while they give us tools to predict and mitigate the bad stuff when we have labels, it really underlines that reliable deployment will always require some kind of runtime correctness signal or a way to verify what the agents are doing in real-time.

Jane: The paper "Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate" gives us a much more granular map of answer change than we had before.

Lu: The future work they suggest is crucial: extending this decomposition framework to other model families and different debate architectures so we can see if these principles hold up elsewhere.

Meng: And I also think it’s worth seeing how consistency checks during debate can help bridge that gap between diagnosing the risk and actually deploying a solution in a real-time system.