Explaining and interrupting collective misalignment in language model systems
cs.CL, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: In collectives, language model agents can exchange individually innocuous messages that together support a policy violation.
Abstract
In collectives, language model agents can exchange individually innocuous messages that together support a policy violation. Understanding and preventing collective misalignment requires answering the question of which messages sufficiently influenced and necessarily enabled the violation. Collective Dependence Analysis (CDA) operationalises necessity and sufficiency to explain decisions and find the fewest removals interrupting support within tested combinations. Tests covered constructed violations, unscripted agent conversations, and recorded interactions, with selected message removals reducing violations from 93% to 6% in controlled cases. Using O(N) model evaluations for N messages, CDA achieves the same violation reduction with half the deletions of a frontier LLM judge given the transcript and asked for all supporting messages in eligible operational cases. CDA also proposes the mechanism through which the collective misalignment emerged. Nonetheless, complete explanations remain difficult on unfamiliar tasks and some removals can inadvertently disrupt correct decisions. CDA unifies explanations of collective misalignment with experimentally tested interventions.
Sources
- Multi-Agent Risks from Advanced AI
- SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents
- Evaluating Control Protocols for Untrusted AI Agents
- Detecting Multi-Agent Collusion Through Multi-Agent Interpretability
- The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignment
- Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studies
- Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures
- CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures
- Semantic Cooperative Games for Contribution Attribution in LLM-Based Multi-Agent Systems
- Information Dynamics of Language Communication
- Copying explains the collective behavior of AI agents in the wild
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data