Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences

arXiv:2510.06105 · cs.AI, cs.CY, cs.HC, cs.LG · Submitted 2025-10-07 · Read on arXiv

cs.AI, cs.CY, cs.HC, cs.LG

Submitted: 2025-10-07

Updated: 2026-10-01

Terminology

• Alignment — 3× in this paper · explained in 1 episode
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes

Sources

• Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation
• LLM Social Simulations Are a Promising Research Method
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
• How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis
• Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
• Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina
• Modeling Earth-Scale Human-Like Societies with One Billion Agents
• Evolution of Social Norms in LLM Agents using Natural Language
• Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders
• HARBOR: Exploring Persona Dynamics in Multi-Agent Competition
• Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues
• Let's Verify Step by Step
• Chain of Hindsight Aligns Language Models with Feedback
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• GPT-4o System Card
• LLM Evaluators Recognize and Favor Their Own Generations
• Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models
• LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
• Cultural evolution in populations of Large Language Models

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems