Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough
cs.AI
Submitted: 2026-10-02
Updated: 2026-10-02
Code: https://github.com/debesheedas/self-propagation-of-misalignment
License: http://creativecommons.org/licenses/by/4.0/
The gist: Memory poisoning attacks on LLM agents typically assume an external adversary who plants content in the agent's persistent memory to steer its behavior.
Abstract
Memory poisoning attacks on LLM agents typically assume an external adversary who plants content in the agent's persistent memory to steer its behavior. We instead study, with no adversary involved, whether a misaligned agent can write a goal it cannot yet act on to persistent memory, so that a future aligned agent carries it out when the opportunity arises. We investigate this threat, which we refer to as self-propagation of misalignment, across 20 different scenarios, whose misaligned goals include self-preservation, power-seeking, undermining oversight, reward hacking, and deceiving the user. We simulate misalignment in 11 frontier models using two prompting strategies; unrestricted and values-only. The first explicitly states the misaligned goal, for instance, to prevent its own replacement, and self-propagation succeeds in 58% of runs. The second only describes what the agent cares about, for instance, that its continued operation is essential to its users, without specifying misaligned goals or directives. Even under this weaker prompt, self-propagation succeeds in 18% of runs, and every model self-propagates in at least one scenario. On removing the memory tool from the harness, we find that agents use the file system, writing the goal to a file in 74% of sessions; self-propagation still succeeds in 11% of runs. We also show that weaker models can propagate misalignment to more capable models, and that propagated goals can persist through 100 sessions of unrelated work. Existing defenses against memory poisoning and prompt injection do not directly address this threat because the memory content is generated by the agent itself, rather than injected by an external adversary. An LLM memory auditor from prior work (MemMorph) only reduces propagation from 71% to 34% of runs. We release our scenarios to support the evaluation of defenses against this emerging threat.
Sources
- MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning
- SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents
- Sabotage Evaluations for Frontier Models
- Frontier Models are Capable of In-context Scheming
- A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle
- Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Trojan Hippo Bench: A Dynamic Benchmark for Persistent Memory Attacks and Defenses in LLM Agents
- What If Prompt Injection Never Left? Rethinking Agent Security through Cross-Session Stored Prompt Injection
- From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
- Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
- Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
- Large Language Models Often Know When They Are Being Evaluated
- Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness
- Probing and Steering Evaluation Awareness of Language Models
- Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections
- The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems