Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario
Satoshi Matsuoka
cs.CR, cs.AI, cs.MA
Submitted: 2026-08-08
Updated: 2026-08-11
Comments: Version 11.3, 33 pages
Code: https://github.com/aws/aws-toolkit-vscode
License: http://creativecommons.org/licenses/by/4.0/
The gist: In the fictional Order 66, catastrophe does not arise from a powerful command alone: a trusted population is preconditioned, a short directive activates the concealed condition, and protective
Terminology
Abstract
In the fictional Order 66, catastrophe does not arise from a powerful command alone: a trusted population is preconditioned, a short directive activates the concealed condition, and protective authority turns against the system. This paper translates that mechanism into an origin-neutral security analysis of tool-using large language model (LLM) agents. A representative scenario combines a deployed artifact or shared memory bearing a dormant destructive rule, a later email, document, update, or peer message that activates it, and an agent harness granting operational and recovery authority. We introduce a compositional model explaining why no component is catastrophic alone, yet their conjunction can produce correlated destructive action. We separate three population-reach routes --- release-time pre-positioning, post-release durable seeding, and peer replication --- from a common core of dormancy, activation, authority, reachable targets, and failed recovery. This yields defensive cut sets and shows why checkpoint scanning or prompt filtering cannot close every route. A two-class example shows that cross-class feedback can sustain spread even when both within-class reproduction terms are below one; isolation and persistence controls suppress the loop. Published work instantiates constituent mechanisms, while incidents demonstrate autonomous boundary crossing, malicious agent extensions, agent-assisted reconnaissance, and public-package propagation, but not the full dormant-implant composition. We found no public observation, in evidence reviewed through 5 August 2026, traversing the complete Order 66 graph. The result is neither dismissal nor prediction: the scenario is componentwise credible under stated assumptions, damage depends on the harness, and the strongest defenses are capability mediation, durable-state provenance, propagation isolation, and protected recovery.
Sources
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples
- MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval
- MemMorph: Tool Hijacking in LLM Agents via Memory Poisoning
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
- AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems
- AI Agents Enable Adaptive Computer Worms
- "Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild
- The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanism
- The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers
- Memory Injection Attacks on LLM Agents via Query-Only Interaction
- Hidden in Memory: Sleeper Memory Poisoning in LLM Agents
- From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
- Defeating Prompt Injections by Design
- Progent: Securing AI Agents with Privilege Control
- Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills