Towards Safe World Models: Risks, Defenses, and Evaluation
cs.CR
Submitted: 2026-10-09
Updated: 2026-10-09
Comments: 55 pages, 6 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: World models support planning, control, learning from imagined experience, and model-mediated evaluation by predicting action-conditioned consequences.
Terminology
Abstract
World models support planning, control, learning from imagined experience, and model-mediated evaluation by predicting action-conditioned consequences. Their safety cannot be inferred from predictive accuracy alone: errors in observations, state and memory, dynamics, evaluation, execution, or feedback can propagate into unsafe decisions and persist through updates. This work organizes the literature on world-model safety around a six-stage lifecycle covering data and observations, state and memory, dynamics and imagination, evaluation and planning, action and control, and feedback and update. We synthesize non-adversarial failures, attacks, defenses, evaluation protocols, benchmarks, and deployment conditions, and distinguish evidence directness from outcome level. The current evidence corpus contains 163 cited works. We use the framework to identify where claims are supported, where attribution remains unresolved, and which measurements are needed to connect model-level behavior to closed-loop and operational safety.
Sources
- Navigation World Models
- Attacking the Trusted Imagination: Oracle-Level Integrity Attacks on Imagine-then-Act World Models
- Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability
- TRAP: Tail-aware Ranking Attack for World-Model Planning
- SAFE: Multitask Failure Detection for Vision-Language-Action Models
- When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models
- Stealthy World Model Manipulation via Data Poisoning
- TrojanWorld: Backdooring World-Model Agents via Imagination Steering
- SafeDreamer: Safe Reinforcement Learning with World Models
- PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning
- When to Trust Your Model: Model-Based Policy Optimization
- Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning
- Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving
- TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents
- A Blind Trust, the Bloody Thrust: When Attacker-Controlled Hook Updates Steer AI Agent Harnesses towards Malicious Behaviors
- BadWAM: When World-Action Models Dream Right but Act Wrong
- Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms
- How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Reference
- ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watchers
- The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPS
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs