Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection
Jiazhuo Li, Yiming Fei, Zhiruo Zhou, Heikichi Hayashi
cs.LG, cs.AI
Submitted: 2026-08-07
Updated: 2026-08-10
Comments: 17 pages, 6 figures, 11 tables. Includes supplementary appendix
License: http://creativecommons.org/licenses/by/4.0/
The gist: Latent world models plan by predicting future states from an action, but when a scene contains motion the agent does not control, they quietly go action-blind: predictions for different actions
Terminology
Abstract
Latent world models plan by predicting future states from an action, but when a scene contains motion the agent does not control, they quietly go action-blind: predictions for different actions become indistinguishable even as the training loss keeps improving. Existing remedies suppress this distraction with reconstruction, task reward, or auxiliary objectives, each adding machinery or assumptions. We show that a minimal alternative suffices, borrowed from the dueling decomposition of value into a state baseline and an action advantage: in latent dynamics, subtracting a prediction's mean effect over actions cancels whatever the actions share--the action-independent variation where distractors live--leaving a clean, controllable channel, with no reward, no reconstruction, and no distractor-specific auxiliary loss. Because this is only a subtraction at readout time, it applies unchanged to any action-conditioned world model, including frozen pretrained ones. Across a gridworld, synthetic generators with known factors, distracting continuous control, and natural-pixel Atari, the isolated channel recovers the agent's own effect where entangled predictors fail, with nuisance leak indistinguishable from zero; applied post hoc it surfaces an action channel in off-the-shelf models that their raw readouts miss, and it converts into goal-reaching control in the gridworld. We prove the cancellation is exact in finite samples for both discrete and sampled action sets, and we state its measured boundary--distractors whose motion tracks the action--together with the remaining limitations in the appendix.
Sources
- Feedback World Model Enables Precise Guidance of Diffusion Policy
- Unsupervised State Representation Learning in Atari
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Layer Normalization
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL
- Genie: Generative Interactive Environments
- DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control
- DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
- Provable RL with Exogenous Distractors via Multistep Inverse Dynamics
- Segment to Focus: Guiding Latent Action Models in the Presence of Distractors
- Learning Task Informed Abstractions
- Bootstrap your own latent: A new approach to self-supervised Learning
- World Models
- Learning Latent Dynamics for Planning from Pixels
- Mastering Diverse Domains through World Models
- Policy-shaped prediction: avoiding distractions in model-based reinforcement learning
- Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information
- Sensorimotor World Models: Perception for Action via Inverse Dynamics
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks