CLAW: Continuous Latent Action World Models via Adversarial Regularization

arXiv:2606.04130 · cs.RO · Submitted 2026-06-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "CLAW: Continuous Latent Action World Models via Adversarial Regularization".

Dev: Here's the gist of the paper. CLAW introduces a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free videos.

Rosa: By simultaneously training the Latent Action Model and the world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone.

Dev: The Core Framework The method leverages an arXiv:2606.04130v1 reciprocal supervision scheme where the world model guides the Latent Action Model (LAM), which in turn informs the world model, resulting in a fully self-supervised, end-to-end training framework for learning latent action world models.

Rosa: This approach employs an adversarial latent regularization strategy that promotes semantically meaningful latent actions and counters future-information leakage.

Paper summary: Rosa: So, this paper is called "CLAW: Continuous Latent Action World Models via Adversarial Regularization," and what it claims is that they've developed a fully end-to-end self-supervised framework to learn both a world model and continuous latent action representations directly from action-free videos.

Dev: It sounds like the core thesis is using adversarial latent regularization and diffusion models to capture structured, meaningful action representations while modeling how the environment transitions predictively, all without needing any actual labels or annotations.

Taro: So what they're saying is that by training the latent action model and the world model at the same time through this reciprocal supervision scheme, CLAW learns to reason about how inferred actions cause environment changes just from looking at visual observations.

Rosa: That means the latent action world model it builds can actually be used for two things: first, it lets an agent learn policies by imitating behaviors directly from those raw videos, which is imitation learning from observation or ILfO, and second, it supports planning where the agent generates sequences of latent actions to reach a specific goal.

Dev: I see how that connects the dots between learning behaviors and planning in a way that's completely self-supervised because they aren't relying on any action labels during training.

Taro: But what about when things go wrong in the real world? Rosa, if this model is learning these latent actions from videos, what happens when it sees something it hasn't seen before, or when the environment misbehaves?

Rosa: Well, that’s where I think we need to look at the adversarial part of their method. They introduce an adversarial latent regularization strategy specifically to promote action representations that are semantically meaningful and to stop them from encoding information about future visual observations that they can actually predict accurately.

Dev: That sounds like a way they're trying to prevent some kind of information leakage where the model cheats by knowing too much about what's coming next, which is a real problem in these kinds of learning setups.

Taro: So, this adversarial signal is meant to enforce that the latent action representation actually captures the underlying structure of how actions affect observation dynamics, not just some random noise.

Paper summary: Rosa: Exactly, they are jointly training both the latent action model and the world model so they simultaneously capture the structure of actions and how those actions respond in the environment.

Dev: And for that structure, they use a reciprocal supervision scheme where the world model guides the latent action model, which then feeds back to inform the world model itself.

Taro: That sounds like a self-contained loop where prediction helps create better representations, and better representations help make better predictions, all based purely on what's in the video data.

Rosa: And that whole setup is built around diffusion-based video generation to predict future frames from past observations and inferred latent actions.

Dev: They’re using a diffusion process where they add Gaussian noise to target frames, and they use a modified causal attention mask to make sure the noisy target tokens can't look at future action tokens or observations beyond their current timestep.

Taro: That causal attention mask sounds like it's trying to keep the temporal causality tight, making sure the world model only predicts what should be possible based on what we already know.

Rosa: The result is that once trained, this latent action world model can be used for imitation learning from observation or ILfO, and it can also support planning by generating sequences of latent actions to hit specified goals.

Dev: So the practical application there is pretty broad; you can use it to clone behaviors from videos or plan complex movements in a simulated environment based on what you've seen.

Taro: But I gotta ask, where does this stop working? They mentioned that the world model generates future observations only based on the most recent frame and the latent action, without retaining memory of past generations.

Rosa: That’s a limitation they admit; it means it doesn't have long-term memory for past generations when making those future predictions, which limits its ability to handle really long sequences or complex histories.

Dev: And another thing they pointed out is that to map those inferred latent actions to actual controls you can use in the real world, you still need some small set of labeled ground-truth action data.

Taro: So even though it's self-supervised for learning the model, the final step of getting usable control still needs some supervision, which is something I think we'll see in future work to fix.

Paper summary: Rosa: That makes sense; the model learns what actions *should* be based on the video structure, but you need that small set of labels to actually translate those learned latent actions into executable controls for a robot.

Dev: So it’s an end-to-end system for learning the model itself, but it still requires some external supervision to bridge the gap to real-world control.

Taro: So where does this leave us? If we're looking at this CLAW paper, what are the big implications for how we think about building autonomous systems that learn from just watching videos?

Rosa: Well, the implication is that you can build a model that learns the structure of actions and environment dynamics purely from unlabelled video data without needing explicit action labels upfront.

Dev: It means we can move towards learning policies through imitation or planning in a much more self-contained way than traditional methods that rely on having labeled demonstrations for every single behavior.

Taro: It also shows how crucial those latent action representations are; they aren't just arbitrary vectors, they’re supposed to be compact and semantically meaningful because of the adversarial regularization pushing them in that direction.

Rosa: So, in short, CLAW gives us a way to learn a world model and continuous latent actions jointly from video data alone by using this specific adversarial regularization setup.

Dev: It's definitely a lot of machinery with the diffusion process and the POMDP formulation they use, but the goal is to see if it can actually handle diverse environments better than existing methods.

Taro: I’m curious to see how robust those latent action representations are when they hit scenarios outside of what was used during training, because that's where I always start looking for failure modes.

Rosa: That's the next thing we need to watch—how well it generalizes when the world doesn't behave exactly like the data it saw.

Dev: Alright, so this paper lays out a framework that uses reciprocal supervision and adversarial regularization to learn latent action world models from video data alone for imitation and planning applications.

Taro: It's a lot of machinery with the diffusion process and POMDP formulation they use, but the goal is to see if it can actually handle diverse environments better than existing methods.

Conclusion: Rosa: So, we're wrapping up our look at CLAW, "Continuous Latent Action World Models via Adversarial Regularization." Basically, they built a system that learns a world model and action representations all together from just watching videos.

Dev: Right, the core idea is this reciprocal supervision where the world model helps the latent action model and vice versa to create these continuous latent actions.

Rosa: They use something called adversarial regularization to make sure those latent actions are actually meaningful, not just random noise trying to trick the system.

Dev: It's trying to stop information leakage, making sure those learned actions really relate to how things change in the environment based on what you see.

Rosa: The authors show that this framework works well for imitation learning and even for goal-directed planning, which is pretty cool when you think about autonomous systems.

Dev: It's a big step because it lets the AI learn to act and plan using just video input without needing explicit demonstrations for every single task.

Rosa: But there are still some catches, like the model not remembering things from way back in the past when predicting what happens next.

Dev: And they admit you still need some labeled data to actually translate those learned latent actions into real world controls that a robot can use.

Rosa: So, the implication is that we can build these models to learn how to navigate and act by just looking at videos, even if it’s not perfect in every situation.

Dev: It's definitely a lot of machinery with the diffusion process and the POMDP formulation they use, but it shows a path toward learning more robust world models for video-based AI.

Tewodros W. Ayalew, Matthew Jeung, Samuel Wheeler, Xiao Zhang, Andre de la Cruz Arce, Kaylene Stocking, Michael Maire

University of Chicago · Toyota Technological Institute at Chicago · Argonne National Laboratory

cs.RO

Submitted: 2026-06-02

Updated: 2026-11-05

Comments: 9 pages, 27 pages of supplementary material

Code: https://github.com/google-research/robodesk

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: CLAW introduces a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free videos.

Key concepts

Latent Action Model (LAM)
This component maps consecutive visual observations into a continuous latent action representation. It uses a Vision Transformer (ViT) backbone and positional encodings to capture structured spatio-temporal tokens, which are then projected into a compact latent action vector used for planning.
World Model Dynamics Modeling
The world model predicts future frames by denoising target video frames conditioned on the inferred latent actions from observed transitions. It uses a diffusion process governed by a stochastic differential equation and a causal attention mask to maintain temporal consistency during prediction.
Adversarial Latent Regularization
This mechanism uses a gradient reversal layer between the latent action and the world model's input to prevent the latent action representation from encoding information about future observations. This forces the learned actions to be meaningful and prevents future-information leakage in the representation.
Reciprocal Supervision Scheme
CLAW trains two models, where one guides the other: the world model informs how actions change environments, and this feedback guides the latent action model. This joint training allows both components to learn robust representations simultaneously from unlabeled video data.

Terminology

Summary

CLAW introduces a fully end-to-end self-supervised framework for learning a world model jointly with continuous latent action representations directly from action-free videos. p. 0 By simultaneously training the Latent Action Model and the world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone. p. 0

The Core Framework

The method leverages an arXiv:2606.04130v1 reciprocal supervision scheme where the world model guides the Latent Action Model (LAM), which in turn informs the world model, resulting in a fully self-supervised, end-to-end training framework for learning latent action world models. p. 1 This approach employs an adversarial latent regularization strategy that promotes semantically meaningful latent actions and counters future-information leakage. p. 1 The resulting model supports both imitation learning from observation (ILfO) and goal-directed planning. p. 0

Latent Action Representation Learning

The Latent Action Model (LAM) uses a ViT backbone to map consecutive observations into a continuous latent action representation, denoted as z i = LAM(o i, o i+1). These patches are projected into the model’s hidden dimension and augmented with both spatial and temporal positional encodings to capture structured spatio temporal token representation. p. 3 The final representation of this token is projected into a compact latent action vector, which is subsequently l2 normalized to produce the latent action embedding. p. 2

World Model Dynamics Modeling

The world model is represented as a diffusion-based video predictor built on a U-ViT backbone that learns to denoise target frames conditioned on latent actions extracted from observed transitions. The forward diffusion process follows a variance-preserving stochastic differential equation that progressively adds Gaussian noise to the target frames. p. 2 A modified causal attention mask is used to enforce temporal causality, ensuring that noisy target tokens cannot attend to future action tokens or observations beyond their timestep. p. 3

Adversarial Regularization Mechanism

Adversarial latent regularization is introduced via a gradient reversal layer between the latent action Zi and the first layer of the world model, applied only for inputs without observation conditioning. This mechanism scales and inverts gradients to discourage the latent action representation from encoding information about future visual observation due to accurate predictions. p. 3 The strength of this adversarial signal is controlled by a gradually increasing weight δ following a sigmoidal schedule. p. 3

Performance and Evaluation

CLAW achieves stronger performance than existing approaches across diverse tasks and embodiments. In imitation learning, the CLAW-based latent action policy consistently outperforms all baselines across both Crafter tasks, achieving higher success rates and collecting significantly more wood on average. p. 5 Furthermore, qualitative analysis shows that CLAW’s latent action representations are robust and transferable, enabling more reliable planning and control in downstream visual tasks. p. 0 The model supports goal-directed planning by generating sequences of latent actions and mapping them to executable actions to reach specified goals. p. 1

Limitations

A key limitation of the latent action world model is that it generates future observations solely based on the most recent frame and latent action, without retaining memory of past generations. Another unavoidable limitation is the need for a small set of labeled ground-truth action data to map inferred latent actions to controls that can be used in the real world. p. 7 The model also generates future observations solely based on the most recent frame and latent action, without retaining memory of past generations. p. 7

Conclusion

In this work, CLAW introduces an end-to-end self-supervised method for jointly learning a world model alongside continuous latent action representations, entirely from unlabeled video. We address the future challenge of information leakage with an adversarial regularization strategy, which allows our model to encode meaningful action representations. p. 7 The learned world model performs competitively with baselines in visual imitation learning, visual planning, and controllability benchmark tasks. p. 7

References

[1] D. P. Bertsekas. Dynamic Programming and Optimal Control: Volume I. p. 8 Athena Scientific, 2012

[2] Y. Tassa, T. Erez, and E. p. 8 Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization

[3] R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming

[4] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting

[5] C. G. Atkeson and J. p. 8 C. Santamar´ıa. p. 8 A comparison of direct and model-based reinforcement learning

[6] M. P. Deisenroth and C. p. 8 E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search

[7] G. A. Miller, G. Eugene, and K. p. 8 H. Pribram. Plans and the structure of behaviour

[8] R. C. Conant and W. p. 8 Ross Ashby. p. 8 Every good regulator of a system must be a model of that system

[9] J. Richalet, A. p. 8 Rault, J.-L. Testud, and J. p. 8 Papon. Model predictive heuristic control

[10] T. Lozano-Perez. p. 8 Robot programming

[11] A. E. Bryson. Applied optimal control: Optimization, estimation and control

[12] A Clark. Whatever next? p. 8 Predictive brains, situated agents, and the future of cognitive science

[13] A. Bubic, D. Y. Von Cramon, and R. p. 8 I. Schubotz. Prediction, cognition and the brain

[14] D. Ha and J. Schmidhuber. p. 8 World models

[15] D. Hafner, T. Lillicrap, I. p. 8 Fischer, R. p. 8 Villegas, D. p. 8 Ha, H. Lee, and J. p. 8 Davidson Learning latent dynamics for planning from pixels

[16] D. Hafner, T. P. Lillicrap, J. p. 8 Ba, and M. Norouzi Dream to control: Learning behaviors by latent imagination

[17] J. Schrittwieser, I. p. 8 Antonoglou, T. p. 8 Hubert, K. Simonyan, L. p. 8 Sifre, S. Schmitt, A. p. 8 Guez, E. Lockhart, D. p. 8 Hassabis, T Graepel et al Mastering Atari Go chess and Shogi by planning with a learned model

[18] D. Hafner, J. Pasukonis, J. p. 8 Ba, and T Lillicrap Mastering diverse domains through world models

[19] A Hu, L. Russell, H Yeo et al GAIA-1: A generative world model for autonomous driving

[20] A Bar, G. Zhou, D Tran et al Navigation world models

[21] S Gao, S Zhou, Y Du et al AdaWorld: Learning adaptable world models with latent actions

[22] Q Garrido, T Nagarajan, B Terver et al Learning latent action world models in the wild

[23] S Ye, J Jang, B Jeon et al Latent action pretraining from videos

[24] J Bruce, M D Dennis et al Genie: Generative interactive environments

[25] D Schmidt and M Jiang Learning to act without actions

[26] X Chen, J Guo, T He et al IGOR: Imagegoal representations are the atomic control units for foundation models in embodied ai

[27] C Zhang, T Pearce, P Zhang et al What do latent action models actually learn? In Advances in Neural Information Processing Systems NeurIPS pages 146676–146697 2026

[28] J M Lee, T Cho, L Zhao et al Why latent actions fail, and how to prevent it arXiv preprint arXiv:2605.20223 14 15

[30] A Nikulin, I Zisman et al Latent action learning requires supervision in the presence of distractors arXiv preprint arXiv:2502.00379 18

[31] F Torabi, G Warnell, and P Stone Recent advances in imitation learning from observation arXiv preprint arXiv:1905.13566 6

[32] R Mendonca, S Bahl, and D Pathak Structured world models from human videos In Proceedings of Robotics Science and Systems RSS 2023 18

[33] F Ebert, C Finn, S Dasari et al Visual foresight: Model-based deep reinforcement learning for vision-based robotic control arXiv preprint arXiv:1812.00568 18

[34] B Hou, G Li, J Jia et al World model for robot learning: A comprehensive survey arXiv preprint arXiv:2605.00080 19

[35] G Williams, N Wagener, B Goldfain et al Information theoretic MPC for model-based reinforcement learning ICRA pages 1714–1721 26

[36] A Nagabandi, G Kahn, R S Fearing and S Levine Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning arXiv preprint arXiv:1708.02596 26

[37] D Hafner, T Lillicrap, M Norouzi and J Ba Mastering Atari with discrete world models arXiv preprint arXiv:2010.02193 18

[38] M Janner, J Fu, M Zhang et al When to trust your model: Model-based policy optimization arXiv preprint arXiv:1906.08253 19

[39] R Sekar, O Rybkin, K Daniilidis et al Planning to explore via self-supervised world models arXiv preprint arXiv:2005.05960 18

[40] V Micheli, E Alonso, and F Fleuret Transformers are sample-efficient world models arXiv preprint arXiv:2209.00588 19

[41] E Alonso, A Jelley, V Micheli et al Diffusion for world modeling: Visual details matter in Atari arXiv preprint arXiv:2405.12399 18

[42] Y Bai, D Tran, A Bar et al Whole-body conditioned egocentric video prediction arXiv preprint arXiv:2506.21552 18

[43] M Assran, A Bardes, D Fan et al V-JEPA 2: Self-supervised video models enable understanding prediction and planning arXiv preprint arXiv:2506.09985 18

[44] J Wu, S Yin, N Feng et al iVideoGPT: Interactive videogpts are scalable world models arXiv preprint arXiv:2405.15223 18

[45] G Zhou, H Pan, Y LeCun and L Pinto DINO-WM: World models on pre-trained visual features enable zero-shot planning arXiv preprint arXiv:2411.04983 18

[46] Y Wang, R Syed, F Wu et al Interactive world simulator for robot policy training and evaluation arXiv preprint arXiv:2603.08546 26

[47] Q Bu, Y Yang, J Cai et al UniVLA: Learning to act anywhere with task-centric latent actions arXiv preprint arXiv:2505.06111 18

[48] Z J Cui, H Pan, A Iyer et al DynaMo: In-domain dynamics pretraining for visuo-motor control arXiv preprint arXiv:2409.12192 26

[49] NVIDIA, N Agarwal et al Cosmos world foundation model platform for physical AI arXiv preprint arXiv:2501.03575 18

[50] H Che, X He, Q Liu et al GameGen-X: Interactive open-world game video generation arXiv preprint arXiv:2411.00769 26

[51] H He, Y Zhang, L Lin Z Xu and L Pan Pre-trained video generative models as world simulators arXiv preprint arXiv:2502.07825 18

[52] Y Huang, W Wan, Y Yang et al CoMO: Controllable motion generation through language guided pose code editing In Proceedings of the European Conference on Computer Vision ECCV pages 180–196 24

[53] J Yang, Y Shi, H Zhu M Liu et al CoMo: Learning continuous latent motion from internet videos for scalable robot learning arXiv preprint arXiv:2505.17006 25

[54] P Intelligence, K Black et al π0.5: a vision-language-action model with open-world generalization 2025 URL https://arxiv. p. 0org/abs/2504.16054 18

[56] C Lynch, M Khansari, T Xiao V Kumar J Tompson S Levine and P Sermanet Learning latent plans from play In Proceedings of the Conference on Robot Learning CoRL pages 1113–1132 2020

[57] A Edwards, H Sahni, Y Schroecker and C Isbell Imitating latent policies from observation In Proceedings of the International Conference on Machine Learning ICML pages 1755–1763 2019

[58] A Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale arXiv preprint arXiv:2010. p. 111929 20

[59] F Bao, S Nie, K Xue Y Cao et al All are worth words: A vit backbone for diffusion models In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR pages 22669–22679 18

[60] Y Ganin, E Ustinova H Ajakan P Germain et al Domain-adversarial training of neural networks Journal of Machine Learning Research 17(59):1–35 18

[61] S Tian, C Finn, and J Wu A control-centric benchmark for video prediction In Proceedings of the International Conference on Learning Representations ICLR 2023 20

[62] Y Zhu, J Wong, A Mandlekar R Mart´ın-Mart´ın et al robosuite: A modular simulation framework and benchmark for robot learning arXiv preprint arXiv:2009.12293 18

[63] H Kannan, D Hafner, C Finn and D Erhan RoboDesk: A multi-task reinforcement learning benchmark https://github.com/google-research/robodesk 21

[64] S Park, K Frans B Eysenbach and S Levine Ogbench: Benchmarking offline goalconditioned rl In International Conference on Learning Representations volume 2025 pages 94937–94982 18

[65] D Hafner Benchmarking the spectrum of agent capabilities arXiv preprint arXiv:2109.06780 18

[66] K Cobbe, C Hesse J Hilton and J Schulman Leveraging procedural generation to benchmark reinforcement learning arXiv preprint arXiv:1912.01588 26

[67] P-T De Boer, D P Kroese S Mannor and R Y Rubinstein A tutorial on the cross-entropy method Annals of Operations Research 134(1):19–67 2005

[68] R Goyal, S Ebrahimi Kahou V Michalski et al The” something something” video database for learning and evaluating visual common sense In Proceedings of the IEEE international conference on computer vision pages 5842–5850 21

[69] T Z Zhao, V Kumar S Levine and C Finn Learning fine-grained bimanual manipulation with low-cost hardware arXiv preprint arXiv:2304.13705 18

[70] H Kress-Gazit, K Hashimoto N Kuppuswamy P Shah et al Robot learning as an empirical science: Best practices for policy evaluation arXiv preprint arXiv:2409.09491 26

[71] K He, X Chen, S Xie Y Li P Dollar and R Girshick Masked autoencoders are scalable vision learners In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR pages 16000–16009 22

[72] K He, X Zhang, S Ren and J Sun Deep residual learning for image recognition In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR pages 770–778 22

[73] S Moon, J Yeom B Park and H O Song Discovering hierarchical achievements in reinforcement learning via contrastive learning In Advances in Neural Information Processing Systems NeurIPS pages 63674–63686 23

[74] S Baker, D Scharstein J P Lewis S Roth M J Black and R Szeliski A database and evaluation methodology for optical flow International journal of computer vision 92(1):1–31 2011

Implementation Details

The Latent Action Model uses a ViT backbone to map consecutive observations to a continuous action code, which is then projected into a compact latent action vector. The policy network is implemented as a convolutional neural network based on a ResNet-18 backbone with the final fully connected layer replaced with a linear projection to the latent action dimension. p. 9 Visual planning is performed using Model Predictive Control combined with the Cross Entropy Method (MPC-CEM) [67]. p. 4 CLAW rollouts are generated using 50 diffusion denoising steps. p. 3

Experimental Results

CLAW achieves the highest success rate on four of the six tasks in the VP2 benchmark.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Continuous Latent Action Representation Learning: The system can learn continuous latent action representations alongside a world model directly from raw, unlabeled video without relying on explicit action labels or annotations. This enables imitation learning from observation (ILfO) and goal-directed planning.

  2. Adversarial Latent Regularization for Semantic Meaning: The method uses an adversarial latent regularization strategy that promotes semantically meaningful latent actions and counters future-information leakage. This ensures the latent actions encode action structure and the ways in which the environment responds, preventing them from simply encoding future observations.

  3. Joint World Model and Action Representation Training: By jointly training the Latent Action Model (LAM) and world model, CLAW learns to reason about how inferred actions induce environment transitions from visual observations alone. This allows for a single end-to-end training framework to capture both action structure and environment dynamics simultaneously.

  4. Goal-Directed Planning via Latent Actions: The learned latent action world model supports planning by generating sequences of latent actions over the learned dynamics to reach specified goals. This sequence generation can then be mapped to executable actions, enabling agents to reach desired goals.

  5. Imitation Learning from Observation (ILfO): The model enables the agent to learn policies by imitating behaviors directly from action-free videos as a form of imitation learning from observation (ILfO) setting. This involves extracting latent actions that serve as targets for training a behavior cloning policy.

  6. Robust Action Transfer: The learned latent action representations support effective action transfer, demonstrating that the model can transfer across various environments and induce the intended dynamics while preserving consistency with the target scene. This is evidenced by retrieved neighbors sharing a coherent dominant flow orientation even when object identity differs.

Sources

Related papers