Cross-Embodiment Robot Foundation World Models with Latent Actions

arXiv:2610.10846 · cs.RO · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Cross-Embodiment Robot Foundation World Models with Latent Actions".

Dev: The gist:

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So what exactly does this paper suggest is the main contribution here? Dev Well, it's about moving away from models that rely on explicit action labels and towards a unified latent action space. Rosa Right, so instead of teaching the model what every single robot does separately, they are creating one shared language for actions across different robots.

Taro: It’s like they’re trying to find that common ground in how robots move, even if their physical setups look completely different. Dev That unified space is key because it lets the world model adapt better when you try to use a robot you haven't seen before during finetuning.

Rosa: It says this unified action space improves performance when the model has to deal with previously unseen robot embodiments. Taro That’s what they’re showing, that the unified space helps them generalize across different physical setups without losing information about the task itself.

Dev: They compare this against an Explicit Action-Conditioned World Model, or EAC-WM, which relies on explicit motion labels for conditioning. Rosa And the result there is pretty telling: explicit action conditioning leads to disjoint action representations across embodiments.

Taro: So if you train a model that only knows how Robot A moves, it struggles when you show it Robot B because their actions don't map onto the same space. Dev That’s the core issue they are pointing out with EAC-WM.

The paper's summary: Rosa: So they summarize this by saying that LAC-WM improves downstream performance significantly over EAC-WM, showing gains of up to forty-six point seven percent on dexterous manipulation tasks and eleven point seven percent on a benchmark called LIBERO <ref:2610.10846#pg1,up to 46.7% on dexterous manipulation>. Dev Those are pretty big numbers for real-world robot skills, especially the manipulation part.

Taro: The summary emphasizes that this improvement comes directly from the unified latent action space, which allows performance to scale positively with the number of embodiments used during pretraining. Rosa So, if you train it on ten different robots, it gets better at handling a new robot than if you only trained it on one specific robot.

Dev: The paper explains that LAC-WM has four parts: an inverse dynamics model, which encodes observations into those latent actions, a forward dynamics model to predict the next frame based on them, a motion decoder to translate those actions back into labels, and an action projector. Rosa It’s a whole pipeline working together in the pretraining phase.

Taro: And they also point out they introduced an auxiliary motion decoding loss which helps align those latent actions with explicit control actions, which is designed to make sure the latent space captures actual actionable content. Dev That’s their way of encouraging the model to learn something physically meaningful in that shared space.

The paper's improvements: Rosa: What are the specific technical improvements they highlight? Taro They show how this unified latent action space allows for semantic consistency in video generation, which is really cool because it means the same latent action prompt can move both a robot’s end-effector and a human’s hand in the same direction. Dev It suggests that their learned latent actions are capturing movement concepts that are shared across different physical forms.

Rosa: They also show this space is much better at capturing motion than the disjoint spaces used in EAC-WM, which makes video generation look more coherent when you switch embodiments. Taro And for planning, they say world-model–based action selection significantly outperforms VLA-only baselines, achieving a twenty-nine point zero percent higher success rate on the S.R metrics compared to the strongest VLA baseline they tested.

Dev: They also look at how it scales when you increase the number of pretraining embodiments used; LAC-WM’s downstream planning performance improves as that number increases, whereas EAC-WM shows the opposite trend with decreased success rates. Rosa That scaling behavior is a really important part of their argument for why this unified approach is better for foundation models.

Conclusion: Rosa: To wrap up, the paper makes a strong case that conditioning on this unified latent action space enhances cross-embodiment learning and adaptation to new robots compared to using a disjoint latent space. Dev Basically, the main point is that sharing one action language across all embodiments helps build more robust robot foundation models for planning.

Taro: They also mention that they found that both the motion decoding loss and crossaugmentation inputs are important when looking at downstream robot planning success rates. Rosa So what this means for us who just listen to the show is that if you’re building a system to control robots, thinking about a shared action space instead of separate ones might be the way forward.

Dev: The paper "Cross-Embodiment Robot Foundation World Models with Latent Actions" shows that by focusing on learning a unified latent action space, we can build world models that are much more adaptable and perform better when we move from lab training to real-world deployment across different hardware.

Rosa: Yeah, so they’ve laid out a clear path for how to make these large robot models actually useful in varied environments. We'll see what the next set of papers brings.

Stanford University

cs.RO

Submitted: 2026-10-07

Updated: 2026-10-07

Project page: https://lacwm.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist: The Latent Action Conditioned Robot World Model (LAC-WM) improves downstream performance over an Explicit Action-Conditioned World Model (EAC-WM) by up to 46.7% on dexterous manipulation

Key concepts

Latent Action-Conditioned Robot World Model (LAC-WM): This model operates using a single, shared latent action space that all robot bodies can use. It abstracts away physical differences so the model learns general control skills applicable to any robot, making it highly adaptable when faced with new robot designs.
Explicit Action-Conditioned World Model (EAC-WM): This method conditions the world model directly on explicit motion labels—specific commands for movement. The paper shows this creates separate action representations for each robot body, which hinders the model's ability to generalize when switching to a new robot.
Unified Latent Action Space: This is a shared mathematical space where actions from different robots are mapped. By forcing all actions into one space, the model can effectively integrate data from many different embodiments during pretraining, leading to better overall performance and scalability.
Scaling Law with Number of Embodiments: This analysis shows how well the world model performs as more robot bodies are used for training. LAC-WM's performance improves as more robots are included, while EAC-WM's success rate decreases, proving the unified space is crucial for efficient cross-embodiment learning.

Terminology

World Model — 89× in this paper · explained in 7 episodes
Action Space — 66× in this paper · explained in 1 episode
Embeddings — 36× in this paper · explained in 2 episodes
Decoder — 23× in this paper · explained in 2 episodes
Latent Space — 12× in this paper · explained in 7 episodes
Fine-Tuning — 11× in this paper · explained in 5 episodes
Foundation Model — 11× in this paper · explained in 3 episodes
Action Encoder — 10× in this paper · explained in 1 episode
Benchmark — 9× in this paper · explained in 2 episodes
Generalization — 9× in this paper · explained in 13 episodes
Adaptation — 7× in this paper · explained in 1 episode
Latent Action Models — 7× in this paper
Foundation Models — 6× in this paper · explained in 7 episodes
Transformer — 6× in this paper · explained in 2 episodes
Action-Conditioned World Model — 5× in this paper · explained in 1 episode
Decode — 5× in this paper · explained in 2 episodes
Coherence — 3× in this paper · explained in 2 episodes
Explicit Action Encoders — 3× in this paper
Ablation — 2× in this paper · explained in 1 episode
Ablation Study — 2× in this paper · explained in 3 episodes
Cross-Embodiment Transfer — 2× in this paper · explained in 2 episodes
Embodied AI — 2× in this paper · explained in 2 episodes
Hyperparameters — 2× in this paper · explained in 2 episodes
Latent Representation — 2× in this paper · explained in 2 episodes
Action Decoder — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 4 episodes
Clustering — 1× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Data Augmentation — 1× in this paper · explained in 7 episodes
Diffusion Model — 1× in this paper · explained in 10 episodes
Diffusion Models — 1× in this paper · explained in 9 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Grounding — 1× in this paper · explained in 3 episodes
K-means Clustering — 1× in this paper · explained in 3 episodes
Latent Representations — 1× in this paper · explained in 3 episodes
Optical Flow — 1× in this paper · explained in 2 episodes
Papers — 1× in this paper · explained in 3 episodes
Quantization — 1× in this paper · explained in 14 episodes
Robustness — 1× in this paper · explained in 5 episodes
Trace — 1× in this paper · explained in 2 episodes
Transferability — 1× in this paper · explained in 5 episodes
VLA Models — 1× in this paper · explained in 3 episodes
Vision-Language-Action (VLA) Models — 1× in this paper · explained in 4 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist: The Latent Action Conditioned Robot World Model (LAC-WM) improves downstream performance over an Explicit Action-Conditioned World Model (EAC-WM) by up to 46.7% on dexterous manipulation and 11.7% on LIBERO, due to its unified latent action space which allows its performance to scale positively with the number of embodiments used during pretraining, contrasting with the disjoint action space in EAC-WM which leads to decreased performance as the number of pretraining embodiments increases.

How it works

LAC-WM operates within a learned unified latent action space shared across diverse embodiments, which improves the world model’s performance when adapted to previously unseen robot embodiments. This unified action space abstracts away embodiment-specific details while preserving task-relevant information, enabling the model to integrate heterogeneous data during pretraining. The LAC-WM consists of four components: an inverse dynamics model (IDM), a forward dynamics model (FDM), a motion decoder, and an action projector.

The pretraining phase involves the IDM encoding image observations into latent actions, which condition the FDM to predict the next frame. The motion decoder translates latent actions back into explicit motion labels, and an action projector maps explicit raw actions of unseen embodiments into the pre-trained unified latent action space during finetuning. During pretraining, the overall training loss combines the self-supervised image latent prediction loss and supervised motion decoding loss using weighting coefficients λrecon and λmotion.

Model Comparison and Analysis

The paper compares LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels, showing that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. The results show that LAC-WM produces higher-quality future predictions than EAC-WM and improves task success rate by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM’s downstream performance to scale positively with the number of embodiments used during pretraining, whereas EAC-WM shows the opposite trend.

Latent Action Space Analysis

UMAP visualization of 7,000 action embeddings from three datasets shows that the latent action embeddings produced by the IDM cluster closely across all three datasets, suggesting they occupy a unified action space. In contrast, the embeddings from the EAE are clearly separated by dataset, indicating that explicit actions do not share a common latent representation. Action transfer results demonstrate that the learned latent action space enables video generation that captures semantically consistent and similar movements across different embodiments. Specifically, the same latent action prompts both the robot’s right end-effector and the human’s right hand to move rightward and upward.

Robot Task Evaluation

In action-conditioned imagined roll-out performance, LAC-WM performs the best compared to models trained from scratch (EAC-WM-S) or pretrained with explicit actions (EAC-WM). In robot planning with action selection, world-model–based action selection significantly outperforms the VLA-only baselines, particularly on the S.R.L and S.R metrics. LAC-WM achieves a 29.0% higher S.R than the strongest VLA baseline, a 22.2% improvement over EAC-WM-S, and a 46.7% improvement over EAC-WM.

Scaling Law with Number of Embodiments and Data Samples

The performance of world models as we scale the number of embodiments used in pre-training shows that LAC-WM’s downstream planning performance improves as the number of pre-training embodiments increases, while EAC-WM shows the opposite trend with decreased S.R. and increased or nearly unchanged δf and δfg. This suggests that a unified action space is critical for crossembodiment pre-training. Furthermore, LAC-WM exhibits consistent performance improvements as either embodiments or samples increase in both scenarios.

Ablating Motion Decoding and Cross-Augmentation

Ablation study of motion decoding loss and crossaugmentation inputs on downstream robot planning shows that LAC-WM achieves a 38% higher success rate, suggesting that both motion decoding and crossaugmentation are crucial for learning latent actions that encode physically meaningful control information.

Conclusion

The superior performance of LAC-WM over EAC-WM indicates that conditioning a world model on a unified action space enhances cross-embodiment learning and improves adaptation to unseen embodiments for robot action planning compared to using a disjoint latent space. The pretraining datasets include motion labels, which is an intentional design choice. The auxiliary motion decoding loss promotes coherence in the latent action representations without constraining the framework to labeled data. The proposed approach is compatible with, and likely to benefit from, large-scale label-free video pretraining. Future work will explore leveraging such unlabeled data to further enhance generalization and transfer capabilities in robotic world models.

Impact Statement

This paper presents work whose goal is to advance the field of robot world model learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

References

Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical ai.

Assran, M., Bardes, A., Fan, D., Garrido, Q., Howes, R., Muckley, M., Rizvi, A., Roberts, C., Sinha, K., Zholus, A., et al. V-jepa 2: Self-supervised video models enable understanding.

Bar, A., Zhou, G., Tran, D., Darrell, T., and LeCun, Y. Navigation world models.

Bu, Q., Cai, J., Chen, L., Cui, X., Ding, Y., Feng, S., Gao, S., He, X., Huang, X., Jiang, S., et al. Agibot world colosseo: A large-scale manipulation platform.

Bu, Q., Yang, Y., Cai, J., Gao, S., Ren, G., Yao, M., Luo, P., and Li, H. Univla: Learning to act anywhere with taskcentric latent actions.

Chen, X., Guo, J., He, T., Zhang, C., Zhang, P., Yang, D. C., Zhao, L., and Bian, J. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai.

Chen, Y., Ge, Y., Tang, W., Li, Y., Ge, Y., Ding, M., Shan, Y., and Liu, X. Moto: Latent motion token as the bridging language for learning robot manipulation from videos.

Cui, Z. J., Pan, H., Iyer, A., Haldar, S., and Pinto, L. Dynamo: In-domain dynamics pretraining for visuo-motor control<ref:2610.

Improvements for AI systems

Bold header: Unified Latent Action Space for Cross-Embodiment Learning

The LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO because the unified latent action space allows LAC-WM’s downstream performance to scale positively with the number of embodiments used during pretraining. This enables a foundational world model that learn[s] more efficiently across embodiments during pretraining and adapt[s] quickly to unseen embodiments through finetuning.

Bold header: Enhanced Action Selection for Dexterous Manipulation

The LAC-WM significantly outperforms VLA-only baselines in action selection, achieving a 29.0% higher S.R. than the strongest VLA baseline and showing lower embedding distances (δf and δfg) compared to EAC-WM, suggesting more accurate subgoal reaching. This allows the robot to select more effective actions for difficult tasks like picking up objects from a counter by leveraging world model predictions guided by the unified latent action space.

Bold header: Robust Adaptation to Unseen Embodiments

The model can adapt quickly to new robots because it utilizes an action projector that maps explicit raw actions into the latent action space, allowing for finetuning in three stages: first fine-tuning IDM and FDM, then training the action projector from scratch, and finally joint fine-tuning. This process ensures that the pretrained FDM can generate rollouts conditioned on embodiment-specific actions when encountering a previously unseen robot.

Abstract

The diversity of robot embodiments and action spaces makes it challenging to build robot world models that generalize across different embodiments. We introduce the Latent Action-Conditioned Robot World Model (LAC-WM), which operates within a learned unified latent action space shared across diverse embodiments. This unified action space improves the world model's performance when adapted to previously unseen robot embodiments. We compare LAC-WM with an Explicit Action-Conditioned World Model (EAC-WM), which conditions on explicit motion labels. Our results show that explicit action conditioning leads to disjoint action representations across embodiments, limiting downstream performance when adapting to new robots. We evaluate both models on dexterous manipulation tasks and a modified LIBERO benchmark. LAC-WM improves downstream performance over EAC-WM by up to 46.7% on dexterous manipulation and 11.7% on LIBERO. Crucially, the unified latent action space allows LAC-WM's downstream performance to scale positively with the number of embodiments used during pretraining. In contrast, the disjoint action space in EAC-WM leads to decreased performance as the number of pretraining embodiments increases. These results highlight the importance of a unified action space for efficient cross-embodiment learning, addressing a key challenge in robotics. Project website: https://lacwm.github.io/