Modality-Autoregressive World-Action Models

The gist The ModAR model introduces a world-action model that autoregressively denoises multiple future-observation modalities before predicting actions, and it outperforms prior WAM formulations.

Modality-Autoregressive World Modeling

ModAR is a World-Action Model (WAM) that predicts future observations as a sequence of multiple modalities, where it autoregressively denoises one modality at a time before finally denoising robot actions ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions This allows each prediction to condition on previously generated modalities. The model trains from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance.

Architecture and Tokenization

The ModAR architecture embeds the current observation from each modality and passes the resulting tokens into a shared diffusion transformer The DiT combines cross-modal blocks with modality-specific experts and output heads. The robot configuration, task embedding, and flow timesteps for each modality condition the transformer through adaLN. Modality tokenization involves patching RGB and depth maps, extracting DINO tokens from the spatial patch tokens of a frozen DINOv2 encoder, and representing point tracks by tracking 2D grid queries with CoTracker3. These modalities are linearly projected into the common token dimension, and a learned modality embedding is added.

Block-Causal Modality Generation

The joint multimodal distribution is modeled as a sequence of modality-autoregressive predictions, formally written as pθ(Yt, At ct) = pθ(At ct, Y<k t) where Y<k t denotes the modalities preceding mk in the generation order. The final action prediction step acts as an inverse dynamics model (IDM), mapping the generated future observations to the actions that induce the predicted transitions. The generation order is set to tracks → DINO → depth → RGB, intuitively ordering targets from compact, structured representations that are easier to predict toward increasingly high-dimensional and detailed representations.

Training and Inference

During training, every output stream is supervised with a JiT-style xprediction objective. For each supervised modality m, the clean target Yb m t is predicted using a linear flow-matching interpolant Ye m t = τmYm t + (1 − τm)ϵ m. The total loss function is L(θ) = X K k=1 Lmk + Lact, where the action term is included only for examples with action labels. During inference, one modality is denoised at a time, and the resulting clean tokens become context for generating the next modality.

Experimental Findings

Controlled experiments show that modality-autoregressive generation performs best across all evaluated data scales and benefits most from additional actionless data. Specifically, predicting point tracks, DINO features, and depth provides additive gains, while additionally predicting future RGB provides no consistent benefit. ModAR achieves the highest average success rate among the formulations at every data scale. Furthermore, adding 1,200 actionless demonstrations improves the average success rate from 66% to 76% for ModAR compared to a mere 1% improvement for Unified. On real-world bimanual manipulation tasks, ModAR achieves the highest success rate among baselines and also improves with human video demonstrations.

Ablation and Comparison

Ablations show that context noise is critical for robust autoregressive generation, as removing it causes the success rate to fall to 63%. Reversing the future-modality order lowers the average success rate to 65%, supporting the hypothesis that more structured modalities act as “scratchpads” for more detailed ones. Removing future RGB prediction does not reduce performance, suggesting that RGB contributes the least of the four modalities to success rate in this setting. ModAR outperforms Unified even when both models' native action predictions are discarded and replaced by a separately trained IDM, suggesting its predicted futures are themselves more useful for predicting actions. The model achieves an end-to-end inference latency of 147.9 ms (6.76 Hz) on a single NVIDIA GeForce RTX 5090 GPU.

Conclusion

ModAR is introduced as a world-action model that autoregressively denoises multiple future-observation modalities before predicting actions. In a controlled simulation study, this formulation achieves the highest average success rate among representative existing WAM formulations and improves the most as we add actionless data. On three real-world tasks, it achieves the highest success rate among baselines and also improves with human video demonstrations. On average, point tracks, DINO features, and depth provide complementary gains, whereas additionally predicting future RGB provides no consistent benefit in our experiments. This work highlights modality-autoregressive prediction as a promising alternative to relying solely on future RGB prediction for world-action modeling.

Limitations

Our experiments cover a limited number of tasks and use discrete task labels rather than language instructions, so they do not establish broad generalization across tasks, objects, or scenes. Sequentially generating multiple modalities also increases inference latency relative to simultaneous or action-only generation. Future work should explore largescale experiments with broader task diversity, generalization evaluation, and training on heterogeneous internet-scale actionless data. Future work can also explore the best order for generating modalities, which may depend on the task or specific scenario.

Acknowledgment

We thank Narek Harutyunyan for assistance in data collection. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant Nos. DGE2140739 and DGE2631988. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. This work used Bridges-2 at Pittsburgh Supercomputing Center through allocation CIS260202p from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants [38], [39].

1. Start with modality-autoregressive generation by ordering targets from compact, structured representations that are easier to predict toward increasingly high-dimensional and detailed representations, specifically using the order tracks → DINO → depth → RGB. This strategy leverages the idea that generating easier-to-predict modalities earlier can expose structure that simplifies subsequent predictions.

2. Implement context noise injection during training by adding noise to context blocks, where you independently sample ϵ ctx j ∼ N (0, I) and τ ctx j ∼ U(1 − β, 1) for every context block mj in every training example to mitigate cascading errors.

3. Utilize a separate inverse-dynamics model (IDM) for action prediction by replacing native action predictors with an IDM that mapping its future-observation predictions to actions, as the paper demonstrates ModAR outperforms Unified, showing that its predicted futures are themselves more useful for predicting actions.

4. Employ modality ablation analysis to determine the necessary predictive set; specifically, removing future RGB prediction is shown to leave performance unchanged, suggesting that predicting future RGB introduces high-variance appearance details while adding little information beyond the more structured targets in our setting.

5. Scale training data systematically by increasing actionless demonstrations from 50 to 1250 per task; this scaling improves the average success rate significantly, showing that ModAR benefits most from additional actionless data.

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex- π on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately 20 times fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.