UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

arXiv:2607.05133 · cs.CV · Submitted 2026-07-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation".

Jane: Here's the gist of the paper. World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality <ref:2607.05133#pg2> The paper proposes UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer <ref:2607.05133#pg3> By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization <ref:2607.05133#pg5> Pruning redundant reasoning tokens The core problem addressed is determining which architecture better transfers video-modeling benefits to trajectory generation <ref:2607.05133#pg3> Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch <ref:2607.05133#pg5> The authors propose treating future video prediction as dense dynamics supervision for the same parameters used in trajectory denoising <ref:2607.05133#pg3> UNIVERSE places historical observations, future video latents, and ego-trajectory tokens in one shared DiT <ref:2607.05133#pg5> This design encourages a shared worldaction representation by having video and trajectory losses update the same attention layers and feedforward blocks <ref:2607.05133#pg5> How the training signal is built The training objective combines a flow-matching loss for future video supervision and a trajectory prediction loss <ref:2607.05133#pg5> The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility mask <ref:2607.05133#pg5> The final objective is L = Lact + λvidLvid, where λvid balances the two objectives <ref:2607.05133#pg5> This ensures that future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation <ref:2607.05133#pg5> Where the gains come from The key finding is that architecture matters, as video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone <ref:2607.05133#pg3> The Modality-Decoupling Visibility Mask shares historical context across modalities while blocking mutual attention between future video and trajectory tokens <ref:2607.05133#pg5> This mask enables efficient trajectory-only inference by removing future-video denoising at test time, achieving a 4.3× speedup while maintaining the generalization benefit of video co-training <ref:2607.05133#pg5> Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs.

Tom: 89.6 for the Two-DiT variant) and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning <ref:2607.05133#pg5> Pruning redundant reasoning tokens The authors introduced several variants to isolate the effects of architectural factors, such as video co-training, mask-based causal decoupling, and shared DiT parameterization <ref:2607.05133#pg5> The UNIVERSE w.o.

Jane: Mask variant obtains comparable indomain performance with a PDMS of 90.9 but does not translate into better deployment behavior <ref:2607.05133#pg5> The no-mask variant relies on coupled video-action rollout and cannot remove one future modality without changing the learned dependency structure <ref:2607.05133#pg5> Where the gains come from The shared denoising backbone matters, as UNIVERSE Two-DiT preserves both video and trajectory objectives but separates them into two diffusion transformers <ref:2607.05133#pg5> This results in the NAVSIM PDMS dropping to 89.6 and weaker zero-shot performance compared to the shared-DiT UNIVERSE <ref:2607.05133#pg5> Visibility decoupling enables flexible deployment, allowing trajectory-only inference by removing future video denoising at test time which reduces inference time by 4.3× <ref:2607.05133#pg5> The results demonstrate that shared-parameter video-action co-training with mask-modulated modality decoupling is effective for generalizable autonomous-driving planners <ref:2607.05133#pg5> The model achieves competitive visual generation quality while being optimized for planningoriented video-action co-training <ref:2607.05133#pg5> The model also supports joint, video-only, and trajectory-only prediction by instantiating only the requested future token groups <ref:2607.05133#pg5> The DPVO-based verification shows that the ego motion implied by the generated future videos is consistent with the trajectory predicted by the model <ref:2607.05133#pg5> Pruning redundant reasoning tokens The paper concludes that UNIVERSE achieves state-of-the-art NAVSIM performance and strong zero-shot generalization to nuScenes and Bench2Drive <ref:2607.05133#pg5> The model is shown to preserve video-action consistency under simulation-domain shifts, even in challenging nighttime and foggy driving conditions <ref:2607.05133#pg5> Future work may focus on lighter video tokenizers or smaller DiT backbones to reduce the computational cost of training large video-action models <ref:2607.05133#pg5> The paper uses public datasets, benchmarks, models, and evaluation tools according to their official licenses and terms of use <ref:2607.

Title and authors: Tom: So we're looking at this paper called "UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation," and the authors are Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Francesco Nex and Hao Cheng and Michael Ying Yang.

Jane: That title sounds pretty technical right? It tells us they're trying to unify video modeling with action planning in autonomous driving using some kind of masking trick.

Lu: Exactly. The core idea is addressing the question of which architecture actually transfers those video modeling benefits over to trajectory generation, because right now, most systems separate them.

Meng: Separating them sounds simpler for training, but it means the video model only gives a hint indirectly to the planning part of the system >

Tom: Right. So this paper is proposing UNIVERSE as a unified video-action model built on just one mask modulated Diffusion Transformer instead of having separate parts for imagining videos and predicting trajectories.

Jane: That's what page one sets up, showing that existing cascaded or dual diffusion transformer designs separate video imagination from action prediction, which weakens the transfer of learned world dynamics to the trajectory branch >

Lu: They argue that this separation means the action model might still overfit dataset specific driving priors because it only gets an indirect regularization from the video model >

Meng: So they are proposing a shared generative backbone where both video and trajectory tokens are co-trained, which is what I'm really curious about for deployment >

Tom: Right. They propose treating future video prediction as dense dynamics supervision for the same parameters used in trajectory denoising, which is a key difference from what exists out there >

Jane: And they introduce this Modality Decoupling Visibility Mask to handle the training side of this unification, which is pretty clever because it shares historical context across modalities while blocking mutual attention between future video and trajectory tokens >

The paper's summary: Tom: So what's the actual proposal for UNIVERSE? Page three lays out that they propose a unified video-action model that co-trains future video latents and ego trajectory tokens within one shared Diffusion Transformer >

Jane: Which means they are using this single shared generative parameter set to allow dense video supervision to directly regularize trajectory denoising, which is a big deal because it connects the two domains much more directly >

Lu: The main mechanism here is that they co-train these tokens within one shared Diffusion Transformer, which encourages a shared worldaction representation by having video and trajectory losses update the same attention layers and feedforward blocks >

Meng: From my side, if they are sharing parameters, I worry about how much redundancy they're actually getting in terms of learning new things versus just reinforcing what's already there >

Tom: They build the training objective by combining a flow matching loss for future video supervision and a trajectory prediction loss, and this combined signal is fed into the shared Diffusion Transformer with a visibility mask >

Jane: The objective function they use is L equals Lact plus lambda vidLvid, where lambda vid balances these two objectives, making sure that future-video prediction provides dense co-training supervision for the same parameters used by trajectory generation >

Lu: That shared parameterization is what they claim is most effective when video and trajectory losses update the same generative backbone because it forces them to learn a single representation of the world action >

The paper's improvements: Tom: Now for what they claim as the main gains, UNIVERSE shows that architecture really matters, and video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone >

Jane: They introduce this Modality-Decoupling Visibility Mask to share historical context across modalities while blocking mutual attention between future video and trajectory tokens >

Lu: That mask is crucial because it prevents causal leakage in the unified video-action training, which means it stops future target leakage during training and inference >

Meng: I like that they also show how this design enables efficient trajectory-only inference by removing future video denoising at test time, achieving a four point three times speedup over joint video action rollout > <ref:2607.05133#pg1>

Tom: That speedup is significant because it keeps the generalization benefit of video co-training while making the model much faster for actual planning when you're driving >

Jane: They show state-of-the-art closed loop planning and strong zero shot action generalization on NAVSIM, nuScenes, and Bench2Drive without needing any fine tuning >

Lu: The experiments show that UNIVERSE achieves ninety-one point zero PDMS on NAVSIM compared to eighty-nine point six for the Two DiT variant, proving that sharing the backbone is what matters for transferring video learned world dynamics >

Conclusion: Tom: So to wrap up, the core finding of this paper on UNIVERSE is that shared parameterization and mask modulated modality decoupling are effective for generalizable autonomous driving planners >

Jane: It means they achieve state-of-the-art performance on NAVSIM and strong zero shot generalization to nuScenes and Bench2Drive without fine tuning >

Lu: They also show the model preserves video action consistency even under simulation domain shifts, even in challenging nighttime and foggy driving conditions >

Meng: From a deployment standpoint, the flexibility is interesting because they can instantiate only the requested future token groups for joint, video only, or trajectory only prediction without changing the learned dependency structure >

Lalam: I'm confident that this unified approach learns transferable planning priors instead of just fitting specific NAVSIM trajectories >

Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye

University of Twente University of Cambridge University of Bath

cs.CV

Submitted: 2026-07-06

Updated: 2026-11-16

Comments: Accepted by NeurIPS 2026. Code: https://github.com/xiaomi-mlab/UNIVERSE

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 98/100

The gist: World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal…

Key concepts

Shared DiT Backbone
Instead of separate models for video and action, UNIVERSE uses one Diffusion Transformer (DiT) for both future video latents and ego-trajectory tokens. This forces the model to learn a single, unified representation of world dynamics that simultaneously supports both visual prediction and motion planning. This shared structure is key to transferring video knowledge directly into trajectory generation.
Mask-Modulated Visibility Mask
This mask controls how information flows between different parts of the model. It shares historical context across modalities while explicitly blocking direct attention between future video tokens and trajectory tokens during training. This prevents 'future target leakage,' ensuring that the video supervision only guides the trajectory prediction without corrupting it.
Flow Matching Loss
This is a specific training objective used to supervise the model's behavior on future data. It combines a loss for predicting future video latents with a loss for predicting trajectories. By optimizing this combined loss, UNIVERSE ensures that the video prediction provides dense, co-training supervision directly shaping the trajectory denoising process.

Terminology

Summary

World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality <ref:2607. p. 0

Where the paper says it

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation Mengmeng Liu1 Diankun Zhang2 Jiuming Liu3 Jianfeng Cui2 Hongwei Xie2 Guang Chen2 Hangjun Ye2 Francesco Nex1 Hao Cheng1 Michael Ying Yang4,* 1University of Twente 2Xiaomi EV 3University of Cambridge 4University of Bath *Corresponding author. Abstract World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality.

05133#pg2> The paper proposes UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer <ref:2607. p. 0
Where the paper says it

Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer.

05133#pg3> By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization <ref:2607. p. 0
Where the paper says it

We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization.

05133#pg5>

Pruning redundant reasoning tokens

The core problem addressed is determining which architecture better transfers video-modeling benefits to trajectory generation <ref:2607.05133#pg3> Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch <ref:2607. p. 0

Where the paper says it

However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning.

05133#pg5> The authors propose treating future video prediction as dense dynamics supervision for the same parameters used in trajectory denoising <ref:2607. p. 9
Where the paper says it

5 Conclusion In this paper, we propose UNIVERSE, a unified video-action diffusion model for autonomous driving. Rather than simply adding future-video prediction, UNIVERSE lets dense video supervision update the same DiT parameters used for trajectory denoising, enabling video-learned scene dynamics and temporal priors to directly regularize action generation.

05133#pg3> UNIVERSE places historical observations, future video latents, and ego-trajectory tokens in one shared DiT <ref:2607. p. 1
Where the paper says it

Based on this insight, we propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer [23]. Unlike cascaded or two-DiT WAMs [11, 24], UNIVERSE places historical observations, future video latents, and ego-trajectory tokens in one shared DiT.

05133#pg5> This design encourages a shared worldaction representation by having video and trajectory losses update the same attention layers and feedforward blocks <ref:2607. p. 1
Where the paper says it

Because video and trajectory losses update the same attention layers and feedforward blocks, dense video dynamics directly shape the parameters used for trajectory denoising, encouraging a shared worldaction representation. This design also improves efficiency over the Two-DiT variant, reducing trajectory inference latency, inference memory, and training memory, as later shown in Table 5.

05133#pg5>

How the training signal is built

The training objective combines a flow-matching loss for future video supervision and a trajectory prediction loss <ref:2607.05133#pg5> The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility mask <ref:2607. p. 4

Where the paper says it

The noisy sample and target velocity are Ym p = (1 − p)ϵ m + pYm 0 , Y˙ m p = Ym 0 − ϵ m. (4) The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility mask: (vˆ act θ , vˆ vid θ ) = fθ(Yact p , Yvid p , p, Cs, Ts; M), (5) where fθ denotes the shared DiT with modality-specific prediction heads.

05133#pg5> The final objective is L = Lact + λvidLvid, where λvid balances the two objectives <ref:2607. p. 4
Where the paper says it

The final objective is L = Lact + λvidLvid, Lm = E vˆ m θ − Y˙ m p 2 2 , m ∈ {act, vid}. (6) Here, λvid balances the two objectives.

05133#pg5> This ensures that future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation <ref:2607. p. 4
Where the paper says it

(6) Here, λvid balances the two objectives. By default, λvid > 0, so future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation.

05133#pg5>

Where the gains come from

The key finding is that architecture matters, as video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone <ref:2607.05133#pg3> The Modality-Decoupling Visibility Mask shares historical context across modalities while blocking mutual attention between future video and trajectory tokens <ref:2607. p. 0

Where the paper says it

By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens.

05133#pg5> This mask enables efficient trajectory-only inference by removing future-video denoising at test time, achieving a 4. p. 0
Where the paper says it

To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents futuretarget leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a 4.

3× speedup while maintaining the generalization benefit of video co-training <ref:2607. p. 2
Where the paper says it

• We enable efficient trajectory-only inference by removing future-video denoising at test time, achieving a 4. 3× speedup while maintaining the generalization benefit of video cotraining.

05133#pg5> Experiments show that UNIVERSE achieves 91. p. 0
Where the paper says it

The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.

0 PDMS on NAVSIM (vs. p. 0
Where the paper says it

Experiments show that UNIVERSE achieves 91. 0 PDMS on NAVSIM (vs.

89.6 for the Two-DiT variant) and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning <ref:2607. p. 0
Where the paper says it

0 PDMS on NAVSIM (vs. 6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.

05133#pg5>

The authors introduced several variants to isolate the effects of architectural factors, such as video co-training, mask-based causal decoupling, and shared DiT parameterization <ref:2607.05133#pg5> The UNIVERSE w. p. 0

Where the paper says it

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation Mengmeng Liu1 Diankun Zhang2 Jiuming Liu3 Jianfeng Cui2 Hongwei Xie2 Guang Chen2 Hangjun Ye2 Francesco Nex1 Hao Cheng1 Michael Ying Yang4,* 1University of Twente 2Xiaomi EV 3University of Cambridge 4University of Bath *Corresponding author. Abstract World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality.

o. Mask variant obtains comparable indomain performance with a PDMS of 90. p. 8
Where the paper says it

Visibility decoupling enables flexible deployment. The no-mask variant obtains comparable indomain performance with a PDMS of 90.

9 but does not translate into better deployment behavior <ref:2607. p. 8
Where the paper says it

The no-mask variant obtains comparable indomain performance with a PDMS of 90. 9, but this does not translate into better deployment behavior.

05133#pg5> The no-mask variant relies on coupled video-action rollout and cannot remove one future modality without changing the learned dependency structure <ref:2607. p. 8
Where the paper says it

9, but this does not translate into better deployment behavior. Because future video and trajectory tokens can attend to each other, the model relies on coupled video-action rollout and cannot remove one future modality without changing the learned dependency structure.

05133#pg5>

The shared denoising backbone matters, as UNIVERSE Two-DiT preserves both video and trajectory objectives but separates them into two diffusion transformers <ref:2607.05133#pg5> This results in the NAVSIM PDMS dropping to 89. p. 6

Where the paper says it

2 Comparison Results on NAVSIM v1 Table 1 reports closed-loop planning results on NAVSIM v1. UNIVERSE achieves a PDMS of 91.

6 and weaker zero-shot performance compared to the shared-DiT UNIVERSE <ref:2607. p. 8
Where the paper says it

Although it still benefits from video co-training at the system level, its NAVSIM PDMS drops to 89. 6, and its zero-shot performance is consistently weaker than the shared-DiT UNIVERSE.

05133#pg5> Visibility decoupling enables flexible deployment, allowing trajectory-only inference by removing future video denoising at test time which reduces inference time by 4. p. 0
Where the paper says it

To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents futuretarget leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a 4.

3× <ref:2607. p. 0
Where the paper says it

Compared with sparse trajectory supervision, future world modeling provides dense signals about Preprint. arXiv:2607.

05133#pg5> The results demonstrate that shared-parameter video-action co-training with mask-modulated modality decoupling is effective for generalizable autonomous-driving planners <ref:2607. p. 9
Where the paper says it

3× faster trajectory inference and lower inference/training memory than coupled or Two-DiT variants. These results demonstrate that shared-parameter video-action co-training with mask-modulated modality decoupling is effective for generalizable autonomous-driving planners.

05133#pg5> The model achieves competitive visual generation quality while being optimized for planningoriented video-action co-training <ref:2607. p. 7
Where the paper says it

82 Table 3: Quantitative evaluation of future-video generation on the nuScenes validation set [57]. UNIVERSE achieves competitive visual generation quality while being optimized for planningoriented video-action co-training.

05133#pg5> The model also supports joint, video-only, and trajectory-only prediction by instantiating only the requested future token groups <ref:2607. p. 4
Where the paper says it

3 Flexible Inference Controller The proposed mask enables flexible inference by instantiating only the requested future token groups. For trajectory-only rollout, future video tokens are removed and the model denoises As+1:s+K conditioned on Cs and Ts.

05133#pg5> The DPVO-based verification shows that the ego motion implied by the generated future videos is consistent with the trajectory predicted by the model <ref:2607. p. 6
Where the paper says it

The predicted trajectories are consistent with the generated future video frames, showing that UNIVERSE learns aligned visual dynamics and ego-motion within one unified generation framework. Generated videos by UNIVERSER 1s 2s 3s Predicted Trajectories by UNIVERSER Generated videos by UNIVERSER Two DIT Ground Truth videos Predicted Trajectories by UNIVERSER Two DIT 0s Figure 4: Zero-shot driving-intention comparison.

05133#pg5>

The paper concludes that UNIVERSE achieves state-of-the-art NAVSIM performance and strong zero-shot generalization to nuScenes and Bench2Drive <ref:2607.05133#pg5> The model is shown to preserve video-action consistency under simulation-domain shifts, even in challenging nighttime and foggy driving conditions <ref:2607.05133#pg5> Future work may focus on lighter video tokenizers or smaller DiT backbones to reduce the computational cost of training large video-action models <ref:2607. p. 2

Where the paper says it

In contrast, our work takes a planning-oriented perspective: UNIVERSE uses future video prediction as dense training-time supervision for trajectory generation, rather than treating video generation only as a standalone simulation task. 2 World–Action Models for Autonomous Driving Control Beyond pure scene generation, world models have increasingly been incorporated into autonomousdriving planning and control.

05133#pg5> The paper uses public datasets, benchmarks, models, and evaluation tools according to their official licenses and terms of use <ref:2607. p. 5
Where the paper says it

Unless otherwise specified, our model only uses front-view camera input without map, box, depth, occupancy, or LiDAR supervision. For cross-dataset evaluation, we directly test the NAVSIM-trained model on the nuScenes validation split of 150 scenes from the official 1,000-scene dataset [57].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified video-action model UNIVERSE allows dense video supervision to directly shape trajectory denoising by co-training future video latents and ego-trajectory tokens within a single mask modulated Diffusion Transformer. This leads to stronger cross domain action generalization compared to architectures that separate video imagination from action prediction.

  2. Introduction of a Modality Decoupling Visibility Mask prevents causal leakage between future video and trajectory tokens by blocking mutual attention, which ensures that future target leakage is prevented during training and inference. This design enables trajectory only inference by removing future video denoising at test time, achieving a 4.3 times speedup over joint video action rollout while maintaining comparable planning accuracy.

  3. The model supports flexible inference modes: trajectory only rollout, video only rollout, or joint video action rollout by instantiating different future token groups under the same shared DiT weights. This flexibility allows for efficient deployment by removing the video branch at test time for trajectory only inference without altering the learned dependency structure.

  4. Training with a flow matching loss on both future trajectory tokens and future video latents ensures that future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation. This shared parameterization, rather than separate modules, is shown to be most effective when video and trajectory losses update the same generative backbone.

  5. The model demonstrates strong zero shot transfer to nuScenes and Bench2Drive without fine tuning, achieving a significant reduction in L2 error and collision rate compared to prior methods like PWM [51] or DriveVLA-W0 [52]. This capability confirms that unified video-action co-training learns transferable planning priors rather than merely fitting NAVSIM-specific trajectories.

  6. UNIVERSE achieves superior video action consistency, with an average DPVO based error of 0.13 compared to 0.15 and 0.19 for other variants, indicating that the coupling by parameters and decoupling by visibility design preserves causal structure while maximizing alignment between visual dynamics and predicted ego motion under domain shift.

Abstract

World Action Models (WAMs) can improve action generalization in autonomous driving by learning scene dynamics through dense future-video supervision. However, deploying such models often requires costly future-video generation, while architectures that bypass video rollout typically separate video and action parameters or alter their learned dependencies at inference. We introduce UNIVERSE, a unified video-action model built on a single mask-modulated Diffusion Transformer (DiT) that combines shared-parameter learning with modality-independent inference. Its Modality-Decoupling Visibility Mask allows future video and trajectory tokens to attend to a common historical context while blocking mutual attention between the two future modalities. As a result, future-video supervision directly updates the parameters used for trajectory denoising, yet video tokens can be omitted at test time without changing the trajectory branch's training-time attention dependencies. The same model supports trajectory-only, video-only, and joint video-action generation. Trajectory-only inference achieves a 4.3x speedup over coupled video-action rollout with comparable planning accuracy. UNIVERSE achieves 91.0 PDMS on NAVSIM, compared with 89.6 for a Two-DiT variant, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning. Controlled ablations further support the complementary benefits of shared parameters, dense video co-training, and modality-decoupled visibility.

Sources

Related papers