UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation".
Jane: Here's the gist of the paper. World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality <ref:2607.05133#pg2> The paper proposes UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer <ref:2607.05133#pg3> By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization <ref:2607.05133#pg5> Pruning redundant reasoning tokens The core problem addressed is determining which architecture better transfers video-modeling benefits to trajectory generation <ref:2607.05133#pg3> Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch <ref:2607.05133#pg5> The authors propose treating future video prediction as dense dynamics supervision for the same parameters used in trajectory denoising <ref:2607.05133#pg3> UNIVERSE places historical observations, future video latents, and ego-trajectory tokens in one shared DiT <ref:2607.05133#pg5> This design encourages a shared worldaction representation by having video and trajectory losses update the same attention layers and feedforward blocks <ref:2607.05133#pg5> How the training signal is built The training objective combines a flow-matching loss for future video supervision and a trajectory prediction loss <ref:2607.05133#pg5> The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility mask <ref:2607.05133#pg5> The final objective is L = Lact + λvidLvid, where λvid balances the two objectives <ref:2607.05133#pg5> This ensures that future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation <ref:2607.05133#pg5> Where the gains come from The key finding is that architecture matters, as video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone <ref:2607.05133#pg3> The Modality-Decoupling Visibility Mask shares historical context across modalities while blocking mutual attention between future video and trajectory tokens <ref:2607.05133#pg5> This mask enables efficient trajectory-only inference by removing future-video denoising at test time, achieving a 4.3× speedup while maintaining the generalization benefit of video co-training <ref:2607.05133#pg5> Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs.
Tom: 89.6 for the Two-DiT variant) and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning <ref:2607.05133#pg5> Pruning redundant reasoning tokens The authors introduced several variants to isolate the effects of architectural factors, such as video co-training, mask-based causal decoupling, and shared DiT parameterization <ref:2607.05133#pg5> The UNIVERSE w.o.
Jane: Mask variant obtains comparable indomain performance with a PDMS of 90.9 but does not translate into better deployment behavior <ref:2607.05133#pg5> The no-mask variant relies on coupled video-action rollout and cannot remove one future modality without changing the learned dependency structure <ref:2607.05133#pg5> Where the gains come from The shared denoising backbone matters, as UNIVERSE Two-DiT preserves both video and trajectory objectives but separates them into two diffusion transformers <ref:2607.05133#pg5> This results in the NAVSIM PDMS dropping to 89.6 and weaker zero-shot performance compared to the shared-DiT UNIVERSE <ref:2607.05133#pg5> Visibility decoupling enables flexible deployment, allowing trajectory-only inference by removing future video denoising at test time which reduces inference time by 4.3× <ref:2607.05133#pg5> The results demonstrate that shared-parameter video-action co-training with mask-modulated modality decoupling is effective for generalizable autonomous-driving planners <ref:2607.05133#pg5> The model achieves competitive visual generation quality while being optimized for planningoriented video-action co-training <ref:2607.05133#pg5> The model also supports joint, video-only, and trajectory-only prediction by instantiating only the requested future token groups <ref:2607.05133#pg5> The DPVO-based verification shows that the ego motion implied by the generated future videos is consistent with the trajectory predicted by the model <ref:2607.05133#pg5> Pruning redundant reasoning tokens The paper concludes that UNIVERSE achieves state-of-the-art NAVSIM performance and strong zero-shot generalization to nuScenes and Bench2Drive <ref:2607.05133#pg5> The model is shown to preserve video-action consistency under simulation-domain shifts, even in challenging nighttime and foggy driving conditions <ref:2607.05133#pg5> Future work may focus on lighter video tokenizers or smaller DiT backbones to reduce the computational cost of training large video-action models <ref:2607.05133#pg5> The paper uses public datasets, benchmarks, models, and evaluation tools according to their official licenses and terms of use <ref:2607.
Title and authors: Tom: So we're looking at this paper called "UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation," and the authors are Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Francesco Nex and Hao Cheng and Michael Ying Yang.
Jane: That title sounds pretty technical right? It tells us they're trying to unify video modeling with action planning in autonomous driving using some kind of masking trick.
Lu: Exactly. The core idea is addressing the question of which architecture actually transfers those video modeling benefits over to trajectory generation, because right now, most systems separate them.
Meng: Separating them sounds simpler for training, but it means the video model only gives a hint indirectly to the planning part of the system >
Tom: Right. So this paper is proposing UNIVERSE as a unified video-action model built on just one mask modulated Diffusion Transformer instead of having separate parts for imagining videos and predicting trajectories.
Jane: That's what page one sets up, showing that existing cascaded or dual diffusion transformer designs separate video imagination from action prediction, which weakens the transfer of learned world dynamics to the trajectory branch >
Lu: They argue that this separation means the action model might still overfit dataset specific driving priors because it only gets an indirect regularization from the video model >
Meng: So they are proposing a shared generative backbone where both video and trajectory tokens are co-trained, which is what I'm really curious about for deployment >
Tom: Right. They propose treating future video prediction as dense dynamics supervision for the same parameters used in trajectory denoising, which is a key difference from what exists out there >
Jane: And they introduce this Modality Decoupling Visibility Mask to handle the training side of this unification, which is pretty clever because it shares historical context across modalities while blocking mutual attention between future video and trajectory tokens >
The paper's summary: Tom: So what's the actual proposal for UNIVERSE? Page three lays out that they propose a unified video-action model that co-trains future video latents and ego trajectory tokens within one shared Diffusion Transformer >
Jane: Which means they are using this single shared generative parameter set to allow dense video supervision to directly regularize trajectory denoising, which is a big deal because it connects the two domains much more directly >
Lu: The main mechanism here is that they co-train these tokens within one shared Diffusion Transformer, which encourages a shared worldaction representation by having video and trajectory losses update the same attention layers and feedforward blocks >
Meng: From my side, if they are sharing parameters, I worry about how much redundancy they're actually getting in terms of learning new things versus just reinforcing what's already there >
Tom: They build the training objective by combining a flow matching loss for future video supervision and a trajectory prediction loss, and this combined signal is fed into the shared Diffusion Transformer with a visibility mask >
Jane: The objective function they use is L equals Lact plus lambda vidLvid, where lambda vid balances these two objectives, making sure that future-video prediction provides dense co-training supervision for the same parameters used by trajectory generation >
Lu: That shared parameterization is what they claim is most effective when video and trajectory losses update the same generative backbone because it forces them to learn a single representation of the world action >
The paper's improvements: Tom: Now for what they claim as the main gains, UNIVERSE shows that architecture really matters, and video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone >
Jane: They introduce this Modality-Decoupling Visibility Mask to share historical context across modalities while blocking mutual attention between future video and trajectory tokens >
Lu: That mask is crucial because it prevents causal leakage in the unified video-action training, which means it stops future target leakage during training and inference >
Meng: I like that they also show how this design enables efficient trajectory-only inference by removing future video denoising at test time, achieving a four point three times speedup over joint video action rollout > <ref:2607.05133#pg1>
Tom: That speedup is significant because it keeps the generalization benefit of video co-training while making the model much faster for actual planning when you're driving >
Jane: They show state-of-the-art closed loop planning and strong zero shot action generalization on NAVSIM, nuScenes, and Bench2Drive without needing any fine tuning >
Lu: The experiments show that UNIVERSE achieves ninety-one point zero PDMS on NAVSIM compared to eighty-nine point six for the Two DiT variant, proving that sharing the backbone is what matters for transferring video learned world dynamics >
Conclusion: Tom: So to wrap up, the core finding of this paper on UNIVERSE is that shared parameterization and mask modulated modality decoupling are effective for generalizable autonomous driving planners >
Jane: It means they achieve state-of-the-art performance on NAVSIM and strong zero shot generalization to nuScenes and Bench2Drive without fine tuning >
Lu: They also show the model preserves video action consistency even under simulation domain shifts, even in challenging nighttime and foggy driving conditions >
Meng: From a deployment standpoint, the flexibility is interesting because they can instantiate only the requested future token groups for joint, video only, or trajectory only prediction without changing the learned dependency structure >
Lalam: I'm confident that this unified approach learns transferable planning priors instead of just fitting specific NAVSIM trajectories >
Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye
University of Twente University of Cambridge University of Bath
cs.CV
Submitted: 2026-07-06
Updated: 2026-11-16
Comments: Accepted by NeurIPS 2026. Code: https://github.com/xiaomi-mlab/UNIVERSE
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 98/100
The gist: World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal…
Key concepts
- Shared DiT Backbone
- Instead of separate models for video and action, UNIVERSE uses one Diffusion Transformer (DiT) for both future video latents and ego-trajectory tokens. This forces the model to learn a single, unified representation of world dynamics that simultaneously supports both visual prediction and motion planning. This shared structure is key to transferring video knowledge directly into trajectory generation.
- Mask-Modulated Visibility Mask
- This mask controls how information flows between different parts of the model. It shares historical context across modalities while explicitly blocking direct attention between future video tokens and trajectory tokens during training. This prevents 'future target leakage,' ensuring that the video supervision only guides the trajectory prediction without corrupting it.
- Flow Matching Loss
- This is a specific training objective used to supervise the model's behavior on future data. It combines a loss for predicting future video latents with a loss for predicting trajectories. By optimizing this combined loss, UNIVERSE ensures that the video prediction provides dense, co-training supervision directly shaping the trajectory denoising process.
Terminology
Summary
World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality <ref:2607. p. 0 UNIVERSE: Unified Video Action Models
for Autonomous Driving with Flexible
Mask-Modulated Modality Generation
Mengmeng Liu1 Diankun Zhang2 Jiuming Liu3 Jianfeng Cui2
Hongwei Xie2 Guang Chen2 Hangjun Ye2
Francesco Nex1 Hao Cheng1 Michael Ying Yang4,*
1University of Twente 2Xiaomi EV
3University of Cambridge 4University of Bath
*Corresponding author. Abstract
World Action Models (WAMs) have shown strong potential for improving action
generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory
branch: the action model may still overfit dataset-specific driving priors, while the
video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape
trajectory denoising, leading to stronger cross-domain action generalization.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The core problem addressed is determining which architecture better transfers video-modeling benefits to trajectory generation <ref:2607.05133#pg3> Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch <ref:2607. p. 0 However, it remains unclear
which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory
branch: the action model may still overfit dataset-specific driving priors, while the
video model only indirectly regularizes planning. 5 Conclusion
In this paper, we propose UNIVERSE, a unified video-action diffusion model for autonomous driving. Rather than simply adding future-video prediction, UNIVERSE lets dense video supervision update
the same DiT parameters used for trajectory denoising, enabling video-learned scene dynamics and
temporal priors to directly regularize action generation. Based
on this insight, we propose UNIVERSE,
a unified video-action model built upon
a single mask-modulated Diffusion Transformer [23]. Unlike cascaded or two-DiT
WAMs [11, 24], UNIVERSE places historical observations, future video latents,
and ego-trajectory tokens in one shared
DiT. Because video and trajectory losses
update the same attention layers and feedforward blocks, dense video dynamics directly shape the parameters used for trajectory denoising, encouraging a shared worldaction representation. This design also improves efficiency over the Two-DiT variant,
reducing trajectory inference latency, inference memory, and training memory, as
later shown in Table 5.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The training objective combines a flow-matching loss for future video supervision and a trajectory prediction loss <ref:2607.05133#pg5> The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility mask <ref:2607. p. 4 The noisy sample and target velocity are
Ym
p = (1 − p)ϵ
m + pYm
0
, Y˙ m
p = Ym
0 − ϵ
m. (4)
The noisy future trajectory and video tokens are jointly fed into the shared DiT with the visibility
mask:
(vˆ
act
θ
, vˆ
vid
θ
) = fθ(Yact
p
, Yvid
p
, p, Cs, Ts; M), (5)
where fθ denotes the shared DiT with modality-specific prediction heads. The final objective is
L = Lact + λvidLvid, Lm = E
vˆ
m
θ − Y˙ m
p
2
2
, m ∈ {act, vid}. (6)
Here, λvid balances the two objectives. (6)
Here, λvid balances the two objectives. By default, λvid > 0, so future-video prediction provides
dense co-training supervision for the same DiT parameters used by trajectory generation.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The key finding is that architecture matters, as video co-training yields stronger action generalization when future video and trajectory denoising share the same generative backbone <ref:2607.05133#pg3> The Modality-Decoupling Visibility Mask shares historical context across modalities while blocking mutual attention between future video and trajectory tokens <ref:2607. p. 0 By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape
trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling
Visibility Mask, which shares historical context across modalities while blocking
mutual attention between future video and trajectory tokens. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling
Visibility Mask, which shares historical context across modalities while blocking
mutual attention between future video and trajectory tokens. This prevents futuretarget leakage and enables trajectory-only inference by removing future-video
denoising at test time, achieving a 4. • We enable efficient trajectory-only inference by removing future-video denoising at test
time, achieving a 4. 3× speedup while maintaining the generalization benefit of video cotraining. The same model also supports
video-only and joint video-action rollouts. Experiments show that UNIVERSE
achieves 91. Experiments show that UNIVERSE
achieves 91. 0 PDMS on NAVSIM (vs. 0 PDMS on NAVSIM (vs. 6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning,
while ablations confirm the importance of single-DiT unification, video co-training,
and mask-based modality decoupling.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The authors introduced several variants to isolate the effects of architectural factors, such as video co-training, mask-based causal decoupling, and shared DiT parameterization <ref:2607.05133#pg5> The UNIVERSE w. p. 0 UNIVERSE: Unified Video Action Models
for Autonomous Driving with Flexible
Mask-Modulated Modality Generation
Mengmeng Liu1 Diankun Zhang2 Jiuming Liu3 Jianfeng Cui2
Hongwei Xie2 Guang Chen2 Hangjun Ye2
Francesco Nex1 Hao Cheng1 Michael Ying Yang4,*
1University of Twente 2Xiaomi EV
3University of Cambridge 4University of Bath
*Corresponding author. Abstract
World Action Models (WAMs) have shown strong potential for improving action
generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. Visibility decoupling enables flexible deployment. The no-mask variant obtains comparable indomain performance with a PDMS of 90. The no-mask variant obtains comparable indomain performance with a PDMS of 90. 9, but this does not translate into better deployment behavior. 9, but this does not translate into better deployment behavior. Because future video and trajectory tokens can attend to each other, the model relies on coupled
video-action rollout and cannot remove one future modality without changing the learned dependency
structure.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The shared denoising backbone matters, as UNIVERSE Two-DiT preserves both video and trajectory objectives but separates them into two diffusion transformers <ref:2607.05133#pg5> This results in the NAVSIM PDMS dropping to 89. p. 6 2 Comparison Results on NAVSIM v1
Table 1 reports closed-loop planning results on NAVSIM v1. UNIVERSE achieves a PDMS of
91. Although it still benefits from video
co-training at the system level, its NAVSIM PDMS drops to 89. 6, and its zero-shot performance is
consistently weaker than the shared-DiT UNIVERSE. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling
Visibility Mask, which shares historical context across modalities while blocking
mutual attention between future video and trajectory tokens. This prevents futuretarget leakage and enables trajectory-only inference by removing future-video
denoising at test time, achieving a 4. Compared with sparse trajectory supervision, future world modeling provides dense signals about
Preprint. arXiv:2607. 3×
faster trajectory inference and lower inference/training memory than coupled or Two-DiT variants. These results demonstrate that shared-parameter video-action co-training with mask-modulated
modality decoupling is effective for generalizable autonomous-driving planners. 82
Table 3: Quantitative evaluation of future-video generation on the nuScenes validation set [57]. UNIVERSE achieves competitive visual generation quality while being optimized for planningoriented video-action co-training. 3 Flexible Inference Controller
The proposed mask enables flexible inference by instantiating only the requested future token groups. For trajectory-only rollout, future video tokens are removed and the model denoises As+1:s+K
conditioned on Cs and Ts. The predicted trajectories are
consistent with the generated future video frames, showing that UNIVERSE learns aligned visual
dynamics and ego-motion within one unified generation framework. Generated videos by UNIVERSER
1s 2s 3s
Predicted Trajectories by UNIVERSER
Generated videos by UNIVERSER Two DIT
Ground Truth videos
Predicted Trajectories by UNIVERSER Two
DIT
0s
Figure 4: Zero-shot driving-intention comparison.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper concludes that UNIVERSE achieves state-of-the-art NAVSIM performance and strong zero-shot generalization to nuScenes and Bench2Drive <ref:2607.05133#pg5> The model is shown to preserve video-action consistency under simulation-domain shifts, even in challenging nighttime and foggy driving conditions <ref:2607.05133#pg5> Future work may focus on lighter video tokenizers or smaller DiT backbones to reduce the computational cost of training large video-action models <ref:2607. p. 2 In contrast, our
work takes a planning-oriented perspective: UNIVERSE uses future video prediction as dense
training-time supervision for trajectory generation, rather than treating video generation only as a
standalone simulation task. 2 World–Action Models for Autonomous Driving Control
Beyond pure scene generation, world models have increasingly been incorporated into autonomousdriving planning and control. Unless otherwise specified, our model only uses front-view
camera input without map, box, depth, occupancy, or LiDAR supervision. For cross-dataset evaluation, we directly test the NAVSIM-trained model on the nuScenes
validation split of 150 scenes from the official 1,000-scene dataset [57].Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified video-action model UNIVERSE allows dense video supervision to directly shape trajectory denoising by co-training future video latents and ego-trajectory tokens within a single mask modulated Diffusion Transformer. This leads to stronger cross domain action generalization compared to architectures that separate video imagination from action prediction.
-
Introduction of a Modality Decoupling Visibility Mask prevents causal leakage between future video and trajectory tokens by blocking mutual attention, which ensures that
future target leakage
is prevented during training and inference. This design enables trajectory only inference by removing future video denoising at test time, achieving a 4.3 times speedup over joint video action rollout while maintaining comparable planning accuracy. -
The model supports flexible inference modes: trajectory only rollout, video only rollout, or joint video action rollout by instantiating different future token groups under the same shared DiT weights. This flexibility allows for efficient deployment by removing the video branch at test time for trajectory only inference without altering the learned dependency structure.
-
Training with a flow matching loss on both future trajectory tokens and future video latents ensures that
future-video prediction provides dense co-training supervision for the same DiT parameters used by trajectory generation.
This shared parameterization, rather than separate modules, is shown to be most effective whenvideo and trajectory losses update the same generative backbone.
-
The model demonstrates strong zero shot transfer to nuScenes and Bench2Drive without fine tuning, achieving a significant reduction in L2 error and collision rate compared to prior methods like PWM [51] or DriveVLA-W0 [52]. This capability confirms that
unified video-action co-training learns transferable planning priors rather than merely fitting NAVSIM-specific trajectories.
-
UNIVERSE achieves superior video action consistency, with an average DPVO based error of 0.13 compared to 0.15 and 0.19 for other variants, indicating that the
coupling by parameters and decoupling by visibility
design preserves causal structure while maximizing alignment between visual dynamics and predicted ego motion under domain shift.
Abstract
World Action Models (WAMs) can improve action generalization in autonomous driving by learning scene dynamics through dense future-video supervision. However, deploying such models often requires costly future-video generation, while architectures that bypass video rollout typically separate video and action parameters or alter their learned dependencies at inference. We introduce UNIVERSE, a unified video-action model built on a single mask-modulated Diffusion Transformer (DiT) that combines shared-parameter learning with modality-independent inference. Its Modality-Decoupling Visibility Mask allows future video and trajectory tokens to attend to a common historical context while blocking mutual attention between the two future modalities. As a result, future-video supervision directly updates the parameters used for trajectory denoising, yet video tokens can be omitted at test time without changing the trajectory branch's training-time attention dependencies. The same model supports trajectory-only, video-only, and joint video-action generation. Trajectory-only inference achieves a 4.3x speedup over coupled video-action rollout with comparable planning accuracy. UNIVERSE achieves 91.0 PDMS on NAVSIM, compared with 89.6 for a Two-DiT variant, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning. Controlled ablations further support the complementary benefits of shared parameters, dense video co-training, and modality-decoupled visibility.
Sources
- World Action Models are Zero-shot Policies
- World Models
- Causal World Modeling for Robot Control
- Unified Video Action Model
- Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets
- Fast-WAM: Do World Action Models Need Test-time Future Imagination?
- ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
- DriveVA: Video Action Models are Zero-Shot Drivers
- GigaWorld-Policy: An Efficient Action-Centered World--Action Model
- ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos
- mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
- Video Generators are Robot Policies
- Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- Large Video Planner Enables Generalizable Robot Control
- Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model
- MagicDrive: Street View Generation with Diverse 3D Geometry Control
- GAIA-1: A Generative World Model for Autonomous Driving
- GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
- MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Driving
- CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Driving
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs