MM-Future: Multi-Mode Joint World-Action Modeling for Autonomous Driving
cs.CV
Submitted: 2026-09-17
Updated: 2026-09-17
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Autonomous driving involves coupled decision-making and scene evolution under multi-mode uncertainty.
Terminology
Abstract
Autonomous driving involves coupled decision-making and scene evolution under multi-mode uncertainty. To capture this coupling and uncertainty, we introduce MM-Future, a world-action model that generates multiple paired scene-action hypotheses and models bidirectional interaction within each pair. Each hypothesis is initialized from a structured action prior and an independent future scene source, which are then co-evolved through a modality-aware diffusion Transformer. To support efficient multi-mode rollout, MM-Future compresses multi-view video into planning-oriented representations, dubbed MM-Tokens. Finally, a future-conditioned proposal scorer ranks trajectory candidates by shared history context and their paired predicted future. On NAVSIM navtest, MM-Future achieves 94.0 PDMS and 91.5 EPDMS, while attaining a 32.3 HD-Score in zero-shot closed-loop evaluation on HUGSIM. Ablations show consistent improvements over both single-mode and action-only variants, validating the benefit of multi-mode joint world-action modeling.
Sources
- CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planning
- Intend, Reflect, Refine: An Adaptive Multimodal Reflection Framework for Autonomous Driving
- DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving
- DriveFuture: Future-Aware Latent World Models for Autonomous Driving
- Map-World: Masked Action planning and Path-Integral World Model for Autonomous Driving
- Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation
- Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
- Generalized Trajectory Scoring for End-to-end Multimodal Planning
- UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
- DriveVA: Video Action Models are Zero-Shot Drivers
- UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning
- DynFlowDrive: Flow-Based Dynamic World Modeling for Autonomous Driving
- The DAWN of World-Action Interactive Models
- DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving
- GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Driving
- SparseDriveV2: Scoring is All You Need for End-to-End Autonomous Driving
- Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving
- SparseWorld: Enhancing End-to-End Autonomous Driving via World Models with Sparse Scene Representation
- Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learning
- IDOL: Inverse-Dynamics-Guided Future Prediction for End-to-End Autonomous Driving
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models