CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting
Quang Minh Dinh, Tuan Kiet Doan
cs.CV, cs.AI, cs.CL
Submitted: 2026-08-07
Updated: 2026-08-11
Comments: Accepted at ECCVW 2026
Code: https://github.com/commaai/commavq
Project page: https://quangminhdinh.github.io/CosmosAlign
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Generative traffic video forecasting aims to synthesize long-horizon, temporally coherent future videos of traffic scenes from a short observation history and textual descriptions.
Terminology
Abstract
Generative traffic video forecasting aims to synthesize long-horizon, temporally coherent future videos of traffic scenes from a short observation history and textual descriptions. In this paper, we present CosmosAlign, a generative traffic video forecasting framework built upon the pretrained Cosmos3-Nano world foundation model. Our approach is motivated by the observation that successfully adapting large pretrained world models to downstream forecasting tasks depends primarily on distribution alignment rather than increased model capacity. To this end, we propose a two-stage LoRA adaptation strategy that first aligns the conditioning-mode distribution with the target forecasting task, and then aligns the training captions with the model's native structured prompting interface through an LLM-based re-captioning pipeline. During inference, we further improve prediction quality using a fully training-free procedure consisting of consensus-based medoid sample selection and motion-adaptive blending of static scene regions. CosmosAlign achieves a final score of 76.49 on the AI City Challenge 2026 Track 5 benchmark, ranking first on the final leaderboard. Our code is publicly available at https://quangminhdinh.github.io/CosmosAlign/.
Sources
- Cosmos 3: Omnimodal World Models for Physical AI
- Cosmos World Foundation Model Platform for Physical AI
- World Simulation with Video Foundation Models for Physical AI
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Revisiting Feature Prediction for Learning Visual Representations from Video
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- World Models
- Dream to Control: Learning Behaviors by Latent Imagination
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
- A Comprehensive Survey on World Models for Embodied AI
- Dolphins: Multimodal Language Model for Driving
- A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
- GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
- Wan: Open and Advanced Large-Scale Video Generative Models
- Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models