Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving
cs.CV
Submitted: 2026-11-02
Updated: 2026-11-02
Comments: 23 pages, 7 figures
Project page: https://um-lab.github.io/ScaleDrive
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics.
Terminology
Abstract
End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- Qwen3-VL Technical Report
- Revisiting Feature Prediction for Learning Visual Representations from Video
- VaViM and VaVAM: Autonomous Driving through Video Generative Modeling
- Pseudo-Simulation for Autonomous Driving
- TrajDiff: End-to-end Autonomous Driving without Perception Annotation
- Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation
- Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving
- BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View
- SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving
- STELLAR: Scaling 3D Perception Large Models for Autonomous Driving
- Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
- MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction
- DINOv2: Learning Robust Visual Features without Supervision
- DINOv3
- Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos
- SparseDriveV2: Scoring is All You Need for End-to-End Autonomous Driving
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Wan: Open and Advanced Large-Scale Video Generative Models
- Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs