NTR: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving
cs.CV, cs.RO
Submitted: 2026-05-29
Updated: 2026-10-28
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch tokens into compact scene tokens for downstream trajectory…
Terminology
Abstract
Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs by compressing dense image patch tokens into compact scene tokens for downstream trajectory generation and scoring. While these scene tokens form a compact visual bottleneck for the planner, they receive supervision solely from the planning objective, providing limited constraints on the encoded visual information. To address this limitation, we introduce Neural Token Reconstruction (NTR), a representation learning framework to directly constrain the compact scene-token bottleneck in perception-free driving. NTR introduces a self-distillation masked latent reconstruction objective that reconstructs masked patch-level latent features using only compact scene tokens as reconstruction memory. This forces reconstruction gradients to pass exclusively through the scene-token bottleneck, encouraging scene tokens to preserve richer and less redundant visual representations for planning. We further introduce semantic priors derived from foundation-model annotations as a weak semantic interface biasing reconstruction targets toward driving-related structures without introducing explicit perception heads. All auxiliary reconstruction components are removed at inference time, leaving the deployed planner unchanged. NTR achieves state-of-the-art performance on three public autonomous driving benchmarks, including 8.0461 RFS on Waymo E2E and 94.1 PDMS / 90.9 EPDMS on NavSim1&2. The learned scene tokens exhibit lower pairwise redundancy and higher effective rank, indicating that effective bottleneck supervision improves both compact visual representation learning and planning performance.
Sources
- End to End Learning for Self-Driving Cars
- Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
- iPad: Iterative Proposal-centric End-to-End Autonomous Driving
- SparseDriveV2: Scoring is All You Need for End-to-End Autonomous Driving
- CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planning
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver
- Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving
- IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
- FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encoder
- HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios
- dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
- DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
- Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillation
- DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs