Generative Scenario Rollouts for End-to-End Autonomous Driving
summary
The gist
Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo)…
In short
Generative Scenario Rollouts (GeRo) is a framework for Vision-Language-Action (VLA) models to plan autonomous driving by jointly performing planning and generating future traffic scenes. It uses an autoregressive strategy to predict future latent tokens and actions, guided by reinforcement learning rewards that ensure both physical safety and alignment with natural language descriptions, leading to state-of-the-art performance.
Key concepts
- Generative Scenario Rollouts (GeRo)
- GeRo is a plug-and-play framework that unifies planning and scenario generation for VLA models. It uses an autoregressive strategy to predict future latent tokens and ego actions based on past context, allowing the model to generate long-horizon, temporally consistent driving scenarios.
- Pretraining Loss (Lpre)
- The overall pretraining loss combines three components: planning loss (for waypoint regression and collision avoidance), motion prediction loss (for agent classification and trajectory prediction), and VLA loss. This joint supervision helps bind linguistic knowledge with behavioral representations during the initial training phase.
- Reinforcement Learning Feedback Loss (GRPO)
- This involves using GRPO to optimize scenario rollouts by calculating rewards that prioritize safety metrics like collision avoidance and time-to-collision (TTC). This feedback ensures the generated scenarios are physically safe and encourage longer distances between potential conflicts.
- Language Alignment Loss (LVLA)
- This loss component is added as a reward during rollout to enforce semantic correctness. It measures the alignment between the language outputs generated by the model and reference descriptions, ensuring that the planned scenario matches the intended linguistic context.
Terminology used across episodes
This episode discusses
- Generative Scenario Rollouts for End-to-End Autonomous Driving · Paper Radio
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Qwen2.5-VL Technical Report
- The Llama 3 Herd of Models · Paper Radio
- ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
- EMMA: End-to-End Multimodal Model for Autonomous Driving
- Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving
- AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning
- Learning Personalized Driving Styles via Reinforcement Learning from Human Feedback
- Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving
- GPT-Driver: Learning to Drive with GPT
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DriveLM: Driving with Graph Visual Question Answering
- SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation
- Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving
- DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
- LLaMA: Open and Efficient Foundation Language Models
- OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
- Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes
- Diffusion-Based Planning for Autonomous Driving with Flexible Guidance
The paper
Generative Scenario Rollouts for End-to-End Autonomous Driving · Read on arXiv
Qualcomm AI Research · Qualcomm Technologies, Inc.
Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and underutilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Generative Scenario Rollouts for End-to-End Autonomous Driving".
Jane: Here's the gist of the paper. Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo), a plugandplay framework that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy These results highlight the promise of generative, languageconditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.
Tom: The Proposed Framework GeRo is a plugandplay framework for VLA models that unifies planning and autoregressive scenario generation The framework has two main phases: Pretraining: GeRo is trained to encode ego and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks This stage is crucial for binding linguistic and behavioral representations.
Jane: Language-conditioned Scenario Rollout: The model autoregressively predicts future latent tokens and ego-aligned action descriptions based on past latent tokens Predictions are stabilized using a consistency loss that mitigates drift and enforces text-action consistency.
Tom: The Training Signal is Built The training signal involves multiple supervision mechanisms across the two stages Pretraining Loss: The overall pretraining loss is defined as Lpre = Lplan + Lmot + LVLA.
Title and authors: Tom: So, "Generative Scenario Rollouts for End-to-End Autonomous Driving" is the title. It tells you exactly what they're doing: using generative scenario rollouts to improve end-to-end autonomous driving systems.
Jane: It's about taking planning and generating future traffic scenes together through this rollout strategy. The authors, Rajeev Yasarla, Deepti Hegde, Shizhong Han, Hsin-Pai Cheng, Yunxiao Shi, Meysam Sadeghigooghari, Shweta Mahajan, Apratim Bhattacharyya and Litian Liu are the main people behind this work.
Lu: I see they're focusing on how to move beyond just relying on sparse trajectory annotations and instead utilize the potential of generative models for scenario-level reasoning and exploration >
Meng: It seems like they are addressing a problem where current language supervision often describes what is happening rather than how actions unfold, which limits the model’s ability to capture procedural nuances necessary for planning and execution.
Lalam: They propose this framework as a plug-and-play training framework that unifies Vision Language Action models with scenario generation for language-aligned action prediction and improved planning robustness >
The paper's summary: Tom: To summarize, the core of Generative Scenario Rollouts is a two-phase approach. First, they pretrain the Vision Language Action model to encode ego vehicle and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks >
Jane: That first stage is really about binding linguistic and behavioral representations together so that the model can actually understand what it means when you ask it a question in English about the scene >
Lu: Then they move to the second stage where this model performs language-conditioned autoregressive generation. Given a scenario description and ego-action questions, it predicts future tokens for T steps, decoding those into ego waypoints, multi-agent trajectories, and language outputs >
Meng: The rollout is optimized with consistency loss to make sure the predictions stay temporally consistent and the reinforcement feedback loss to incorporate safety metrics like collision avoidance >
Lalam: And they also have this specific language alignment loss called LVLA which rewards the model for generating language outputs that match reference descriptions, which helps with semantic correctness >
The paper's improvements: Tom: The authors claim this unification of planning and generation leads to state-of-the-art performance on benchmarks. On Bench2Drive, they improved the driving score by plus fifteen point seven and the success rate by plus twenty six point two >
Jane: And they have results from nuScenes too, where using Qwen for their model reduced the L2 trajectory error from zero point nine six to zero point three one which is a sixty-seven point seven percent reduction >
Lu: The ablation study shows that incorporating scenario descriptions and visual question answering improves baseline performance, indicating that language grounding during pretraining enhances planning robustness >
Meng: And they found that using collision and time-to-collision rewards during scenario rollouts boosts the driving score by plus four point one and the success rate by plus five point three >
Lalam: They also showed that introducing ego-action question answer pairs and leveraging the language alignment loss LVLA as an additional reward further enhances performance by adding plus one point one to the driving score and two point one percent to the success rate >
Conclusion: Tom: So, to wrap up, Generative Scenario Rollouts is about unifying planning, multi-agent motion prediction, and scenario generation for robust end-to-end planning with Vision Language Action models. It sets a new state-of-the-art performance level.
Jane: It shows how aligning textual reasoning with action execution through reinforcement learning with GRPO can lead to safer and more interpretable planning behavior in complex driving environments.
Lu: The key finding is that scenario-grounded rollouts enable the ego vehicle to dynamically adapt its trajectory in response to evolving scene context, like slowing near crosswalks or yielding vehicles >
Meng: It also shows strong zero-shot sim-to-real generalization across different datasets, which is important when you have to deploy these models in the real world.
Lalam: Ultimately, this paper on Generative Scenario Rollouts for End-to-End Autonomous Driving demonstrates how scenario rollouts enable safe, context aware planning by aligning policy optimization with both physical safety and semantic correctness >
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI