Generative Scenario Rollouts for End-to-End Autonomous Driving

summary

The gist

Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo)…

In short

Generative Scenario Rollouts (GeRo) is a framework for Vision-Language-Action (VLA) models to plan autonomous driving by jointly performing planning and generating future traffic scenes. It uses an autoregressive strategy to predict future latent tokens and actions, guided by reinforcement learning rewards that ensure both physical safety and alignment with natural language descriptions, leading to state-of-the-art performance.

Key concepts

Generative Scenario Rollouts (GeRo)
GeRo is a plug-and-play framework that unifies planning and scenario generation for VLA models. It uses an autoregressive strategy to predict future latent tokens and ego actions based on past context, allowing the model to generate long-horizon, temporally consistent driving scenarios.
Pretraining Loss (Lpre)
The overall pretraining loss combines three components: planning loss (for waypoint regression and collision avoidance), motion prediction loss (for agent classification and trajectory prediction), and VLA loss. This joint supervision helps bind linguistic knowledge with behavioral representations during the initial training phase.
Reinforcement Learning Feedback Loss (GRPO)
This involves using GRPO to optimize scenario rollouts by calculating rewards that prioritize safety metrics like collision avoidance and time-to-collision (TTC). This feedback ensures the generated scenarios are physically safe and encourage longer distances between potential conflicts.
Language Alignment Loss (LVLA)
This loss component is added as a reward during rollout to enforce semantic correctness. It measures the alignment between the language outputs generated by the model and reference descriptions, ensuring that the planned scenario matches the intended linguistic context.

Terminology used across episodes

This episode discusses

The paper

Generative Scenario Rollouts for End-to-End Autonomous Driving · Read on arXiv

Qualcomm AI Research · Qualcomm Technologies, Inc.

Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and underutilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Generative Scenario Rollouts for End-to-End Autonomous Driving".

Jane: Here's the gist of the paper. Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo), a plugandplay framework that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy These results highlight the promise of generative, languageconditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.

Tom: The Proposed Framework GeRo is a plugandplay framework for VLA models that unifies planning and autoregressive scenario generation The framework has two main phases: Pretraining: GeRo is trained to encode ego and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks This stage is crucial for binding linguistic and behavioral representations.

Jane: Language-conditioned Scenario Rollout: The model autoregressively predicts future latent tokens and ego-aligned action descriptions based on past latent tokens Predictions are stabilized using a consistency loss that mitigates drift and enforces text-action consistency.

Tom: The Training Signal is Built The training signal involves multiple supervision mechanisms across the two stages Pretraining Loss: The overall pretraining loss is defined as Lpre = Lplan + Lmot + LVLA.

Title and authors: Tom: So, "Generative Scenario Rollouts for End-to-End Autonomous Driving" is the title. It tells you exactly what they're doing: using generative scenario rollouts to improve end-to-end autonomous driving systems.

Jane: It's about taking planning and generating future traffic scenes together through this rollout strategy. The authors, Rajeev Yasarla, Deepti Hegde, Shizhong Han, Hsin-Pai Cheng, Yunxiao Shi, Meysam Sadeghigooghari, Shweta Mahajan, Apratim Bhattacharyya and Litian Liu are the main people behind this work.

Lu: I see they're focusing on how to move beyond just relying on sparse trajectory annotations and instead utilize the potential of generative models for scenario-level reasoning and exploration >

Meng: It seems like they are addressing a problem where current language supervision often describes what is happening rather than how actions unfold, which limits the model’s ability to capture procedural nuances necessary for planning and execution.

Lalam: They propose this framework as a plug-and-play training framework that unifies Vision Language Action models with scenario generation for language-aligned action prediction and improved planning robustness >

The paper's summary: Tom: To summarize, the core of Generative Scenario Rollouts is a two-phase approach. First, they pretrain the Vision Language Action model to encode ego vehicle and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks >

Jane: That first stage is really about binding linguistic and behavioral representations together so that the model can actually understand what it means when you ask it a question in English about the scene >

Lu: Then they move to the second stage where this model performs language-conditioned autoregressive generation. Given a scenario description and ego-action questions, it predicts future tokens for T steps, decoding those into ego waypoints, multi-agent trajectories, and language outputs >

Meng: The rollout is optimized with consistency loss to make sure the predictions stay temporally consistent and the reinforcement feedback loss to incorporate safety metrics like collision avoidance >

Lalam: And they also have this specific language alignment loss called LVLA which rewards the model for generating language outputs that match reference descriptions, which helps with semantic correctness >

The paper's improvements: Tom: The authors claim this unification of planning and generation leads to state-of-the-art performance on benchmarks. On Bench2Drive, they improved the driving score by plus fifteen point seven and the success rate by plus twenty six point two >

Jane: And they have results from nuScenes too, where using Qwen for their model reduced the L2 trajectory error from zero point nine six to zero point three one which is a sixty-seven point seven percent reduction >

Lu: The ablation study shows that incorporating scenario descriptions and visual question answering improves baseline performance, indicating that language grounding during pretraining enhances planning robustness >

Meng: And they found that using collision and time-to-collision rewards during scenario rollouts boosts the driving score by plus four point one and the success rate by plus five point three >

Lalam: They also showed that introducing ego-action question answer pairs and leveraging the language alignment loss LVLA as an additional reward further enhances performance by adding plus one point one to the driving score and two point one percent to the success rate >

Conclusion: Tom: So, to wrap up, Generative Scenario Rollouts is about unifying planning, multi-agent motion prediction, and scenario generation for robust end-to-end planning with Vision Language Action models. It sets a new state-of-the-art performance level.

Jane: It shows how aligning textual reasoning with action execution through reinforcement learning with GRPO can lead to safer and more interpretable planning behavior in complex driving environments.

Lu: The key finding is that scenario-grounded rollouts enable the ego vehicle to dynamically adapt its trajectory in response to evolving scene context, like slowing near crosswalks or yielding vehicles >

Meng: It also shows strong zero-shot sim-to-real generalization across different datasets, which is important when you have to deploy these models in the real world.

Lalam: Ultimately, this paper on Generative Scenario Rollouts for End-to-End Autonomous Driving demonstrates how scenario rollouts enable safe, context aware planning by aligning policy optimization with both physical safety and semantic correctness >

More episodes

← Home