Generative Scenario Rollouts for End-to-End Autonomous Driving

arXiv:2601.11475 · cs.CV · Submitted 2026-01-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Generative Scenario Rollouts for End-to-End Autonomous Driving".

Jane: Here's the gist of the paper. Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo), a plugandplay framework that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy These results highlight the promise of generative, languageconditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.

Tom: The Proposed Framework GeRo is a plugandplay framework for VLA models that unifies planning and autoregressive scenario generation The framework has two main phases: Pretraining: GeRo is trained to encode ego and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks This stage is crucial for binding linguistic and behavioral representations.

Jane: Language-conditioned Scenario Rollout: The model autoregressively predicts future latent tokens and ego-aligned action descriptions based on past latent tokens Predictions are stabilized using a consistency loss that mitigates drift and enforces text-action consistency.

Tom: The Training Signal is Built The training signal involves multiple supervision mechanisms across the two stages Pretraining Loss: The overall pretraining loss is defined as Lpre = Lplan + Lmot + LVLA.

Title and authors: Tom: So, "Generative Scenario Rollouts for End-to-End Autonomous Driving" is the title. It tells you exactly what they're doing: using generative scenario rollouts to improve end-to-end autonomous driving systems.

Jane: It's about taking planning and generating future traffic scenes together through this rollout strategy. The authors, Rajeev Yasarla, Deepti Hegde, Shizhong Han, Hsin-Pai Cheng, Yunxiao Shi, Meysam Sadeghigooghari, Shweta Mahajan, Apratim Bhattacharyya and Litian Liu are the main people behind this work.

Lu: I see they're focusing on how to move beyond just relying on sparse trajectory annotations and instead utilize the potential of generative models for scenario-level reasoning and exploration >

Meng: It seems like they are addressing a problem where current language supervision often describes what is happening rather than how actions unfold, which limits the model’s ability to capture procedural nuances necessary for planning and execution.

Lalam: They propose this framework as a plug-and-play training framework that unifies Vision Language Action models with scenario generation for language-aligned action prediction and improved planning robustness >

The paper's summary: Tom: To summarize, the core of Generative Scenario Rollouts is a two-phase approach. First, they pretrain the Vision Language Action model to encode ego vehicle and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks >

Jane: That first stage is really about binding linguistic and behavioral representations together so that the model can actually understand what it means when you ask it a question in English about the scene >

Lu: Then they move to the second stage where this model performs language-conditioned autoregressive generation. Given a scenario description and ego-action questions, it predicts future tokens for T steps, decoding those into ego waypoints, multi-agent trajectories, and language outputs >

Meng: The rollout is optimized with consistency loss to make sure the predictions stay temporally consistent and the reinforcement feedback loss to incorporate safety metrics like collision avoidance >

Lalam: And they also have this specific language alignment loss called LVLA which rewards the model for generating language outputs that match reference descriptions, which helps with semantic correctness >

The paper's improvements: Tom: The authors claim this unification of planning and generation leads to state-of-the-art performance on benchmarks. On Bench2Drive, they improved the driving score by plus fifteen point seven and the success rate by plus twenty six point two >

Jane: And they have results from nuScenes too, where using Qwen for their model reduced the L2 trajectory error from zero point nine six to zero point three one which is a sixty-seven point seven percent reduction >

Lu: The ablation study shows that incorporating scenario descriptions and visual question answering improves baseline performance, indicating that language grounding during pretraining enhances planning robustness >

Meng: And they found that using collision and time-to-collision rewards during scenario rollouts boosts the driving score by plus four point one and the success rate by plus five point three >

Lalam: They also showed that introducing ego-action question answer pairs and leveraging the language alignment loss LVLA as an additional reward further enhances performance by adding plus one point one to the driving score and two point one percent to the success rate >

Conclusion: Tom: So, to wrap up, Generative Scenario Rollouts is about unifying planning, multi-agent motion prediction, and scenario generation for robust end-to-end planning with Vision Language Action models. It sets a new state-of-the-art performance level.

Jane: It shows how aligning textual reasoning with action execution through reinforcement learning with GRPO can lead to safer and more interpretable planning behavior in complex driving environments.

Lu: The key finding is that scenario-grounded rollouts enable the ego vehicle to dynamically adapt its trajectory in response to evolving scene context, like slowing near crosswalks or yielding vehicles >

Meng: It also shows strong zero-shot sim-to-real generalization across different datasets, which is important when you have to deploy these models in the real world.

Lalam: Ultimately, this paper on Generative Scenario Rollouts for End-to-End Autonomous Driving demonstrates how scenario rollouts enable safe, context aware planning by aligning policy optimization with both physical safety and semantic correctness >

Qualcomm AI Research · Qualcomm Technologies, Inc.

cs.CV

Submitted: 2026-01-16

Updated: 2026-11-10

Comments: Accepted NeurIPS 2026

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 86/100

The gist: Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo)…

Key concepts

Generative Scenario Rollouts (GeRo)
GeRo is a plug-and-play framework that unifies planning and scenario generation for VLA models. It uses an autoregressive strategy to predict future latent tokens and ego actions based on past context, allowing the model to generate long-horizon, temporally consistent driving scenarios.
Pretraining Loss (Lpre)
The overall pretraining loss combines three components: planning loss (for waypoint regression and collision avoidance), motion prediction loss (for agent classification and trajectory prediction), and VLA loss. This joint supervision helps bind linguistic knowledge with behavioral representations during the initial training phase.
Reinforcement Learning Feedback Loss (GRPO)
This involves using GRPO to optimize scenario rollouts by calculating rewards that prioritize safety metrics like collision avoidance and time-to-collision (TTC). This feedback ensures the generated scenarios are physically safe and encourage longer distances between potential conflicts.
Language Alignment Loss (LVLA)
This loss component is added as a reward during rollout to enforce semantic correctness. It measures the alignment between the language outputs generated by the model and reference descriptions, ensuring that the planned scenario matches the intended linguistic context.

Terminology

Summary

Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems This paper proposes Generative Scenario Rollouts (GeRo), a plugandplay framework that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy These results highlight the promise of generative, languageconditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving. p. 0

Where the paper says it

Generative Scenario Rollouts for End-to-End Autonomous Driving Rajeev Yasarla1 , Deepti Hegde1 , Shizhong Han1 , Hsin-Pai Cheng1 , Yunxiao Shi1 , Meysam Sadeghigooghari2 , Shweta Mahajan1‡ , Apratim Bhattacharyya1 , Litian Liu1 , Risheek Garrepalli1 , Thomas Svantesson2 , Fatih Porikli1 , Hong Cai1 1 Qualcomm AI Research* 2 Qualcomm Technologies, Inc. Abstract Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems.

The Proposed Framework

GeRo is a plugandplay framework for VLA models that unifies planning and autoregressive scenario generation The framework has two main phases:

** Pretraining:**

GeRo is trained to encode ego and agent dynamics into a compact shared token space under supervision from planning, motion, and language tasks This stage is crucial for binding linguistic and behavioral representations.

Language-conditioned Scenario Rollout:

The model autoregressively predicts future latent tokens and ego-aligned action descriptions based on past latent tokens Predictions are stabilized using a consistency loss that mitigates drift and enforces text-action consistency.

The Training Signal is Built

The training signal involves multiple supervision mechanisms across the two stages

Pretraining Loss:

The overall pretraining loss is defined as Lpre = Lplan + Lmot + LVLA. The planning head is supervised by using two losses: an L1 loss for waypoint regression and a collision loss The motion prediction head is supervised using (i) focal loss [26] for classification of agent types, (ii) L1 loss for trajectory prediction, and (iii) L1 loss for 3D bounding box detection. p. 3

Where the paper says it

Pretraining loss functions The planning head is supervised by using two losses: an L1 loss for waypoint regression and a collision loss. The motion prediction head is supervised using (i) focal loss [26] for classification of agent types, (ii) L1 loss for trajectory prediction, and (iii) L1 loss for 3D bounding box detection.

Rollout Loss:

Temporal consistency is enforced through two complementary mechanisms: first, the latent distribution of rollout predictions are aligned with the pretrained latent distribution for future time steps using KL-Divergence Second, when ground truth data is available, imitation learning is applied by supervising the model with corresponding waypoints and motion trajectories to enforce scenario generation.

RL Feedback Loss:

Reinforcement learning based feedback with GRPO [34] is introduced for the scenario rollouts This loss incorporates rewards relating to collision avoidance, time-to-collision (TTC), and language-aligned scenario generation using question–answer pairs relating to the action of the ego-vehicle.

Where the Gains Come From

The improvements stem from integrating generative rollouts with reinforcement learning to achieve state-of-the-art performance The proposed reward function for GRPO jointly optimizes trajectory accuracy and semantic alignment with language descriptions during scenario rollouts This reward formulation aligns policy optimization with both physical safety and semantic correctness. The interactive Visual Question Answering (VQA) component grounds ego-vehicle intent in natural language, enhancing interpretability and providing a mechanism for languageguided reasoning in complex driving environments. p. 1

Where the paper says it

GeRo introduces an interactive Visual Question Answering (VQA) component that grounds ego-vehicle intent in natural language, allowing the model to answer scenariospecific questions during rollouts. This enhances interpretability and provides a mechanism for languageguided reasoning in complex driving environments.

Performance on Benchmarks

GeRo significantly outperforms VLA baselines [10] and other end-to-end motion planners On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. p. 0

Where the paper says it

On Bench2Drive, GeRo improves driving score and success rate by +15. 2, respectively.

In the open-loop evaluation on nuScenes, GeRo (Qwen) reduces L2 trajectory error from 0. p. 6
Where the paper says it

Open-loop evaluation uses L2 trajectory error and collision rate, following ST-P3 [13]. Implementation Details Model Architecture We implement the GeRo framework on two models.

96 to 0.31 (-67.7%) and collision rate from 0. p. 4
Where the paper says it

Empirically, this RL post-training significantly improves rollout quality. At each rollout step, the surrogate reward is computed as: Rt+∆ = −Coll({τ˜t+∆}) − TTC−1 ({τ˜t+∆}) − LVLA(˜qt+∆), (3) where Coll(·) computes the collision rate, and TTC(·) measures the time-to-collision for the predicted trajectories {τˆt+∆}.

60 to 0.14 (-76.7%). This demonstrates strong cross-dataset generalization enabled by scenario-grounded rollouts. p. 7
Where the paper says it

1%) with collisions at 0. 3%), highlighting strong cross-dataset generalization enabled by scenario-grounded rollouts.

Ablation Study Insights

The ablation study shows that incorporating scenario descriptions and visual question answering (VQA) improves baseline performance, indicating that language grounding during pretraining enhances planning robustness Further incorporating ground-truth trajectories for Lmot and Lplan boosts performance by an additional +6.6 DS and +10.3% SR. Using collision and time-to-collision (TTC) rewards during scenario rollouts improves DS by +4. p. 7

Where the paper says it

IDs 7–9 analyze the effect of GRPO-based reinforcement learning losses LGRPO. Using collision and time-to-collision (TTC) rewards during scenario rollouts improves DS by +4.

1 and SR by +5.3%. Finally, introducing ego-action question–answer pairs Q and leveraging the corresponding language alignment loss LVLA as an additional reward further enhances performance (+1. p. 7
Where the paper says it

1 and SR by +5. Finally, introducing ego-action question–answer pairs Q (Figure 3) and leveraging the corresponding language alignment loss LVLA as an additional reward further enhances performance (+1.

1 DS, +2.1% SR).

Conclusion

In this work, we introduced GeRo, a generative scenario rollout framework that unifies planning, multi-agent motion prediction, and scenario generation for robust end-to-end planning with Vision-Language-Action models By integrating reinforcement learning with GRPO and aligning textual reasoning with action execution, our proposed GeRo sets the new state-of-the-art performance with its robust, language-guided end-to-end planning. On standard autonomous driving benchmarks, GeRo provides significant improvements under both closed-loop and open-loop settings, as well as strong zero-shot sim-to-real generalization. p. 8

Where the paper says it

sets the new state-of-the-art performance with its robust, language-guided end-to-end planning. On standard autonomous driving benchmarks, GeRo provides significant improvements under both closed-loop and open-loop settings, as well as strong zero-shot sim-to-real generalization.

The ego vehicle dynamically adapts its trajectory in response to evolving scene context—slowing near crosswalks, yielding to oncoming or turning vehicles, and proceeding only after potential conflicts have cleared. p. 7
Where the paper says it

Qualitative results Figure 4 illustrates GeRo’s languageconditioned rollouts in complex urban environments, encompassing challenges such as occlusions, pedestrian crossings, yielding interactions, and wet road conditions. The ego vehicle dynamically adapts its trajectory in response to evolving scene context—slowing near crosswalks, yielding to oncoming or turning vehicles, and proceeding only after potential conflicts have cleared.

This alignment underscores how scenario-grounded rollouts enable safe, context-aware planning. p. 7
Where the paper says it

These behaviors align with the reasoning articulated in the generated ego-action responses, demonstrating a correspondence between textual intent and physical execution. This alignment underscores how scenario-grounded rollouts enable safe, context-aware planning.

The model consistently improves both closed-loop and open-loop performance across baselines like Qwen2. p. 0
Where the paper says it

(Top) Closed-loop metrics (Driving Score and Success Rate) and open-loop metric (Trajectory L2 error) for different models. GeRo consistently improves both closed-loop and open-loop performance across baselines like Qwen2.

5VL [3] and ORION [10]. p. 0
Where the paper says it

GeRo consistently improves both closed-loop and open-loop performance across baselines like Qwen2. 5VL [3] and ORION [10].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Generative Scenario Rollouts (GeRo) unify planning and autoregressive scenario generation to enable temporally consistent and context-aware driving behavior. This allows for scenario rollouts that predict future motions of traffic agents and ego actions, supporting long-horizon reasoning.

  2. Pretraining the VLA model jointly on planning, multi-agent motion prediction with auxiliary bounding box cues, and visual question answering binds linguistic and behavioral representations. This stage is crucial for binding linguistic and behavioral representations, enabling text-aligned generation beyond paired annotations.

  3. Language-conditioned Scenario Rollout autoregressively predicts future latent tokens based on past tokens and scenario context to generate ego trajectories, agent motions, and language outputs. This enables the model to generate the subsequent tokens ztilde t+delta+1 resulting in a rollout from t to t + T.

  4. Temporal consistency is enforced via KL-Divergence aligning the latent distribution of rollout predictions with the pretrained latent distribution for future time steps using Ltc, which mitigates compounding errors in autoregressive generation.

  5. Reinforcement Learning with GRPO feedback optimizes scenario rollouts by computing surrogate rewards that capture safety-critical metrics like Collision Loss and Time-to-Collision (TTC). This ensures high-fidelity and interpretable planning behavior by penalizing collisions and encouraging longer TTC.

  6. Integrating language alignment loss LVLA as a reward further enhances performance, aligning policy optimization with semantic correctness by measuring semantic alignment between generated language outputs and reference descriptions.

Abstract

Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and underutilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.

Sources

Related papers