Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving

arXiv:2508.16947 · cs.RO, cs.AI · Submitted 2025-08-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Drive As You Like".

Dev: The gist The proposed M-Diffusion Planner is an RL-guided multi-strategy framework with a diffusion-based multi-head planner integrated with LLM-based semantic understanding,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: We're diving into the specifics of "Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving." This paper argues that existing imitation learning systems are too biased toward common driving patterns, missing the diversity of human driving behaviors, and they struggle to understand what a user actually wants from them.

Dev: They propose an M-Diffusion Planner as the solution. The thesis is that we need a framework that combines LLM-based semantic understanding with a diffusion-based multi-head planner to enable dynamic perception of user intent and generate trajectories aligned with those diverse needs.

Taro: So, in plain terms, the system starts by using an LLM to interpret what the user is asking for in their interaction. This interpretation then selects one specific driving strategy from a set of available strategies that are all controlled by different heads within a shared diffusion planner.

Rosa: That's right. The key feature they highlight is this dual decoupling: separating the intent understanding part from the trajectory planning part, and learning the general planning capability across those different strategies through strategy alignment.

Dev: They also introduce a lightweight GRPO-based strategy alignment method, which freezes the shared backbone of the planner and only optimizes those specific output layers with a relative preference reward. This allows for efficient adaptation without needing huge amounts of new data for every single driving style.

Taro: It sounds like they are trying to solve the problem where you can't just have one perfect policy that covers every possible driving scenario because human needs change constantly.

Rosa: Precisely. They show that after pre-training with imitation learning, these strategy-specific rewards specialize each head to match user preferences, which is what allows for those diverse driving styles without requiring explicit style annotations in the initial data.

Dev: The architecture itself is quite complex on the input side. They have an Encoder using an MLP-Mixer followed by a Transformer layer that encodes mixed scenario inputs into compact embeddings.

Taro: And then the Decoder is built on a DiT, which allows for trajectory generation conditioned not just on the general context but also directly on those high-level planning strategies selected by the LLM.

Rosa: The multi-level fusion part is also important because they concatenate the future trajectory with current vehicle states to give it a clear starting point, and they fuse historical features using an MLP-Mixer that mixes tokens along both temporal and spatial dimensions.

Dev: They also encode static objects like road boundaries and signs using an MLP and Transformer block incorporated through multi-head cross-attention, as well as injecting global semantic cues into the diffusion process via a temporal condition.

Taro: So, it’s a lot of conditioning going into the process—contextual encodings, historical features, object encodings, and global navigation cues—all feeding into that final output layer selection.

Rosa: That's the core of M-Diffusion Planner: taking rich scenario inputs and high-level strategy instructions to generate highly personalized motion plans. This framework aims to bridge the gap between simple behavior regression and truly context-aware, user-dependent driving requirements.

Conclusion: Dev: So looking at "Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving," the authors Ding Fan et al. are proposing this M-Diffusion Planner as a novel way to handle personalized driving needs through RL and LLMs.

Rosa: The title itself, "Drive As You Like," suggests a system that is finally capable of adapting its behavior to what the user likes or wants in a driving situation, not just rigidly following pre-programmed rules.

Taro: What this means practically is that instead of being stuck reproducing high-frequency behaviors from old datasets, an autonomous vehicle could potentially switch between different driving modes on demand based on the driver's current needs.

Dev: They are showing that you can use a diffusion model structure to generate diverse, high-quality trajectories and then use RL to fine-tune specific heads for each strategy. This allows for real-time strategy switching without needing a full model retraining or reloading every time the user changes their mind.

Rosa: The conclusion is that this framework offers both adaptability and efficiency for interactive driving applications, allowing the planner to support trajectory generation conditioned on multi-modal user instructions.

Taro: It suggests a future where autonomous vehicles are not just predictable machines but truly responsive partners that can adjust their style based on the user's specific context.

Dev: The paper demonstrates that this approach delivers competitive or superior planning performance when evaluated on benchmarks like nuPlan, showing distinct strategy-aligned behaviors.

Rosa: So, in short, M-Diffusion Planner is a framework that leverages diffusion models and reinforcement learning to create a system that can generate varied, high-quality driving plans tailored to individual user requirements dynamically.

Monash University of Malaysia · University of Utah, USA

cs.RO, cs.AI

Submitted: 2025-08-23

Updated: 2026-10-15

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist The proposed M-Diffusion Planner is an RL-guided multi-strategy framework with a diffusion-based multi-head planner integrated with LLM-based semantic understanding, enabling dynamic

Key concepts

M-Diffusion Planner: This is the core architecture that combines a diffusion model with multiple planning heads. It separates intent understanding (handled by an LLM) from trajectory generation. When a user requests a driving style, the LLM picks the right strategy, and this strategy activates one of several specialized planner heads within the diffusion model to create the desired path.
Dual-Decoupling Design: The framework uses two main separation techniques: first, separating intent understanding from trajectory planning. Second, it learns general planning capabilities by aligning different strategies. This design allows the system to flexibly switch between different driving behaviors while ensuring that each specialized head remains effective and robust.
GRPO (Group Relative Policy Optimization): This is the reinforcement learning method used to align individual planner heads with user preferences. It freezes the main model backbone and only updates the strategy-specific output layers using a relative preference reward. This makes adaptation efficient, allowing different driving styles to be learned quickly without needing extensive new training data.

Terminology

Adaptation — 1× in this paper · explained in 1 episode
Alignment — 1× in this paper
Benchmark — 1× in this paper · explained in 2 episodes
GRPO — 1× in this paper · explained in 6 episodes
Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 6 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes
Two-stage training — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed M-Diffusion Planner is an RL-guided multi-strategy framework with a diffusion-based multi-head planner integrated with LLM-based semantic understanding, enabling dynamic perception of user intent and generation of diverse, preference-aligned trajectories.

Problem Motivation

Existing imitation learning systems are restricted to reproducing high-frequency biased behaviors, overlooking the inherent behavioral diversity of human driving <ref:2508.16947#pg5>. These current approaches lack explicit modeling of user intent and flexible policy adaptation <ref:2508.16947#pg3>. Specifically, most existing methods rely on imitation learning–based approaches which formulate motion planning as a supervised learning problem over large-scale expert trajectories <ref:2508.16947#pg4>. This formulation introduces a strong bias toward high-frequency behaviors in the dataset, causing the model to overlook low-frequency but task-critical strategies <ref:2508.16947#pg3>. Furthermore, existing interaction methods treat language as a soft constraint rather than a principled basis for planning decisions, leading to limited controllability <ref:2508.16947#pg4>.

M-Diffusion Planner Architecture

The framework consists of two key components: a strategy-aligned diffusion planner with multiple output layers and an LLM-based user understanding module empowered by refined prompt engineering <ref:2508.16947#pg4>. At the beginning of each planning episode, the LLM interprets and reasons over the user’s interactive input to identify the most appropriate driving strategy <ref:2508.16947#pg4>. The selected strategy then activates the corresponding output head of the diffusion planner, thereby initiating the motion planning process <ref:2508.16947#pg4>.

The model architecture includes several modules for multi-level fusion and conditioning:

Encoder: Includes an MLP-Mixer followed by a Transformer layer which encodes heterogeneous scenario inputs by alternately mixing features along token and channel dimensions, generating compact and fixed-length embeddings <ref:2508.16947#pg4>.
Decoder: Built upon a DiT with strategy-aware multi-output layer outputs, allowing trajectory generation conditioned on both contextual encodings and high-level planning strategies <ref:2508.16947#pg4>.
Multi-level Fusion: The model employs a multi-level fusion mechanism where the future trajectory is concatenated with current vehicle states to provide clear initialization <ref:2508.16947#pg4>. Historical neighboring and lane features are fused using an MLP-Mixer, which alternately mixes tokens along the temporal and spatial dimensions to extract compact feature representations <ref:2508.16947#pg4>.
Static Object Encoding: Road boundaries and traffic signs are encoded by an MLP and a Transformer block, incorporated through multi-head cross-attention (MHCA) <ref:2508.16947#pg4>.
Navigation Cues: The global semantic representation Qn, extracted via an MLP-Mixer, is injected into the diffusion process through the temporal condition Qt <ref:2508.16947#pg4>.

Training and Alignment Paradigm

The training employs a two-stage paradigm to balance trajectory quality and strategy alignment <ref:2508.16947#pg4>.

Pre-training (Imitation Learning): Imitation learning ensures each policy head achieves safe and high-quality planning <ref:2508.16947#pg4>. A weight-sharing initialization scheme is used, where a single-output diffusion planner is trained first, and its weights are transferred to initialize the multi-head M-Diffusion Planner <ref:2508.16947#pg4>.
Post-training (RL Alignment): Constrained Group Relative Policy Optimization (GRPO) further aligns each head with user preferences <ref:2508.16947#pg4>. This method freezes the shared backbone and optimizes only strategy-specific output layers with a relative preference reward, enabling efficient adaptation with minimal post-training data <ref:2508.16947#pg4>. The GRPO objective is formulated as JGRPO(θ) = Eq∼p(Q), τˆ1:G∼πθold (Oq) LGRPO i = min ρiAi, clip(ρi, 1 − ε, 1 + ε)Ai - β DKL(πθ∥ πref).

Reward Design and Strategy Learning

The reward function is composed of multiple terms reflecting key behavioral factors, including average velocity, average acceleration, average jerk, and average following distance <ref:2508.16947#pg4>. The reward is then standardized into trajectory-level advantages Ai = ri − µr σr + ϵ <ref:2508.16947#pg4>. A key ablation study compared explicit and implicit rewards, revealing a trade-off between strategy specialization and overall robustness <ref:2508.16947#pg4>. Explicit rewards facilitate clear behavioral differentiation but may introduce biases, whereas implicit rewards promote balanced optimization across multiple objectives <ref:2508.16947#pg4>.

Experimental Validation

Experiments were conducted on the nuPlan benchmark under both open-loop and closed-loop settings <ref:2508.16947#pg4>.

Open-loop Evaluation: This assesses distinctiveness among strategies, focusing on velocity, acceleration, and jerk <ref:2508.16947#pg4>. Results showed that the aggressive driving strategy exhibits higher speeds and increased acceleration compared to other strategies <ref:2508.16947#pg4>.
Closed-loop Evaluation: This evaluates robustness in interactive traffic environments using the val14 test set <ref:2508.16947#pg4>. The M-Diffusion Planner demonstrated strong performance, with the Conservative strategy producing trajectories with lower jerk, consistent with enhancing ride smoothness and stability <ref:2508.16947#pg4>. A case study confirmed that the LLM correctly interpreted a high-level instruction to select the aggressive strategy for a commuting scenario <ref:2508.16947#pg4>.

Conclusion

The proposed framework introduces M-Diffusion Planner, a novel interactive planning framework that leverages the diffusion model’s capacity to learn complex driving behaviors and generate diverse, high-quality trajectories <ref:2508.16947#pg4>. By incorporating GRPO to fine-tune a set of strategy modes, the planner adapts to various driving styles without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The framework supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. Open-loop and closed-loop experiments demonstrate that M-Diffusion Planner not only generates clearly differentiated strategy-specific trajectories, but also delivers competitive or superior planning performance compared to existing methods on the nuPlan Val14 benchmark <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:2508.16947#pg4>. The planner supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:2508.16947#pg4>. The planner supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:2508.16947#pg4>. The planner supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:2508.16947#pg4>. The planner supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:2508.16947#pg4>. The planner supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:2508.16947#pg4>. The planner supports real-time strategy switching without requiring model retraining or reloading, offering both adaptability and efficiency for interactive autonomous driving applications <ref:2508.16947#pg4>. The planner supports trajectory generation conditioned on multi-modal user instructions, enabling behaviors aligned with user-specified driving modes <ref:

Improvements for AI systems

The system can generate diverse, preference-aligned trajectories by using a strategy-aligned diffusion planner with multiple output layers conditioned on a strategy inferred from LLM reasoning, enabling trajectory generation aligned with diverse user requirements.
The AI can adapt its driving behavior in real-time through a mechanism that allows for real-time strategy switching without requiring model retraining or reloading, supporting interactive autonomous driving applications efficiently.
The system will exhibit distinct behavioral characteristics based on the chosen strategy; for example, the aggressive driving strategy produces higher speeds, accompanied by increased acceleration, while the comfortoriented strategy produces trajectories with the lowest jerk.
The planner can be fine-tuned via a reinforcement learning process using constrained Group Relative Policy Optimization (GRPO), which is described as a method that optimizes only strategy-specific output layers with a relative preference reward, enabling efficient adaptation with minimal post-training data.