Preference-Conditioned Multi-Objective Reinforcement Learning for Tactical Decision Making in Heavy-Duty Vehicles

The gist The proposed approach learns a set of Pareto-optimal policies that capture the trade-offs among three conflicting objectives: safety, quantified in terms of collisions and successful completion; energy efficiency and time efficiency, quantified using energy cost and driver cost

Problem Formulation

The study addresses the challenge of tactical decision making for a heavy-duty truck in a stochastic highway environment with multiple conflicting objectives Tactical decisions include adaptive cruise control and lane changes, aiming to balance safety, time efficiency, and energy efficiency. The problem is formulated within a Multi-Objective Reinforcement Learning (MORL) framework modeled as a Multi-Objective Markov Decision Process (MOMDP). The reward function r: S × A × S → R d is vector-valued, with d components corresponding to distinct objectives. The agent’s experience consists of transitions (st, at, st+1, rt+1), where rt = (r(1)t,..., r(d)t) quantifies the instantaneous contributions to each objective. Optimality is defined in terms of Pareto dominance: a policy π′ dominates π if it performs at least as well in all objectives and strictly better in at least one. The set of non-dominated value vectors forms the Pareto frontier, representing all achievable trade-offs beyond which improvement in one objective necessarily degrades another. User preferences are incorporated via a scalarization function or utility function, u: R d → R, which maps the multi-objective value vector to a scalar utility according to user-defined preferences. Linear scalarization is adopted as u(Vπ; w) = w⊤Vπ = X d i=1 wiVπ i, where the weight vector w lies on the unit simplex. The set of policies that maximize the scalarized return for some w forms the Convex Hull (CH), and the minimal subset containing one optimal policy per weight vector is the Convex Coverage Set (CCS), providing a compact approximation of all linearly Pareto-optimal solutions.

Multi-Objective Reinforcement Learning Framework

The proposed framework builds on Generalized Policy Improvement with Linear Support (GPI-LS) and extends it to a policy-gradient RL setting using Proximal Policy Optimization (PPO). A multi-objective PPO (MOPPO) architecture with preference-conditioned actor and critic networks is proposed, which learn per-objective value functions and action logits. Scalarization is applied only at the loss level, preserving the structure of individual objectives during training. The MOPPO framework involves several components including weight-conditioned feature extraction, a multi-objective actor that outputs action logits Z(a s) ∈ R A×d, and a multi-objective critic that outputs a vector-valued estimate V(s) ∈ R d. The rollout buffer stores transitions (st, at, log π(atst, wt), rt+1, donet+1, V(st), wt) for subsequent training.

GPI-based Weight Selection and Optimization

The procedure presented in Algorithm 1 extends the GPI-LS framework to a policy-gradient RL setting by iteratively constructing a set of policies Π = π(as, w) conditioned on preference (weight) vectors. At each iteration, the algorithm selects a weight vector w and learns a policy πw that optimizes the corresponding scalarized objective. Weight selection at each iteration is guided by the concept of corner weights Wcorner, derived from the vertices of a polyhedron P defined by constraints. The utility loss for a weight vector w ∈ W, denoted as ∆(w, Π) = v∗ w − maxπ∈Π vπ w, is estimated using Generalized Policy Improvement (GPI). The optimal policy estimate is computed by maximizing scalarized action probabilities across the learned policies as given by πˆopt(s; w) ∈ arg max a∈A max π∈Π π(a s, w). The policy is then updated using this w to maximize the scalarized return.

Safety and Control Architecture

The tactical decision-making architecture integrates MORL with model-based low-level controllers, ensuring dynamically feasible policies. The longitudinal controller uses the Intelligent Driver Model (IDM) [34] to compute acceleration/deceleration based on desired speed and time gap. The lateral controller uses the LC2013 model for lane changes. A safety filter for lane changes is implemented based on gap constraints, permitting a maneuver only if all of the following conditions hold:

1. Front gap in current lane: sfront,cur − (vego − vfront,cur)texit ≥ smin(vego, vego − vfront,cur) (smin defined in IDM [34]).

2. Front gap in target lane: sfront,tar − (vego − vfront,tar)tenter ≥ smin(vego, vego − vfront,tar), and sfront,tar − (vego - vfront,tar) Tlc ≥ smin(vego, vego - vfront,tar).

3. Rear gap in target lane: srear,target − (vrear,tar - vego)tenter ≥ smin(vrear,tar, vrear − vego).

Experimental Results and Comparison

The experiments demonstrate that the proposed method efficiently approximates the convex coverage set of the Pareto frontier and enables preference-aware policy selection. The results show a clear and interpretable Pareto structure, illustrating the trade-offs between energy cost and driver cost across zero, medium, and high traffic settings. In the zero-traffic scenario, policies with low cruising speeds cluster in the cost space, effectively preserving only those with minimal energy consumption and eliminating trade-offs in the intermediate region of the Pareto front. When traffic is present, interactions introduce variability in both travel time and energy usage, leading to multiple non-dominated policies spanning a broader region of the Pareto front. The comparison with the baseline method indicates that GPI-LS MOPPO identifies a larger set of feasible policies and covers a broader range of trade-offs between driver cost and energy cost. The ablation study highlights the importance of the safety filter, as removing it reduces the number of feasible policies on the Pareto frontier. The framework is computationally more efficient than GPI-LS based on DQN in MORL Baselines.

Conclusion and Impact

The developed multi-objective reinforcement learning framework effectively approximates the Pareto frontier and captures different trade-offs between success rate, driver cost and energy cost. These results demonstrate the effectiveness of the MORL method for heavy-duty vehicle applications. The framework supports the development of adaptive, preference-aware autonomous driving policies that can reflect different operational preferences depending on stakeholder needs. This work has the potential to improve the safety, efficiency, and cost-effectiveness of decision-making for heavy-duty autonomous vehicles. By leveraging multi-objective learning, the proposed approach enables adaptive decisionmaking policies that can reflect different operational preferences depending on stakeholder needs. Careful validation, safety constraints, and conservative deployment strategies are necessary before real-world use.

Traffic Environment Modeling

The custom RL environment is developed based on Simulation of Urban Mobility (SUMO) [19].

1. The system can learn robust, preference-aware driving policies by explicitly modeling trade-offs between safety, time efficiency, and energy efficiency using a Multi-Objective Reinforcement Learning (MORL) framework. This allows the agent to capture the fundamental conflict between energy cost and driver cost across various traffic densities, resulting in a clear and interpretable Pareto structure.

2. The proposed MOPPO architecture with preference-conditioned actor and critic networks that learn per-objective value functions and action logits enables dynamic adaptation to changing operational priorities. This capability allows the system to select policies based on user preferences without retraining, as it can adapt well to varying operational priorities without retraining, for instance across countries with different energy or driver costs.

3. The hierarchical decision-making architecture ensures dynamically feasible control by separating high-level tactical decisions from low-level physical execution. This structure allows the agent to balance safety, efficiency, and operational costs while maintaining safety constraints through a rule-based safety filter based on gap constraints.