TADPO: Reinforcement Learning Goes Off-road

The gist The authors introduce TADPO, a novel policy gradient formulation that extends Proximal Policy Optimization (PPO), to enable concurrent learning from fixed demonstrations and on-policy interactions to tackle long-horizon planning and hard exploration problems in off-road autonomous driving.

TADPO Formulation

TADPO is presented as a novel extension of Proximal Policy Optimization (PPO) that enables concurrent learning from fixed demonstrations and on-policy interactions to tackle longhorizon planning and hard exploration problems in off-road driving Given a pre-trained teacher policy µ, the loss function L TAD is defined to train a student policy πθ by incorporating teacher rollouts The loss is computed solely on teacher rollouts, where actions at each time step t are sampled from the teacher policy, i.e., at ∼ µ The ratio ρt(θ) is defined as the ratio of probability of at under πθ to the probability under µ The advantage estimator ∆ˆt estimates the advantage of the achieved return of the teacher rollout at state st over the expected return of the student This loss ensures that policy gradient propagates only when two conditions are met: (i) the teacher rollout’s return exceeds the student policy’s expected return, and (ii) the student policy’s probability of executing action at is not substantially higher than that of the teacher policy L entropy modulates the student policy’s exploration.

Training Procedure

TADPO involves learning a student policy πθ through concurrent utilization of teacher demonstrations and student rollouts Transitions are collected from both the pre-trained teacher policy µ and the student policy πθ being trained, and stored in separate teacher and student buffers, respectively With probability p, the algorithm samples transitions from the teacher’s buffer and performs a TADPO update Otherwise, it samples transitions from the student’s buffer and performs a standard PPO update. This alternating process continues for multiple iterations and epochs, allowing the student to learn from both its own experiences and the teacher’s expertise.

End-to-End System Architecture

The system employs a hierarchical architecture to achieve end-to-end off-road autonomy A global planner generates sparse waypoints utilizing a coarse global map, which are tracked by an RL controller trained using TADPO An MPPI controller interpolates sparse waypoints using the cost function from [13] to generate dense waypoints for training the teacher policy The student policy πθ is then trained to distill the teacher behavior via the TADPO training procedure while operating solely with sparse waypoints provided by the global planner. Observations combine proprioceptive and visual inputs, including normalized speed, roll, pitch, and waypoint encodings for teachers using dense maps

Simulation and Real-World Evaluation

The algorithms are trained in BeamNG.tech [43], a simulator that offers a highly realistic environment The training involves collecting 15 teacher demos for obstacle-rich and extreme slope terrains, and 20 for hybrid terrains In real-world deployment, the policy is used on a Sabercat vehicle to evaluate zero-shot sim-to-real performance The evaluation metrics include Success Rate (sr), Completion Percentage (cp), and Mean Speed (ms). The results show that TADPO outperforms RL and IL baselines, with the TADPO policy achieving a high percentage of obstacle avoidance and minimum cross-track error for long-distance high speed control.

Key Findings

The work demonstrates strong performance in simulation and, importantly, zero-shot sim-to-real transfer on a full-scale off-road vehicle TADPO outperforms RL and IL baselines Ablation studies show that ϵµ = 0.5 and constant p = 0.5 yield the best performance in baseline comparisons. This work represents, to our knowledge, the first deployment of end-to-end RLbased policies on a full-scale off-road platform.

Conclusion

TADPO is introduced as an extension of PPO that enables simultaneous learning from expert demonstrations and on-policy environment interactions to tackle long-horizon planning and hard exploration challenges By training a TADPO-based policy, the authors develop an end-to-end off-road autonomy pipeline capable of real-time, long-range navigation in complex, obstacle-rich, and diverse terrains. Our experiments show that TADPO outperforms RL and IL baselines Our experiments demonstrate strong performance in simulation and, importantly, zero-shot sim-to-real transfer on a full-scale off-road vehicle. This work represents, to our knowledge, the first deployment of end-to-end RLbased policies on a full-scale off-road platform. The authors' future work includes extending this framework to more diverse terrains.

Acknowledgment

This work was supported in part by the U.S. Army Research Office and the U.S. Army Futures Command under Contract No. W519TC-23-C-0030 The authors would like to acknowledge the contributions of Prasanna Kannappan, Anoushka Alavilli, Sam Zieger, Jessica Kasemer, and Daniela Resasco The authors would like to thank BeamNG GmbH for the academic license for BeamNG.tech.

References

The paper cites numerous related works, including those focusing on off-road driving [1], [2], [3] and on-road driving research [4], [5], and various RL methods like PPO and SAC Specific techniques mentioned include Model Predictive Path Integral (MPPI) [9] and Long Range Navigator (LRN) [15] for long-horizon planning, as well as Self-supervised methods like SALON [19] and Riskaware IRL for costmaps. Foundational vision models such as DinoV2 [39], SAM [40], and SAM2 [41] are also leveraged to bridge the domain gap between simulation and real-world. The paper compares TADPO against baselines such as DAgger, PPO, SAC, and IQL. The performance comparison in Table II shows that TADPO’s success rate (sr) and completion percentage (cp) notably exceed those of other real-time baseline methods.

--- Page 1 ---

TADPO: Reinforcement Learning Goes Off-road Zhouchonghao Wu∗1, Raymond Song∗1, Vedant Mundheda∗1, Luis E.

1. No longer struggle with exploration in complex long-horizon tasks by utilizing TADPO, which extends PPO to incorporate demonstrations from a teacher policy µ and trains concurrently using both its own on-policy rollouts and demonstrations. This allows the student policy to learn from expert knowledge while exploring beyond it, addressing the issue that standard RL methods often fail to acquire robust policies without additional guidance.

2. Achieve zero-shot sim-to-real transfer on full-scale off-road vehicles by training a policy with TADPO and deploying it without fine-tuning on real world images or scenarios. This is achieved by training the controller to output forward velocity and yaw rate given waypoint information and forwardfacing image using a frozen vision backbone, bridging the domain gap.

3. Enable real-time, long-range navigation in complex terrains by employing a hierarchical architecture where an RL controller tracks sparse waypoints generated by a global planner, allowing it to incorporate long-horizon planning capabilities to effectively track these waypoints. This system can navigate extreme slopes and obstacle-rich terrains in simulation and deployment.

4. Improve performance over baselines like PPO and SAC by using the TADPO loss function, which ensures that the policy gradient update propagates only when the teacher rollout’s return exceeds the student policy’s expected return, and (ii) the student policy’s probability of executing action at is not substantially higher than that of the teacher policy. This mechanism results in stable updates and higher reward attainment by the student.