BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
cs.LG, cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
License: http://creativecommons.org/licenses/by/4.0/
The gist: Expected-cost constraints can still permit rare, high-cost events.
Terminology
Abstract
Expected-cost constraints can still permit rare, high-cost events. Monte Carlo conditional value at risk (CVaR) gradients can be noisy at high confidence, whereas critics that model an outcome distribution add complexity. We propose BCPPO (Bachelier-Inspired Constrained Proximal Policy Optimization), a proximal policy optimization (PPO) method. Separately initialized cost-prediction networks (critics), trained with random sample masks, produce disagreement that marks predictions sensitive to which state-action regions occur in the training data and to critic training. A Bachelier formula for the expected amount above a reference level converts this disagreement into a smooth policy-update penalty. Gradients from this penalty do not alter the critics, so temporal-difference (TD) critic learning is unchanged. A saturation-aware controller adjusts the mean-cost penalty and stops accumulated error from growing while that penalty is clipped. Deployment retains only the policy network. The disagreement penalty is neither a tail-event probability nor a guaranteed error bound, and it provides no safety guarantee. Across 175 runs with shared tasks, costs, budgets, training steps, and evaluation seeds, no comparator attains both higher mean return and lower mean CVaR than BCPPO in any task. On Push1, BCPPO has no lower return and no higher CVaR than every comparator, with at least one strict gain. These results support a practical balance among reward, caution around cost predictions that vary across trained critics, and policy-only deployment.
Sources
- Constrained Policy Optimization
- Constrained Policy Optimization via Bayesian World Models
- A Distributional Perspective on Reinforcement Learning
- SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancy
- Risk-Constrained Reinforcement Learning with Percentile Risk Criteria
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Implicit Quantile Networks for Distributional Reinforcement Learning
- Distributional Reinforcement Learning with Quantile Regression
- Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
- Safe Langevin Soft Actor Critic
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Constrained Variational Policy Optimization for Safe Reinforcement Learning
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Proximal Policy Optimization Algorithms
- Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation
- RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes
- Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
- Policy Gradient for Coherent Risk Measures
- Optimizing the CVaR via Sampling
- Projection-Based Constrained Policy Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks