Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

The paper addresses safe reinforcement learning, which maximizes reward subject to safety constraints. For Constrained Markov Decision Processes (CMDPs), the authors note a key structural property: whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient-based methods do not exploit this structure and often settle in the feasible interior.

The authors identify limitations in existing approaches:

• Primal–dual methods enforce the constraint through a learned multiplier, so the policy's location relative to an active boundary is controlled only indirectly through the dual dynamics.

• Primal and projection-based methods restrict or correct updates by feasibility — a one-sided condition that does not say how much of the remaining cost budget a feasible policy should use.

• Gradient-manipulation methods coordinate the two objectives through gradient geometry, which captures local alignment but not where the policy sits relative to the cost limit.

The key insight is that standard updates do not separate reward improvement along a cost level set from signed regulation across cost levels toward an active boundary.

The authors introduce BSPG, "a first-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual-driven normal component that regulates the policy toward the active boundary from either side."

The method decomposes the update into two orthogonal components:

1. Tangential reward gradient: gr⊥ = gr − (⟨gr, gc⟩/∥gc∥2) gc, which removes from the reward gradient its component along the cost gradient, preserving cost to first order while retaining reward ascent.

2. Boundary-attraction term: gb = −δ̂ · gc/(∥gc∥ + ε), where δ̂ is the estimated constraint residual and ε is a numerical stabilizer. This term "provides signed attraction toward the boundary: when δ̂ > 0, the iterate is infeasible and gb acts to decrease Jc; when δ̂ < 0, the iterate lies in the feasible interior and gb acts to increase Jc."

The combined update is gup = gr⊥ + η·gb, where η > 0 is the boundary-attraction coefficient.

The update admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable. The implicit multiplier is:

λt = ⟨gr, gc⟩/∥gc∥2 + ηδ̂/(∥gc∥ + ε)

This means each BSPG step is equivalent to a gradient-ascent step on the Lagrangian L(θ, λ) = Jr(πθ) − λJc(πθ), with the multiplier λt induced automatically by the local gradient geometry and the boundary signal.

The paper proves that when no unconstrained reward maximizer is feasible, every constrained occupancy-measure maximizer uses the full cost budget. Specifically: every constrained-optimal policy π⋆ ∈ arg maxπ∈Π: Jc(π)≤d Jr(π) satisfies Jc(π⋆) = d.

The proof relies on the convexity of the occupancy-measure polytope and the linearity of Jr, Jc in the occupancy measure.

The paper shows that under positive gradient alignment, a fraction β of the unused cost budget can be converted into a first-order reward gain of βδ⟨gr, gc⟩/∥gc∥2 while preserving strict feasibility. This demonstrates that not every interior policy incurs a reward loss.

Under Assumptions 1–3 (smoothness and bounded gradients, constraint regularity with ∥gc∥ ≥ µ, and diminishing step sizes):

Theorem 6.1 (Boundary Convergence): The constraint residual δt satisfies:

V(θt+1) ≤ V(θt) − µ′αtδt2 + CRαt2

where V(θ) = ½δ(θ)2, µ′ = ηµ2/(G+ε), and CR = (G2+LD)(G2+η2D2). Consequently, Jc(πθt) → d from either side of the boundary.

Corollary 6.2 (Finite-horizon residual bound): The expected squared residual satisfies:

E[δτ2] ≤ (V0 + CR·Σαt2)/(µ′AT)

For constant αt = ᾱ/√T over a fixed horizon T, the bound is O(1/√T).

Theorem 6.3 (Reward Ascent): gr, gr⊥ = ∥gr⊥∥2 ≥ 0. Equality holds iff gr ∥ gc. On the manifold M, the BSPG update is a strict ascent direction for Jr unless gr ∥ gc.

Theorem 6.4 (Convergent-limit characterization): If exact-gradient BSPG iterates converge to θ∗, then θ∗ ∈ M and gr(θ∗) = λ∗gc(θ∗) with λ∗ = ⟨gr(θ∗), gc(θ∗)⟩/∥gc(θ∗)∥2. If, in addition, θ∗ is a local maximizer of Jr on F, then λ∗ ≥ 0 and θ∗ satisfies the KKT conditions of (3).

The authors note that "the Lyapunov function V = ½δ2 is symmetric in the sign of δ, so the boundary-seeking mechanism operates identically whether the current iterate is feasible or infeasible. This contrasts with Lagrangian methods, where the multiplier update acts only on positive violations."

The method is evaluated on SafetyPointGoal1-v0 using the OmniSafe framework against baselines CRPO and ESPO. The paper reports three metrics: average episodic reward, average episodic cost, and boundary proximity (the empirical residual magnitude Ĉ − d).

Results show: "BSPG attains the highest episodic reward on SafetyPointGoal1-v0. At the same time, BSPG maintains episodic cost near the constraint limit, while both baselines remain more conservative and settle well inside the feasible region. In the boundary-proximity plot, BSPG remains consistently closest to zero, indicating tighter tracking of the active constraint boundary."

The authors explicitly note: "Theorem 4.1 and Corollary 4.2 identify the global constrained target; Theorems 6.1–6.4 show that the ideal parameter update regulates this target cost level and characterize convergent runs. They do not establish global reward convergence for an arbitrary nonconvex policy class (which would require additional landscape structure or a separate tabular analysis), nor sample complexity or convergence guarantees for the stochastic normalized PPO implementation."

The residual estimator in Eq. (19) is described as an implementation heuristic since no general unbiasedness or variance-reduction claim follows without additional assumptions on the critic and the sampling process.

The paper concludes: By combining a tangential reward term with a signed boundary-attraction term, BSPG encourages boundary-aligned optimization rather than conservative interior solutions. The method attains a strong reward–safety trade-off in the evaluated task by making fuller use of the safety budget.

Based on this paper, here are the specific improvements I can make to AI systems:

• Improvement: Replace conservative safety margins with active boundary-seeking behavior. Instead of keeping the system safely away from constraints, the AI will now optimize reward while deliberately operating at the constraint boundary when beneficial.

• Capability: The AI can now maximize performance (e.g., throughput, speed, resource utilization) up to the exact safety limit, rather than leaving unused capacity. For example, in autonomous driving, it would drive at the maximum safe speed rather than conservatively below it.

• Improvement: Implement the signed residual-driven normal component that regulates from both sides of the constraint boundary—not just from infeasible to feasible, but also from overly-conservative feasible states toward the boundary.

• Capability: The AI can self-correct when it drifts too far into the safe zone (wasting potential reward) just as effectively as when it violates constraints. This prevents both unsafe behavior and underperformance.

• Improvement: Eliminate the need for a learned dual variable or manual penalty tuning. The multiplier is computed automatically from local gradient geometry and the constraint residual signal.

• Capability: The AI adapts its constraint satisfaction pressure in real-time without hyperparameter tuning for each new task. This makes the system more robust across different environments and constraint types without requiring per-task calibration.

• Improvement: Decompose the update into tangential (reward-improving while preserving cost) and normal (boundary-regulating) components that are mathematically orthogonal.

• Capability: The AI can simultaneously improve reward and adjust constraint satisfaction without one objective interfering with the other. This is particularly useful in robotics where the system must optimize energy efficiency while maintaining safety limits.

• Improvement: Apply the proven convergence guarantee that ensures the constraint residual shrinks to zero at a known rate, regardless of whether starting from feasible or infeasible states.

• Capability: The AI can provide predictable, verifiable safety convergence in safety-critical applications like medical treatment planning or financial trading, where we need formal assurance that constraints will be met within a specified time horizon.

• Improvement: Ensure that when the AI converges, it satisfies the Karush-Kuhn-Tucker optimality conditions—meaning it reaches a true constrained optimum, not just a feasible solution.

• Capability: The AI can guarantee that its final policy is not merely safe but is provably the best possible reward under the safety constraints, eliminating suboptimal conservative solutions.

• Improvement: Use the symmetric Lyapunov function V = ½δ2 to treat constraint violation and underutilization with equal importance.

• Capability: The AI will not exhibit asymmetric behavior—it will be equally aggressive in correcting unsafe behavior and in exploiting unused safety budget. This creates balanced, efficient operation across diverse scenarios.

• Improvement: Apply the first-order method that requires only gradient information (no Hessian or second-order methods), with proven finite-horizon residual bounds.

• Capability: The AI can operate efficiently in high-dimensional policy spaces (e.g., large language model fine-tuning, complex robotic control) with low computational overhead while maintaining formal safety guarantees.

What the improved AI system can do overall: It can learn policies that achieve maximum reward by operating precisely at safety boundaries, self-correcting from both constraint violations and excessive conservatism, with formal convergence guarantees and no need for manual safety-penalty tuning. This is applicable to autonomous systems, resource allocation, portfolio management, and any domain where performance must be maximized under hard constraints.

Safe reinforcement learning maximizes reward subject to safety constraints. For Constrained Markov Decision Processes, the linear-programming view over occupancy measures implies that whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient-based methods do not exploit this structure and often settle in the feasible interior. We introduce Boundary-Seeking Policy Gradient (BSPG), a first-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual-driven normal component that regulates the policy toward the active boundary from either side; the combined direction admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable. Under exact gradients and stated regularity conditions, the constraint residual converges to zero from either side with a finite-horizon O(1/sqrt T) bound, the tangential component is a reward-ascent direction on the boundary, and any convergent parameter sequence is stationary on the active constraint set, satisfying the KKT conditions when the limit is also a local maximizer over the feasible set. This complements existing analyses, which certify feasibility but do not characterize the constraint value at convergence. On a standard Safety-Gymnasium navigation task, BSPG attains higher reward while tracking the boundary more tightly than the compared baselines.