Variational Consequence-Driven Offline Reinforcement Learning
cs.LG
Submitted: 2026-10-28
Updated: 2026-11-06
Comments: accepted by NeurIPS 2026
Code: https://github.com/DarrenJum/StBR
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Offline reinforcement learning (RL) typically mitigates distribution shift by imposing divergence constraints on the action distributions of the learned and behavior policies.
Terminology
Abstract
Offline reinforcement learning (RL) typically mitigates distribution shift by imposing divergence constraints on the action distributions of the learned and behavior policies. However, this paradigm cannot ensure the underlying dynamic consistency structure: even minor action deviations can lead to drastically different transitions, inadvertently driving the agent away from the offline dataset. To address this, we explore policy constraints through their induced state-consequence distributions rather than relying on pure action-distribution matching. Nevertheless, directly enforcing such consequence-aware constraints is computationally intractable without access to the true underlying dynamics model. To bypass this issue, we propose Structural Behavior Regularization (StBR), which replaces the intractable objective with a closed-form latent divergence. Structuring this latent space to reflect transition dynamics renders the latent divergence a theoretically bounded surrogate for the true consequence-distribution divergence. By regularizing the learned policy with this latent divergence, StBR adaptively tightens the behavior constraint in dynamics-sensitive regions, effectively preventing deviations outside the offline dataset. Empirically, we demonstrate that StBR achieves superior performance across diverse offline RL benchmarks.
Sources
- Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Auto-Encoding Variational Bayes
- Offline Reinforcement Learning with Implicit Q-Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Flow Q-Learning
- Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis