ReBRAC-v2: The Return of the King
cs.LG, cs.AI
Submitted: 2026-08-02
Updated: 2026-10-30
Comments: https://github.com/DT6A/ReBRAC-v2
Code: https://github.com/DT6A/ReBRAC-v2
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value guidance.
Terminology
Abstract
Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value guidance. We ask how far systematic modernization of a conventional behavior-regularized actor-critic can go while preserving algorithmic simplicity. ReBRAC-v2 directly trains an exact-likelihood normalizing-flow actor with likelihood, MSE, and MAE behavior regularization, classification-based residual critics, staged optimization, and multi-sample inference. We develop one shared configuration through roughly 600 Bayesian proposals on six OGBench tasks, then adapt only two behavior-regularization coefficients using a 16-point grid for the main comparisons. Across ten common OGBench categories, ReBRAC-v2 averages 74.8 versus 52.3 for the next-best method and ranks first in eight categories. Without structural changes, it also obtains the strongest compared averages on D4RL AntMaze (90.2) and Adroit (33.6). Motivated by these results, we extend evaluation to all 235 state-based single-task problems across the 47 dataset variants in the original OGBench suite. To our knowledge, this is the first complete evaluation in this setting; no compared baseline reports results across the full suite. Per-task scores and configurations establish ReBRAC-v2 as a reference baseline for future work beyond the common task subset. The complete cloning objective retains the highest average after coefficient retuning of its alternatives, while further ablations characterize training, capacity, and inference choices. These results demonstrate the strength of systematic engineering within a minimalist offline RL foundation.
Sources
- Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows
- Layer Normalization
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
- Flow Actor-Critic for Offline Reinforcement Learning
- Density estimation using Real NVP
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Dropout Q-Functions for Doubly Efficient Reinforcement Learning
- Drift Q-Learning
- Offline Reinforcement Learning with Implicit Q-Learning
- Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
- Learning to Optimize for Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Decoupled Q-Chunking
- Decoupled Weight Decay Regularization
- Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?
- NinA: Normalizing Flows in Action. Training VLA Models with Normalizing Flows
- The Role of Deep Learning Regularizations on Actors in Offline RL
- Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
- Mining Generalizable Activation Functions
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis