Fully Offline Reinforcement Learning
cs.LG, cs.AI
Submitted: 2025-05-28
Updated: 2026-10-30
Code: https://github.com/obertTLange/gymnax
Project page: https://rltheorybook.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of…
Terminology
Abstract
Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.
Sources
- A Tutorial on Meta-Reinforcement Learning
- Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning
- Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
- A Clean Slate for Offline Reinforcement Learning
- Adam: A Method for Stochastic Optimization
- Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
- Offline Reinforcement Learning with Implicit Q-Learning
- Batch Policy Learning under Constraints
- Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
- Reinforcement Learning: An Overview
- Hyperparameter Selection for Offline Reinforcement Learning
- Proximal Policy Optimization Algorithms
- A Strong Baseline for Batch Imitation Learning
- Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
- A Review of Off-Policy Evaluation in Reinforcement Learning
- What are the Statistical Limits of Offline RL with Linear Function Approximation?
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis