Fully Offline Reinforcement Learning

arXiv:2505.22442 · cs.LG, cs.AI · Submitted 2025-05-28 · Read on arXiv

cs.LG, cs.AI

Submitted: 2025-05-28

Updated: 2026-10-30

Code: https://github.com/obertTLange/gymnax

Project page: https://rltheorybook.github.io

License: http://creativecommons.org/licenses/by/4.0/

The gist: Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of…

Terminology

Abstract

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Sources

Related papers