Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
cs.LG, cs.AI
Submitted: 2026-09-27
Updated: 2026-09-27
Code: https://github.com/ikostrikov/implicit_q_learning
License: http://creativecommons.org/licenses/by/4.0/
The gist: Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online
Terminology
Abstract
Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.
Sources
- Continual Backprop: Stochastic Gradient Descent with Persistent Randomness
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- An Empirical Study of Implicit Regularization in Deep Offline RL
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
- Plasticity Loss in Deep Reinforcement Learning: A Survey
- Offline Reinforcement Learning with Implicit Q-Learning
- Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
- Maintaining Plasticity in Continual Learning via Regenerative Regularization
- Continuous control with deep reinforcement learning
- Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
- Understanding and Preventing Capacity Loss in Reinforcement Learning
- Disentangling the Causes of Plasticity Loss in Neural Networks
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learning
- Online Pre-Training for Offline-to-Online Reinforcement Learning
- Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks