Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
Zeyu Bian, Ying Zhou, Yifan Cui
stat.ML, cs.LG
Submitted: 2026-07-28
License: http://creativecommons.org/licenses/by/4.0/
The gist: Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error.
Terminology
Abstract
Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.
Sources
- A Tale of Two Cities: Pessimism and Opportunism in Offline Dynamic Pricing
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- A Review of Off-Policy Evaluation in Reinforcement Learning
- Causal Inference with a Hidden Treatment
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches