Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting
stat.ML, cs.LG
Submitted: 2025-12-29
Updated: 2026-08-30
License: http://creativecommons.org/licenses/by/4.0/
The gist: Fitted Q-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift, value-function realizability alone does not ensure convergence, and existing
Terminology
Abstract
Fitted Q-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift, value-function realizability alone does not ensure convergence, and existing analyses often require Bellman completeness. We trace this instability to a geometric mismatch: standard FQE projects Bellman targets in the norm induced by the offline distribution, which need not preserve Bellman contraction. We therefore study occupancy-weighted FQE, which changes only the regression weights. Weighting by a target-policy discounted occupancy ratio aligns the projection norm with the target-policy dynamics and restores contraction of the population projected Bellman operator. We derive finite-sample guarantees with estimated occupancy ratios and function-class misspecification, separating finite-iteration, statistical, approximation, and ratio-estimation errors. Exact occupancy weighting removes the need for Bellman completeness; with estimated weights, approximate completeness and value-function realizability reduce sensitivity to ratio-estimation error, with exact realizability yielding higher-order dependence. Combining occupancy-weighted FQE with fitted occupancy-ratio evaluation gives an end-to-end guarantee governed by the complexities and direct approximation errors of the value-function and occupancy-ratio classes. Under coverage, joint realizability of these two classes suffices for consistent estimation without Bellman or critic-side completeness. Controlled experiments illustrate the projection-norm mechanism and the finite-sample tradeoff between contraction and coverage.
Sources
- Harnessing Density Ratios for Online Reinforcement Learning
- A Unifying View of Coverage in Linear Off-Policy Evaluation
- Sequential causal inference in a single world of connected units
- Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning
- Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
- Learning density ratios in causal inference using Bregman-Riesz regression
- COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
- Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration
- Emphatic Temporal-Difference Learning
- Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Source-Condition Analysis of Kernel Adversarial Estimators
- Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
- A Researcher's Guide to Empirical Risk Minimization
- Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inference
- Nonparametric Instrumental Variable Inference with Many Weak Instruments
- Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
- The Role of Coverage in Online Reinforcement Learning
- GenDICE: Generalized Offline Estimation of Stationary Values
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches