O(1/n) Convergence Rate for RVI Q-Learning
cs.LG
Submitted: 2026-11-05
Updated: 2026-11-05
Comments: 24 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: We establish finite-time convergence guarantee for synchronous relative value iteration (RVI) Q-learning for finite weakly communicating Markov decision processes (MDPs).
Terminology
Abstract
We establish finite-time convergence guarantee for synchronous relative value iteration (RVI) Q-learning for finite weakly communicating Markov decision processes (MDPs). For stochastic Krasnosel'ski ı--Mann iterations driven by a non-expansive, linearly regular map on a finite-dimensional normed space, we establish last-iterate mean-squared residual bounds under martingale-difference noise whose conditional second moment grows at most linearly with the squared residual. Suitable polynomial stepsizes of order n-a yield an O(n-a) rate for 0<a at most1. We verify linear regularity and the noise condition for the centered Bellman operator for finite weakly communicating MDPs. Applying the general result gives an O(n-1) bound on the mean-squared Bellman span residual of RVI Q-learning iterates when a suitable stepsize sequence is used. The positive gap between optimal and suboptimal stationary deterministic policies then yields an O(n-1) bound on the expected suboptimality of the greedy policy. Consequently, O(ε-1) synchronous iterations suffice for an ε-optimal policy in expectation. These guarantees are instance-dependent, with constants that depend on the hitting-time parameters and the suboptimality gap of the second-best policy.
Sources
- Achieving $\varepsilon^{-2}$ Dependence for Average-Reward Q-Learning with a New Contraction Principle
- Stochastic approximation with cone-contractive operators: Sharp $\ell_\infty$-bounds for $Q$-learning
- On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis