Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems
cs.LG, cs.AI, cs.SY, eess.SY, math.OC
Submitted: 2024-07-24
Updated: 2025-07-24
Comments: 42 pages, 4 figures. Accepted for publication in SIAM Journal on Control and Optimization (2025)
Journal ref: SIAM J. Control Optim. 63 (5), 3452-3474 (2025)
DOI: 10.1137/24M1695075
Code: https://github.com/yiliehuang/sublinear-regret-LQ-RL
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study reinforcement learning (RL) for a class of continuous-time linear-quadratic (LQ) control problems for diffusions…
Terminology
Abstract
We study reinforcement learning (RL) for a class of continuous-time linear-quadratic (LQ) control problems for diffusions, where states are scalar-valued and running control rewards are absent but volatilities of the state processes depend on both state and control variables. We apply a model-free approach that relies neither on knowledge of model parameters nor on their estimations, and devise an RL algorithm to learn the optimal policy parameter directly. Our main contributions include the introduction of an exploration schedule and a regret analysis of the proposed algorithm. We provide the convergence rate of the policy parameter to the optimal one, and prove that the algorithm achieves a regret bound of O(N 3 over 4) up to a logarithmic factor, where N is the number of learning episodes. We conduct a simulation study to validate the theoretical results and demonstrate the effectiveness and reliability of the proposed algorithm. We also perform numerical comparisons between our method and those of the recent model-based stochastic LQ RL studies adapted to the state- and control-dependent volatility setting, demonstrating a better performance of the former in terms of regret bounds.
Sources
- Actor-Critic learning for mean-field control in continuous time
- Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study
- Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
- Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning
- Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models
- Continuous-time q-learning for mean-field control problems
- Regret Bounds for Episodic Risk-Sensitive Linear Quadratic Regulator
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis