Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
cs.LG, cs.AI, cs.SY, eess.SY, math.OC
Submitted: 2025-07-01
Updated: 2026-11-16
Comments: 37 pages, 10 figures
Journal ref: IEEE Transactions on Automatic Control (2026)
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study reinforcement learning (RL) for the same class of continuous-time stochastic linear--quadratic (LQ) control problems as in…
Terminology
Abstract
We study reinforcement learning (RL) for the same class of continuous-time stochastic linear--quadratic (LQ) control problems as in, where volatilities depend on both states and controls while states are scalar-valued and running control rewards are absent. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in, which require extensive tuning for implementations and ignore learning progresses during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts.
Sources
- Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems
- A survey on intrinsic motivation in reinforcement learning
- Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
- Large-Scale Study of Curiosity-Driven Learning
- Exploration by Random Network Distillation
- Go-Explore: a New Approach for Hard-Exploration Problems
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models
- Regret of exploratory policy improvement and q-learning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis