Expert-Guided Online Reinforcement Learning for Stable and Sample-Efficient Control of Dynamical Systems

arXiv:2511.11131 · eess.SY, cs.SY · Submitted 2025-11-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Expert-Guided Online Reinforcement Learning for Stable and Sample-Efficient Control of Dynamical Systems".

Dev: Here's the gist of the paper. Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure The authors studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings Their theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy They verify their theoretical finding with simulation results on a linearized inverted pendulum.

Rosa: Theoretical Framework for Policy Learning The paper formulates learning the one-step policy as a policy gradient problem by optimizing the average expected cost regularized with the behavior cloning loss The one-step policy is defined as a linear state-feedback controller with additive Gaussian noise, specifically µK(x) = Kx + z, where z N (0, Wz) The one-step policy loss function for the LQR problem is given by l lin φo(K) = E x∼D z∼N (0,Wz) h J(µK) + α squared (Kx + z − µφb(x, z))⊤(Kx + z − µφb(x, z)).

Dev: Convergence Analysis of the Loss Function The authors prove that the one-step policy loss l lin φo is gradient dominant by showing it is coercive and L-smooth.

Rosa: Specifically, Lemma 3 shows the gradient of the one-step policy loss l lin φo(K) is given by ∇Kl lin φo(K) =(Suu(θ)(K − K∗))Ex∼D

xx⊤: + α(K − K∗)Ex∼D

xx⊤: .

Paper summary: Rosa: So, to wrap up what we just touched on, this paper is focusing on how expert guidance helps us train control policies for linear quadratic problems when we only have offline data available Dev They introduce Flow Q-learning which uses normalizing flows to model the policy distribution, letting them create more flexible and expressive policies than traditional methods Rosa The core of their innovation is this one step policy network that they optimize using a Q-function regularized by behavioral cloning from expert trajectories Dev

Taro: So they’re trying to use those expert demonstrations to guide the flow-based model so it learns a better policy structure, which is useful when you don't have access to the actual system dynamics model Rosa It’s about using what we know from examples—like how someone expertly drives a car or controls a robot—to inform the learning process Dev

Dev: They are specifically studying two things: convergence and stabilizability of this one step policy while it's learning these control problems offline Rosa That means they’re checking if the policy actually settles down to a good answer, and whether that final answer is stable enough to actually keep the system from falling apart Taro

Taro: I mean, stability is a big deal. If the learned policy gets unstable, it doesn't matter how accurate it was during training; it just won't work in reality Dev And they prove that by showing their loss function is gradient dominant and smooth, they can guarantee this stability if you pick the learning rate right Rosa

Rosa: That’s a strong claim because proving stability under these conditions is rare in these kinds of online learning setups Taro It's not just about getting a good reward; it’s about the control loop staying within safe bounds during the training phase itself Dev So they are showing that this specific formulation of the loss helps them nail both accuracy and safety simultaneously Rosa

Dev: And they do this by using a new formulation of the one step policy loss based on average expected cost, and regularizing it with behavioral cloning loss Taro It’s a clever way to combine two different learning goals into one optimization problem for control systems Rosa

Conclusion: Rosa: So, looking at the whole thing, the title "Expert-Guided Online Reinforcement Learning for Stable and Sample-Efficient Control of Dynamical Systems" really captures the essence Dev It points to using that expert guidance to get control systems that are both stable during training and sample efficient when we only have limited data Rosa

Taro: I think what it means in plain terms is that we can use examples, even if they're just demonstrations, to train a controller without needing millions of real-world trials for every single scenario Dev It makes the process faster and more practical for complex systems where gathering new data is expensive or dangerous Rosa

Dev: The authors are showing that they can establish theoretical guarantees—convergence and stability—for this specific flow Q-learning approach when applied to linear quadratic problems in offline settings Taro That gives us confidence that we aren't just getting lucky with the training, but there’s a mathematical reason why the system should behave predictably Rosa

Rosa: And for someone just listening, it means that if you have expert data and you're working on a well-defined control problem like a linear quadratic regulator, this method gives you a mathematically sound path to get a controller running without worrying about catastrophic failure during the learning phase Dev It’s less about finding the perfect solution and more about building a reliable one through expert help Taro

Taro: I think the real implication is that we can start bridging that gap between purely model-based control, which relies on knowing all the physics, and data-driven learning, which relies on experience Rosa This paper shows how you can use both to build robust online controllers when you're stuck in an offline data environment Dev

Dev: Exactly. It sets a foundation for using expert knowledge as a strong prior in reinforcement learning for control tasks Taro And the authors also mentioned looking at extending this analysis to nonlinear systems and federated learning later on, which suggests they see this as a stepping stone for broader applications Rosa

Farnaz Adib Yaghmaie, Arunava Naha

Linkoping University

eess.SY, cs.SY

Submitted: 2025-11-14

Updated: 2026-11-12

Comments: Submitted to L4DC

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure The authors studied the convergence property and stabilizablity of the…

Key concepts

One-Step Policy
This is a simple control strategy where the action taken at any state depends only on that state, not on previous actions. In this paper, it's modeled as a linear state-feedback controller with added Gaussian noise to handle uncertainty.
Behavioral Cloning Loss
This loss term forces the learned policy to mimic the expert's behavior directly from observed data. It acts as a regularization technique, ensuring the learned policy stays close to what experts have done in the offline dataset.
Gradient Dominant Loss
A loss function is gradient dominant if its gradient consistently points towards a minimum, making optimization easier. The authors proved their specific one-step policy loss meets this condition, which is crucial for proving that the learning process will successfully find a good solution.
Stabilizability
This concept ensures that the controller learned through reinforcement learning actually keeps the system stable. The paper proves that under certain conditions, the resulting policy is guaranteed to stabilize systems like an inverted pendulum.

Terminology

Summary

Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure The authors studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings Their theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy They verify their theoretical finding with simulation results on a linearized inverted pendulum. p. 0

Where the paper says it

In this paper, we studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings. Our theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss.

Theoretical Framework for Policy Learning

The paper formulates learning the one-step policy as a policy gradient problem by optimizing the average expected cost regularized with the behavior cloning loss The one-step policy is defined as a linear state-feedback controller with additive Gaussian noise, specifically µK(x) = Kx + z, where z N (0, Wz) The one-step policy loss function for the LQR problem is given by l lin φo(K) = E x∼D z∼N (0,Wz) h J(µK) + α squared (Kx + z − µφb(x, z))⊤(Kx + z − µφb(x, z)).

Convergence Analysis of the Loss Function

The authors prove that the one-step policy loss l lin φo is gradient dominant by showing it is coercive and L-smooth. Specifically, Lemma 3 shows the gradient of the one-step policy loss l lin φo(K) is given by ∇Kl lin φo(K) =(Suu(θ)(K − K∗))Ex∼D[xx⊤] + α(K − K∗)Ex∼D[xx⊤]. p. 6

Where the paper says it

The proof of the Theorem 1 is also given in the appendix A. Lemma 3 The gradient of the one-step policy loss l lin φo (16) is given by ∇Kl lin φo (K) =(Suu(θ)(K − K∗ ))Ex∼D[xx⊤] + α(K − K∗ )Ex∼D[xx⊤], (18) where K∗ is the optimal gain ∇Kl lin φo (K∗ ) = 0.

Using these lemmas, Theorem 1 establishes that the one-step policy loss l lin φo is gradient dominant. p. 7
Where the paper says it

ADIB YAGHMAIE∗ NAHA∗ Using Lemmas 3-4, one can show that l lin φo is gradient dominant. Theorem 1 The one-step policy loss l lin φo (16) is gradient dominant, i.

Stabilizability of the Learned Policy

The stabilizability of the updated controller is guaranteed by Theorem 2, which states that for any α > 0, the one-step policy µφo learned via gradient descent in (17) with a step size η < 2L is stabilizing. This result follows because l lin φo is coercive and L-smooth with constant L. p. 7

Where the paper says it

In addition, the convergence rate is linear l lin φo (Kt ) − l lin φo (K∗ ) ≤ (1 − 2ηµ + η 2Lµ) t (l lin φo (K0 ) − l lin φo (K∗ )). (22) Proof Since l lin φo is coercive and L-smooth with constant L (Lemma 4), the controller gain learned via the gradient descent in (17) with a step size η < 2 L is stabilizing; i.

Furthermore, the convergence rate is linear, as shown by Theorem 2, which gives a bound on the convergence of the loss function. p. 8
Where the paper says it

Furthermore, the norm of the gradient of the one-step policy loss function with respect to the policy parameters is shown in Fig. The gradient-norm decays steadily with the training epochs, which is also consistent with our convergence analysis.

Simulation Results on Inverted Pendulum

The authors conduct simulation studies using a linearized inverted pendulum system as a benchmark example. The training protocol involves using an offline setting where data is stored in a replay buffer D, and the one-step actor policy is trained with a learning rate of 0. p. 3

Where the paper says it

learning from the replay buffer D or in an offline-online setup where data from online interaction is continuously added to the replay buffer. In the following, we give the loss functions for training the networks in the flow Q-learning algorithm The critic network Qφc : This is trained using the critic loss lφc = Ex,u,c,x′∼D u ′∼πφo [(Qφc (x, u) + c(x, u) − Qφ¯c (x ′ , u′ ))2 ], (3) where φ¯c denotes the parameters of the target critic network.

1 and behavioral cloning regularization weight α = 0. p. 8
Where the paper says it

It is trained using a learning rate of 0. 1 and batch size 256, with behavioral cloning regularization weight α = 0.

10. The performance comparison shows that the proposed method slightly outperforms the MPC baseline and performs close to the optimal LQR controller. p. 9
Where the paper says it

Here, each episode consists of 200 time-steps. We observe that the proposed method, after convergence, slightly outperforms the MPC baseline and performs close to the optimal LQR controller.

Future Research Directions

In future work, the authors plan to extend their convergence analysis to the complete AC algorithm, which means analyzing both the Q-function and one-step policy together. They also aim to investigate ways to extend their theoretical results to nonlinear systems and federated learning setups. p. 9

Where the paper says it

In our future work, we plan to extend our convergence analysis to the complete AC algorithm, which means analyzing the convergence of both the Q-function and one-step policy together. Furthermore, we also aim to investigate ways to extend our theoretical results to nonlinear systems and the federated learning setup.

Acknowledgments

Farnaz Adib Yaghmaie is supported by the Excellence Center at Linkoping–Lund in Information Technology (ELLIIT), ZENITH, and partially by Sensor informatics and Decision-making for the Digital Transformation (SEDDIT). Arunava Naha is a Wallenberg AI, Autonomous Systems and Software Program (WASP)- funded faculty member, and the work is partially supported by WASP, funded by the Knut and Alice Wallenberg Foundation. p. 9

Where the paper says it

This work was partly performed within the Competence Center SEDDIT-Sensor Informatics and Decision Making for the Digital Transformation, supported by Sweden’s Innovation Agency within the research and innovation program Advanced Digitalization. Additionally, Arunava Naha is a Wallenberg AI, Autonomous Systems and Software Program (WASP)- funded faculty member, and the work is partially supported by WASP, funded by the Knut and Alice Wallenberg Foundation.

References

Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic? p. 9

Where the paper says it

References Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic?

p. 9
Where the paper says it

References Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic?

model-based action-gradient- estimator policy optimization Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. p. 9
Where the paper says it

Advances in neural information processing systems, 33:313–324, 2020. Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar.

p. 9
Where the paper says it

Advances in neural information processing systems, 33:313–324, 2020. Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar.

Imitation learning for autonomous driv-ing: Insights from real-world testing Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. p. 9
Where the paper says it

Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Imitation learning for autonomous driv- ¨ ing: Insights from real-world testing.

p. 9
Where the paper says it

Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Imitation learning for autonomous driv- ¨ ing: Insights from real-world testing.

Global convergence of policy gradient methods for the linear quadratic regulator Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. p. 9
Where the paper says it

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.

p. 9
Where the paper says it

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.

Deep reinforcement learning that matters Gary Hewer. p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

An iterative technique for the computation of the steady state gains for the discrete optimal regulator Bin Hu, Kaiqing Zhang, Na Li, Mehran Mesbahi, Maryam Fazel, and Tamer Bas¸ar. p. 9
Where the paper says it

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.

p. 9
Where the paper says it

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.

Toward a theoretical foundation of policy optimization for learning control policies B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Perez. Deep reinforcement learning for autonomous driving: A survey Jens Kober, J. p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey Vijay R Konda and John N Tsitsiklis. p. 0
Where the paper says it

Keywords: Actor-Critic methods, Linear Quadratic Regulator, Flow Q-Learning 1. Introduction Actor-critic methods (Konda and Tsitsiklis, 2000; Lillicrap et al.

p. 0
Where the paper says it

Keywords: Actor-Critic methods, Linear Quadratic Regulator, Flow Q-Learning 1. Introduction Actor-critic methods (Konda and Tsitsiklis, 2000; Lillicrap et al.

Actor-critic algorithms Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. p. 0
Where the paper says it

SE Departement of Electrical Engineering, Linkoping University, Link ¨ oping, Sweden ¨ ∗ The authors contributed equally to this work. Abstract Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure.

p. 0
Where the paper says it

SE Departement of Electrical Engineering, Linkoping University, Link ¨ oping, Sweden ¨ ∗ The authors contributed equally to this work. Abstract Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure.

Offline reinforcement learning: Tutorial, review, and perspectives on open problems Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. p. 0
Where the paper says it

, 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al. , 2013), game playing (Silver et al.

p. 0
Where the paper says it

, 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al. , 2013), game playing (Silver et al.

Continuous control with deep reinforcement learning Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

Asynchronous methods for deep reinforcement learning Andrew Y Ng, Stuart Russell, et al. p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

p. 0
Where the paper says it

, 2015; Mnih et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied in various domains, including robotics (Kober et al.

Algorithms for inverse reinforcement learning Seohong Park, Qiyang Li, and Sergey Levine. p. 0
Where the paper says it

, 2018) or inverse reinforcement learning (Ng et al. , 2000), typically in offline RL setups which allows training effective policies from pre-collected datasets without further environment interactions (Levine et al.

p. 0
Where the paper says it

, 2018) or inverse reinforcement learning (Ng et al. , 2000), typically in offline RL setups which allows training effective policies from pre-collected datasets without further environment interactions (Levine et al.

Flow q-learning Martin L Puterman. p. 2
Where the paper says it

To learn the average cost, one needs to assume ergodicity of the MDP under the policy πφ (Puterman, 2014). Flow Q-learning Flow Q-learning (Park et al.

p. 2
Where the paper says it

To learn the average cost, one needs to assume ergodicity of the MDP under the policy πφ (Puterman, 2014). Flow Q-learning Flow Q-learning (Park et al.

Markov decision processes: discrete stochastic dynamic programming Georg Schildbach, Lorenzo Fagiano, Christoph Frei, and Manfred Morari. p. 2
Where the paper says it

Gradient dominant function: A function f : R n → R is gradient dominant if there exists a constant µ > 0 such that for all x ∈ R n f(x) − f(x ∗ ) ≤ 1 2µ ∥∇f(x)∥ 2 2 , where x ∗ is a global minimizer of f. MDP: We consider a Markov Decision Process (MDP) defined by the tuple (X , U, P, c, Σ0), where X → R n and U → R m denote the state and action spaces, P : X × U × X → [0, 1] is the transition probability function such that P(x ′ | x, u) denotes the probability of transitioning to state x ′ from state x under action u, c : X × U → R is the stage cost, and Σ0 is the initial state distribution.

p. 2
Where the paper says it

Gradient dominant function: A function f : R n → R is gradient dominant if there exists a constant µ > 0 such that for all x ∈ R n f(x) − f(x ∗ ) ≤ 1 2µ ∥∇f(x)∥ 2 2 , where x ∗ is a global minimizer of f. MDP: We consider a Markov Decision Process (MDP) defined by the tuple (X , U, P, c, Σ0), where X → R n and U → R m denote the state and action spaces, P : X × U × X → [0, 1] is the transition probability function such that P(x ′ | x, u) denotes the probability of transitioning to state x ′ from state x under action u, c : X × U → R is the stage cost, and Σ0 is the initial state distribution.

The scenario approach for stochastic model predictive control with bounds on closed-loop constraint violations Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. p. 0
Where the paper says it

This separation of roles is particularly efficient for control problems which typically have continuous action spaces as actor-critic methods directly learn a parameterized policy and as such avoid the need for computationally expensive value function optimization in TD approaches (Sutton et al. Despite their success, actor-critic methods face several challenges, including stability and convergence issues when applied to closed-loop dynamical systems (Henderson et al.

p. 0
Where the paper says it

This separation of roles is particularly efficient for control problems which typically have continuous action spaces as actor-critic methods directly learn a parameterized policy and as such avoid the need for computationally expensive value function optimization in TD approaches (Sutton et al. Despite their success, actor-critic methods face several challenges, including stability and convergence issues when applied to closed-loop dynamical systems (Henderson et al.

Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost Faraz Torabi, Garrett Warnell, and Peter Stone. p. 9
Where the paper says it

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.

p. 9
Where the paper says it

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.

Behavioral cloning from observation Toshiaki Tsuji, Yasuhiro Kato, Gokhan Solak, Heng Zhang, Tadej Petric, Francesco Nori, and ˇ Arash Ajoudani. A survey on imitation learning for contact-rich tasks in robotics Farnaz Adib Yaghmaie, Fredrik Gustafsson, and Lennart Ljung. p. 0
Where the paper says it

Proceedings of Machine Learning Research vol vvv:1–15, 2026 Convergence of Flow-Policy Gradient Learning for Linear Quadratic Regulator Problems Farnaz Adib Yaghmaie∗ FARNAZ. YAGHMAIE@LIU.

p. 0
Where the paper says it

Proceedings of Machine Learning Research vol vvv:1–15, 2026 Convergence of Flow-Policy Gradient Learning for Linear Quadratic Regulator Problems Farnaz Adib Yaghmaie∗ FARNAZ. YAGHMAIE@LIU.

Linear quadratic control using model-free reinforcement learning

Appendix A Helper Lemmas

A.3 Proof of Theorem 1

The proof of Theorem 1 involves combining the terms derived from the average cost and the behavioral cloning loss to obtain a final bound. The final result shows that l lin φo(K) − l lin φo(K∗) = Tr[(((Suu(θ) + αI)−1∇Kl lin φo(K)(Ex∼D[xx⊤])−1)⊤ × (Ru + α 2I)(Suu(θ) + αI−1∇Kl lin φo(K)). p. 6

Where the paper says it

Lemma 3 The gradient of the one-step policy loss l lin φo (16) is given by ∇Kl lin φo (K) =(Suu(θ)(K − K∗ ))Ex∼D[xx⊤] + α(K − K∗ )Ex∼D[xx⊤], (18) where K∗ is the optimal gain ∇Kl lin φo (K∗ ) = 0. Lemma 4 The one-step policy loss l lin φo (16) is L-smooth where L is given by L := ∥Suu(θ) + αI∥2∥Ex∼D[xx⊤]∥2.

A.2 Proof of Lemma 4

The smoothness of l lin φo can be easily verified by observing that ∇Kl lin φo in (18) is linear in K. Indeed, the norm of the second derivative is bounded as ∇2Kl lin φo(K) squared ≤ Suu(θ) + αI 2Ex∼D[xx⊤] squared. p. 6

Where the paper says it

Lemma 3 The gradient of the one-step policy loss l lin φo (16) is given by ∇Kl lin φo (K) =(Suu(θ)(K − K∗ ))Ex∼D[xx⊤] + α(K − K∗ )Ex∼D[xx⊤], (18) where K∗ is the optimal gain ∇Kl lin φo (K∗ ) = 0. Lemma 4 The one-step policy loss l lin φo (16) is L-smooth where L is given by L := ∥Suu(θ) + αI∥2∥Ex∼D[xx⊤]∥2.

Simulation Results

Figure 3 compares the total episodic costs for the proposed method, MPC, and LQR. The observation is that after convergence, the proposed method slightly outperforms the MPC baseline and performs close to the optimal LQR controller. p. 9

Where the paper says it

Here, each episode consists of 200 time-steps. We observe that the proposed method, after convergence, slightly outperforms the MPC baseline and performs close to the optimal LQR controller.

Conclusion

In this paper, they proved that the one-step policy loss is gradient-dominated and smooth, thereby stabilizing the learned one-step policy during learning when the learning rate is appropriately selected. The authors successfully studied the convergence and stabilizability of the one-step policy in linear quadratic problems under offline settings. p. 0

Where the paper says it

Central to this innovation is the “the one-step policy” network, which is optimized through a Q-function that is regularized with the behavioral cloning from expert trajectories, allowing learning more expressive policies using flow-based generative models. In this paper, we studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings.

A.3 Proof of Lemma 3

The one-step policy loss l lin φo contains two parts: Ez∼N (0,Wz) [J(µK)], E x∼D z∼N (0,Wz) [α squared (Kx + z − µφb(x, z))⊤(Kx + z − µφb(x, z))]:= LDistill. To find ∇KEz∼N (0,Wz) [J(µK)], they use the policy gradient theorem (Silver et al. p. 1

Where the paper says it

Formulating based on the average expected cost rather than the Q function, which was originally used in (Park et al. , 2025), allows us to apply the policy gradient framework (Silver et al.

, 2014). Based on the definition of Qφc(x, u) in (15), they find that ∇KEz∼N (0,Wz) [J(µK)] = E x∼D z∼N (0,Wz) [Sux(θ))xx⊤ + Suu(θ)ux⊤]. p. 6
Where the paper says it

Q-FUNCTION AND ONE-STEP POLICY STRUCTURE For the linear system in (6) with the quadratic cost in (7) and the average cost formulation in (8), the one-step policy is considered as µK(x) = Kx + z, z ∼ N (0, Wz) and the Qφc function is given by (Yaghmaie et al. , 2023) Qφc (x, u) = 1 2 x u ⊤ Sxx(θ) S ⊤ ux(θ) Sux(θ) Suu(θ) x u − J(µK), (15) where J(µK) is the average expected cost associated with the one-step policy, which is given in (12) and K is the controller gain to be optimized.

A.1 Proof of Lemma 3

The gradient of the one-step policy loss l lin φo(K) is given by ∇Kl lin φo(K) =(Suu(θ)(K − K∗))Ex∼D[xx⊤] + α(K − K∗)Ex∼D[xx⊤]. The optimal gain, denoted by K∗ makes the gradient of the one-step policy loss l lin φo(K) equal to zero. p. 6

Where the paper says it

The proof of the Theorem 1 is also given in the appendix A. Lemma 3 The gradient of the one-step policy loss l lin φo (16) is given by ∇Kl lin φo (K) =(Suu(θ)(K − K∗ ))Ex∼D[xx⊤] + α(K − K∗ )Ex∼D[xx⊤], (18) where K∗ is the optimal gain ∇Kl lin φo (K∗ ) = 0.

A.2 Proof of Lemma 4

The smoothness of l lin φo can be easily verified by observing that ∇Kl lin φo in (18) is linear in K. Indeed, ∇2Kl lin φo(K) squared = (Suu(θ) + αI)Ex∼D[xx⊤] squared ≤ Suu(θ) + αI 2Ex∼D[xx⊤] squared. p. 6

Where the paper says it

Lemma 3 The gradient of the one-step policy loss l lin φo (16) is given by ∇Kl lin φo (K) =(Suu(θ)(K − K∗ ))Ex∼D[xx⊤] + α(K − K∗ )Ex∼D[xx⊤], (18) where K∗ is the optimal gain ∇Kl lin φo (K∗ ) = 0. Lemma 4 The one-step policy loss l lin φo (16) is L-smooth where L is given by L := ∥Suu(θ) + αI∥2∥Ex∼D[xx⊤]∥2.

Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic? model-based action-gradient- estimator policy optimization Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Imitation learning for autonomous driv-ing: Insights from real-world testing Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters Gary Hewer. An iterative technique for the computation of the steady state gains for the discrete optimal regulator Bin Hu, Kaiqing Zhang, Na Li, Mehran Mesbahi, Maryam Fazel, and Tamer Bas¸ar. Toward a theoretical foundation of policy optimization for learning control policies B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Perez. Deep reinforcement learning for autonomous driving: A survey Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming Georg Schildbach, Lorenzo Fagiano, Christoph Frei, and Manfred Morari. The scenario approach for stochastic model predictive control with bounds on closed-loop constraint violations Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation Toshiaki Tsuji, Yasuhiro Kato, Gokhan Solak, Heng Zhang, Tadej Petric, Francesco Nori, and ˇ Arash Ajoudani. A survey on imitation learning for contact-rich tasks in robotics Farnaz Adib Yaghmaie, Fredrik Gustafsson, and Lennart Ljung.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Policy Learning via Flow Q-Learning can be adapted to learn more expressive policies using flow-based generative models, allowing for better exploration in control problems. This capability is particularly useful when system dynamics are unknown or inaccurate, as expert trajectories guide the learning process.

  2. The proposed method allows for learning a linear state feedback controller even when the true system dynamics are unknown by formulating the one-step policy loss based on the average expected cost regularized with behavioral cloning loss. This enables training effective policies from offline datasets without further environment interactions.

  3. The learned one-step policy remains stabilizing during the course of learning, which is guaranteed by proving that the one-step policy remains stabilizing during the course of learning through showing that l lin φo is gradient dominant. This ensures that the controller gain learned via gradient descent in (17) results in a controller gain K0 ∈ K.

  4. The system can converge to the optimal policy at a linear rate, as proven by showing that the one-step policy loss is gradient dominant, which leads to the bound l lin φo(K) − l lin φo(K∗) = Tr[(((Suu(θ) + αI)−1∇Kl lin φo(K)(Ex∼D[xx⊤]))−1)⊤ × (Ru + α2I)(Suu(θ) + αI)—1∇Kl lin φo(K))].

  5. The improved AI system can outperform model-based baselines like scenario-based MPC and optimal LQR controllers in simulation, performing close to the optimal LQR controller on tasks like controlling a linearized inverted pendulum from offline data.

Abstract

Online reinforcement learning (RL) often requires extensive interactions with the environment, which makes the training costly. Additionally, exploration can destabilise the dynamical system, which may be unsafe. On the other hand, offline learning from expert demonstrations avoids unsafe exploration but is limited by the quality and coverage of the offline dataset. In this paper, we propose a model-free RL approach that combines expert demonstrations with limited online interactions. Specifically, we optimise a policy objective consisting of an online performance term and a behavioural cloning regulariser computed from offline expert demonstrations. The regulariser encourages the learned policy to stay close to expert behaviour, reducing the need to explore unfamiliar actions. Expert demonstrations guide policy learning and reduce the need for many online interactions and exploration. For stochastic linear time-invariant (LTI) systems, we establish conditions for linear convergence of exact policy gradient updates and for all intermediate policies to remain stabilising. Although our theoretical analysis focuses on linear systems, we also evaluate the proposed approach on nonlinear systems to demonstrate its effectiveness beyond the linear setting. Numerical results show that the proposed method reaches comparable performance in fewer gradient steps and with less number of samples than a purely online RL baseline.

Sources

Related papers