Expert-Guided Online Reinforcement Learning for Stable and Sample-Efficient Control of Dynamical Systems
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Expert-Guided Online Reinforcement Learning for Stable and Sample-Efficient Control of Dynamical Systems".
Dev: Here's the gist of the paper. Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure The authors studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings Their theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy They verify their theoretical finding with simulation results on a linearized inverted pendulum.
Rosa: Theoretical Framework for Policy Learning The paper formulates learning the one-step policy as a policy gradient problem by optimizing the average expected cost regularized with the behavior cloning loss The one-step policy is defined as a linear state-feedback controller with additive Gaussian noise, specifically µK(x) = Kx + z, where z N (0, Wz) The one-step policy loss function for the LQR problem is given by l lin φo(K) = E x∼D z∼N (0,Wz) h J(µK) + α squared (Kx + z − µφb(x, z))⊤(Kx + z − µφb(x, z)).
Dev: Convergence Analysis of the Loss Function The authors prove that the one-step policy loss l lin φo is gradient dominant by showing it is coercive and L-smooth.
Rosa: Specifically, Lemma 3 shows the gradient of the one-step policy loss l lin φo(K) is given by ∇Kl lin φo(K) =(Suu(θ)(K − K∗))Ex∼D
xx⊤: + α(K − K∗)Ex∼D
xx⊤: .
Paper summary: Rosa: So, to wrap up what we just touched on, this paper is focusing on how expert guidance helps us train control policies for linear quadratic problems when we only have offline data available Dev They introduce Flow Q-learning which uses normalizing flows to model the policy distribution, letting them create more flexible and expressive policies than traditional methods Rosa The core of their innovation is this one step policy network that they optimize using a Q-function regularized by behavioral cloning from expert trajectories Dev
Taro: So they’re trying to use those expert demonstrations to guide the flow-based model so it learns a better policy structure, which is useful when you don't have access to the actual system dynamics model Rosa It’s about using what we know from examples—like how someone expertly drives a car or controls a robot—to inform the learning process Dev
Dev: They are specifically studying two things: convergence and stabilizability of this one step policy while it's learning these control problems offline Rosa That means they’re checking if the policy actually settles down to a good answer, and whether that final answer is stable enough to actually keep the system from falling apart Taro
Taro: I mean, stability is a big deal. If the learned policy gets unstable, it doesn't matter how accurate it was during training; it just won't work in reality Dev And they prove that by showing their loss function is gradient dominant and smooth, they can guarantee this stability if you pick the learning rate right Rosa
Rosa: That’s a strong claim because proving stability under these conditions is rare in these kinds of online learning setups Taro It's not just about getting a good reward; it’s about the control loop staying within safe bounds during the training phase itself Dev So they are showing that this specific formulation of the loss helps them nail both accuracy and safety simultaneously Rosa
Dev: And they do this by using a new formulation of the one step policy loss based on average expected cost, and regularizing it with behavioral cloning loss Taro It’s a clever way to combine two different learning goals into one optimization problem for control systems Rosa
Conclusion: Rosa: So, looking at the whole thing, the title "Expert-Guided Online Reinforcement Learning for Stable and Sample-Efficient Control of Dynamical Systems" really captures the essence Dev It points to using that expert guidance to get control systems that are both stable during training and sample efficient when we only have limited data Rosa
Taro: I think what it means in plain terms is that we can use examples, even if they're just demonstrations, to train a controller without needing millions of real-world trials for every single scenario Dev It makes the process faster and more practical for complex systems where gathering new data is expensive or dangerous Rosa
Dev: The authors are showing that they can establish theoretical guarantees—convergence and stability—for this specific flow Q-learning approach when applied to linear quadratic problems in offline settings Taro That gives us confidence that we aren't just getting lucky with the training, but there’s a mathematical reason why the system should behave predictably Rosa
Rosa: And for someone just listening, it means that if you have expert data and you're working on a well-defined control problem like a linear quadratic regulator, this method gives you a mathematically sound path to get a controller running without worrying about catastrophic failure during the learning phase Dev It’s less about finding the perfect solution and more about building a reliable one through expert help Taro
Taro: I think the real implication is that we can start bridging that gap between purely model-based control, which relies on knowing all the physics, and data-driven learning, which relies on experience Rosa This paper shows how you can use both to build robust online controllers when you're stuck in an offline data environment Dev
Dev: Exactly. It sets a foundation for using expert knowledge as a strong prior in reinforcement learning for control tasks Taro And the authors also mentioned looking at extending this analysis to nonlinear systems and federated learning later on, which suggests they see this as a stepping stone for broader applications Rosa
Farnaz Adib Yaghmaie, Arunava Naha
Linkoping University
eess.SY, cs.SY
Submitted: 2025-11-14
Updated: 2026-11-12
Comments: Submitted to L4DC
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure The authors studied the convergence property and stabilizablity of the…
Key concepts
- One-Step Policy
- This is a simple control strategy where the action taken at any state depends only on that state, not on previous actions. In this paper, it's modeled as a linear state-feedback controller with added Gaussian noise to handle uncertainty.
- Behavioral Cloning Loss
- This loss term forces the learned policy to mimic the expert's behavior directly from observed data. It acts as a regularization technique, ensuring the learned policy stays close to what experts have done in the offline dataset.
- Gradient Dominant Loss
- A loss function is gradient dominant if its gradient consistently points towards a minimum, making optimization easier. The authors proved their specific one-step policy loss meets this condition, which is crucial for proving that the learning process will successfully find a good solution.
- Stabilizability
- This concept ensures that the controller learned through reinforcement learning actually keeps the system stable. The paper proves that under certain conditions, the resulting policy is guaranteed to stabilize systems like an inverted pendulum.
Terminology
Summary
Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure The authors studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings Their theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy They verify their theoretical finding with simulation results on a linearized inverted pendulum. p. 0 In this
paper, we studied the convergence property and stabilizablity of the one-step policy during learning
for linear quadratic problems under the offline settings. Our theoretical results are based on a new
formulation of the one-step policy loss based on the average expected cost, and regularized with the
behavioral cloning loss.Where the paper says it
Theoretical Framework for Policy Learning
The paper formulates learning the one-step policy as a policy gradient problem by optimizing the average expected cost regularized with the behavior cloning loss The one-step policy is defined as a linear state-feedback controller with additive Gaussian noise, specifically µK(x) = Kx + z, where z N (0, Wz) The one-step policy loss function for the LQR problem is given by l lin φo(K) = E x∼D z∼N (0,Wz) h J(µK) + α squared (Kx + z − µφb(x, z))⊤(Kx + z − µφb(x, z)).
Convergence Analysis of the Loss Function
The authors prove that the one-step policy loss l lin φo is gradient dominant by showing it is coercive and L-smooth. Specifically, Lemma 3 shows the gradient of the one-step policy loss l lin φo(K) is given by ∇Kl lin φo(K) =(Suu(θ)(K − K∗))Ex∼D[xx⊤] + α(K − K∗)Ex∼D[xx⊤]. p. 6 The proof of the Theorem 1 is also given in the appendix A. Lemma 3 The gradient of the one-step policy loss l
lin
φo
(16) is given by
∇Kl
lin
φo
(K) =(Suu(θ)(K − K∗
))Ex∼D[xx⊤] + α(K − K∗
)Ex∼D[xx⊤], (18)
where K∗
is the optimal gain ∇Kl
lin
φo
(K∗
) = 0. ADIB YAGHMAIE∗ NAHA∗
Using Lemmas 3-4, one can show that l
lin
φo
is gradient dominant. Theorem 1 The one-step policy loss l
lin
φo
(16) is gradient dominant, i.Where the paper says it
Where the paper says it
Stabilizability of the Learned Policy
The stabilizability of the updated controller is guaranteed by Theorem 2, which states that for any α > 0, the one-step policy µφo learned via gradient descent in (17) with a step size η < 2L is stabilizing. This result follows because l lin φo is coercive and L-smooth with constant L. p. 7 In addition, the convergence rate is linear
l
lin
φo
(Kt
) − l
lin
φo
(K∗
) ≤ (1 − 2ηµ + η
2Lµ)
t
(l
lin
φo
(K0
) − l
lin
φo
(K∗
)). (22)
Proof Since l
lin
φo
is coercive and L-smooth with constant L (Lemma 4), the controller gain learned
via the gradient descent in (17) with a step size η < 2
L
is stabilizing; i. Furthermore, the
norm of the gradient of the one-step policy loss function with respect to the policy parameters is
shown in Fig. The gradient-norm decays steadily with the training epochs, which is also consistent with our convergence analysis.Where the paper says it
Where the paper says it
Simulation Results on Inverted Pendulum
The authors conduct simulation studies using a linearized inverted pendulum system as a benchmark example. The training protocol involves using an offline setting where data is stored in a replay buffer D, and the one-step actor policy is trained with a learning rate of 0. p. 3 learning from the replay
buffer D or in an offline-online setup where data from online interaction is continuously added to
the replay buffer. In the following, we give the loss functions for training the networks in the flow
Q-learning algorithm
The critic network Qφc
: This is trained using the critic loss
lφc = Ex,u,c,x′∼D
u
′∼πφo
[(Qφc
(x, u) + c(x, u) − Qφ¯c
(x
′
, u′
))2
], (3)
where φ¯c denotes the parameters of the target critic network. It is trained using a learning rate of 0. 1 and
batch size 256, with behavioral cloning regularization weight α = 0. Here, each episode consists of 200 time-steps. We observe that the proposed method, after convergence, slightly outperforms the MPC baseline and performs close to the
optimal LQR controller.Where the paper says it
Where the paper says it
Where the paper says it
Future Research Directions
In future work, the authors plan to extend their convergence analysis to the complete AC algorithm, which means analyzing both the Q-function and one-step policy together. They also aim to investigate ways to extend their theoretical results to nonlinear systems and federated learning setups. p. 9 In our future work, we plan to extend
our convergence analysis to the complete AC algorithm, which means analyzing the convergence of
both the Q-function and one-step policy together. Furthermore, we also aim to investigate ways to
extend our theoretical results to nonlinear systems and the federated learning setup.Where the paper says it
Acknowledgments
Farnaz Adib Yaghmaie is supported by the Excellence Center at Linkoping–Lund in Information Technology (ELLIIT), ZENITH, and partially by Sensor informatics and Decision-making for the Digital Transformation (SEDDIT). Arunava Naha is a Wallenberg AI, Autonomous Systems and Software Program (WASP)- funded faculty member, and the work is partially supported by WASP, funded by the Knut and Alice Wallenberg Foundation. p. 9 This work was partly performed within the Competence Center SEDDIT-Sensor Informatics and Decision Making for the Digital Transformation, supported
by Sweden’s Innovation Agency within the research and innovation program Advanced Digitalization. Additionally, Arunava Naha is a Wallenberg AI, Autonomous Systems and Software Program
(WASP)- funded faculty member, and the work is partially supported by WASP, funded by the Knut
and Alice Wallenberg Foundation.Where the paper says it
References
Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic? p. 9 References
Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic? References
Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic? Advances in neural information processing systems, 33:313–324,
2020. Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Advances in neural information processing systems, 33:313–324,
2020. Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Imitation learning for autonomous driv- ¨
ing: Insights from real-world testing. Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Imitation learning for autonomous driv- ¨
ing: Insights from real-world testing. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. Keywords: Actor-Critic methods, Linear Quadratic Regulator, Flow Q-Learning
1. Introduction
Actor-critic methods (Konda and Tsitsiklis, 2000; Lillicrap et al. Keywords: Actor-Critic methods, Linear Quadratic Regulator, Flow Q-Learning
1. Introduction
Actor-critic methods (Konda and Tsitsiklis, 2000; Lillicrap et al. SE
Departement of Electrical Engineering, Linkoping University, Link ¨ oping, Sweden ¨
∗ The authors contributed equally to this work. Abstract
Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into
an actor-critic structure. SE
Departement of Electrical Engineering, Linkoping University, Link ¨ oping, Sweden ¨
∗ The authors contributed equally to this work. Abstract
Flow Q-learning has recently been introduced to integrate learning from expert demonstrations into
an actor-critic structure. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2013), game playing (Silver et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2013), game playing (Silver et al. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2015; Mnih
et al. , 2016) are widely used in Reinforcement Learning (RL) and have been successfully applied
in various domains, including robotics (Kober et al. , 2018) or inverse reinforcement learning (Ng et al. , 2000), typically
in offline RL setups which allows training effective policies from pre-collected datasets without further environment interactions (Levine et al. , 2018) or inverse reinforcement learning (Ng et al. , 2000), typically
in offline RL setups which allows training effective policies from pre-collected datasets without further environment interactions (Levine et al. To learn the average cost, one needs to assume ergodicity of the MDP under the policy πφ
(Puterman, 2014). Flow Q-learning
Flow Q-learning (Park et al. To learn the average cost, one needs to assume ergodicity of the MDP under the policy πφ
(Puterman, 2014). Flow Q-learning
Flow Q-learning (Park et al. Gradient dominant function: A function f : R
n → R is gradient dominant if there exists a
constant µ > 0 such that for all x ∈ R
n
f(x) − f(x
∗
) ≤
1
2µ
∥∇f(x)∥
2
2
,
where x
∗
is a global minimizer of f. MDP: We consider a Markov Decision Process (MDP) defined by the tuple (X , U, P, c, Σ0),
where X → R
n
and U → R
m denote the state and action spaces, P : X × U × X → [0, 1] is
the transition probability function such that P(x
′
| x, u) denotes the probability of transitioning to
state x
′
from state x under action u, c : X × U → R is the stage cost, and Σ0 is the initial state
distribution. Gradient dominant function: A function f : R
n → R is gradient dominant if there exists a
constant µ > 0 such that for all x ∈ R
n
f(x) − f(x
∗
) ≤
1
2µ
∥∇f(x)∥
2
2
,
where x
∗
is a global minimizer of f. MDP: We consider a Markov Decision Process (MDP) defined by the tuple (X , U, P, c, Σ0),
where X → R
n
and U → R
m denote the state and action spaces, P : X × U × X → [0, 1] is
the transition probability function such that P(x
′
| x, u) denotes the probability of transitioning to
state x
′
from state x under action u, c : X × U → R is the stage cost, and Σ0 is the initial state
distribution. This separation of roles is particularly efficient for control problems which typically have continuous action
spaces as actor-critic methods directly learn a parameterized policy and as such avoid the need for
computationally expensive value function optimization in TD approaches (Sutton et al. Despite their success, actor-critic methods face several challenges, including stability and convergence
issues when applied to closed-loop dynamical systems (Henderson et al. This separation of roles is particularly efficient for control problems which typically have continuous action
spaces as actor-critic methods directly learn a parameterized policy and as such avoid the need for
computationally expensive value function optimization in TD approaches (Sutton et al. Despite their success, actor-critic methods face several challenges, including stability and convergence
issues when applied to closed-loop dynamical systems (Henderson et al. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. Proceedings of Machine Learning Research vol vvv:1–15, 2026
Convergence of Flow-Policy Gradient Learning for Linear Quadratic
Regulator Problems
Farnaz Adib Yaghmaie∗
FARNAZ. YAGHMAIE@LIU. Proceedings of Machine Learning Research vol vvv:1–15, 2026
Convergence of Flow-Policy Gradient Learning for Linear Quadratic
Regulator Problems
Farnaz Adib Yaghmaie∗
FARNAZ. YAGHMAIE@LIU.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Appendix A Helper Lemmas
A.3 Proof of Theorem 1
The proof of Theorem 1 involves combining the terms derived from the average cost and the behavioral cloning loss to obtain a final bound. The final result shows that l lin φo(K) − l lin φo(K∗) = Tr[(((Suu(θ) + αI)−1∇Kl lin φo(K)(Ex∼D[xx⊤])−1)⊤ × (Ru + α 2I)(Suu(θ) + αI−1∇Kl lin φo(K)). p. 6 Lemma 3 The gradient of the one-step policy loss l
lin
φo
(16) is given by
∇Kl
lin
φo
(K) =(Suu(θ)(K − K∗
))Ex∼D[xx⊤] + α(K − K∗
)Ex∼D[xx⊤], (18)
where K∗
is the optimal gain ∇Kl
lin
φo
(K∗
) = 0. Lemma 4 The one-step policy loss l
lin
φo
(16) is L-smooth where L is given by
L := ∥Suu(θ) + αI∥2∥Ex∼D[xx⊤]∥2.Where the paper says it
A.2 Proof of Lemma 4
The smoothness of l lin φo can be easily verified by observing that ∇Kl lin φo in (18) is linear in K. Indeed, the norm of the second derivative is bounded as ∇2Kl lin φo(K) squared ≤ Suu(θ) + αI 2Ex∼D[xx⊤] squared. p. 6 Lemma 3 The gradient of the one-step policy loss l
lin
φo
(16) is given by
∇Kl
lin
φo
(K) =(Suu(θ)(K − K∗
))Ex∼D[xx⊤] + α(K − K∗
)Ex∼D[xx⊤], (18)
where K∗
is the optimal gain ∇Kl
lin
φo
(K∗
) = 0. Lemma 4 The one-step policy loss l
lin
φo
(16) is L-smooth where L is given by
L := ∥Suu(θ) + αI∥2∥Ex∼D[xx⊤]∥2.Where the paper says it
Simulation Results
Figure 3 compares the total episodic costs for the proposed method, MPC, and LQR. The observation is that after convergence, the proposed method slightly outperforms the MPC baseline and performs close to the optimal LQR controller. p. 9 Here, each episode consists of 200 time-steps. We observe that the proposed method, after convergence, slightly outperforms the MPC baseline and performs close to the
optimal LQR controller.Where the paper says it
Conclusion
In this paper, they proved that the one-step policy loss is gradient-dominated and smooth, thereby stabilizing the learned one-step policy during learning when the learning rate is appropriately selected. The authors successfully studied the convergence and stabilizability of the one-step policy in linear quadratic problems under offline settings. p. 0 Central to this innovation is the “the one-step policy” network, which is
optimized through a Q-function that is regularized with the behavioral cloning from expert trajectories, allowing learning more expressive policies using flow-based generative models. In this
paper, we studied the convergence property and stabilizablity of the one-step policy during learning
for linear quadratic problems under the offline settings.Where the paper says it
A.3 Proof of Lemma 3
The one-step policy loss l lin φo contains two parts: Ez∼N (0,Wz) [J(µK)], E x∼D z∼N (0,Wz) [α squared (Kx + z − µφb(x, z))⊤(Kx + z − µφb(x, z))]:= LDistill. To find ∇KEz∼N (0,Wz) [J(µK)], they use the policy gradient theorem (Silver et al. p. 1 Formulating based on the average expected cost rather than the Q function, which was originally used in
(Park et al. , 2025), allows us to apply the policy gradient framework (Silver et al. Q-FUNCTION AND ONE-STEP POLICY STRUCTURE
For the linear system in (6) with the quadratic cost in (7) and the average cost formulation in (8), the
one-step policy is considered as µK(x) = Kx + z, z ∼ N (0, Wz) and the Qφc
function is given
by (Yaghmaie et al. , 2023)
Qφc
(x, u) = 1
2
x
u
⊤
Sxx(θ) S
⊤
ux(θ)
Sux(θ) Suu(θ)
x
u
− J(µK), (15)
where J(µK) is the average expected cost associated with the one-step policy, which is given in
(12) and K is the controller gain to be optimized.Where the paper says it
Where the paper says it
A.1 Proof of Lemma 3
The gradient of the one-step policy loss l lin φo(K) is given by ∇Kl lin φo(K) =(Suu(θ)(K − K∗))Ex∼D[xx⊤] + α(K − K∗)Ex∼D[xx⊤]. The optimal gain, denoted by K∗ makes the gradient of the one-step policy loss l lin φo(K) equal to zero. p. 6 The proof of the Theorem 1 is also given in the appendix A. Lemma 3 The gradient of the one-step policy loss l
lin
φo
(16) is given by
∇Kl
lin
φo
(K) =(Suu(θ)(K − K∗
))Ex∼D[xx⊤] + α(K − K∗
)Ex∼D[xx⊤], (18)
where K∗
is the optimal gain ∇Kl
lin
φo
(K∗
) = 0.Where the paper says it
A.2 Proof of Lemma 4
The smoothness of l lin φo can be easily verified by observing that ∇Kl lin φo in (18) is linear in K. Indeed, ∇2Kl lin φo(K) squared = (Suu(θ) + αI)Ex∼D[xx⊤] squared ≤ Suu(θ) + αI 2Ex∼D[xx⊤] squared. p. 6 Lemma 3 The gradient of the one-step policy loss l
lin
φo
(16) is given by
∇Kl
lin
φo
(K) =(Suu(θ)(K − K∗
))Ex∼D[xx⊤] + α(K − K∗
)Ex∼D[xx⊤], (18)
where K∗
is the optimal gain ∇Kl
lin
φo
(K∗
) = 0. Lemma 4 The one-step policy loss l
lin
φo
(16) is L-smooth where L is given by
L := ∥Suu(θ) + αI∥2∥Ex∼D[xx⊤]∥2.Where the paper says it
Pierluca D’Oro and Wojciech Jaskowski. How to learn a useful critic? model-based action-gradient- estimator policy optimization Hidayet Ersin Dursun, Yusuf Guven, and Tufan Kumbasar. Imitation learning for autonomous driv-ing: Insights from real-world testing Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters Gary Hewer. An iterative technique for the computation of the steady state gains for the discrete optimal regulator Bin Hu, Kaiqing Zhang, Na Li, Mehran Mesbahi, Maryam Fazel, and Tamer Bas¸ar. Toward a theoretical foundation of policy optimization for learning control policies B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Perez. Deep reinforcement learning for autonomous driving: A survey Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming Georg Schildbach, Lorenzo Fagiano, Christoph Frei, and Manfred Morari. The scenario approach for stochastic model predictive control with bounds on closed-loop constraint violations Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation Toshiaki Tsuji, Yasuhiro Kato, Gokhan Solak, Heng Zhang, Tadej Petric, Francesco Nori, and ˇ Arash Ajoudani. A survey on imitation learning for contact-rich tasks in robotics Farnaz Adib Yaghmaie, Fredrik Gustafsson, and Lennart Ljung.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Policy Learning via Flow Q-Learning can be adapted to learn more expressive policies using flow-based generative models, allowing for better exploration in control problems. This capability is particularly useful when system dynamics are unknown or inaccurate, as expert trajectories guide the learning process.
-
The proposed method allows for learning a linear state feedback controller even when the true system dynamics are unknown by formulating the one-step policy loss based on the average expected cost regularized with behavioral cloning loss. This enables training effective policies from offline datasets without further environment interactions.
-
The learned one-step policy remains stabilizing during the course of learning, which is guaranteed by proving that
the one-step policy remains stabilizing during the course of learning
through showing thatl lin φo is gradient dominant.
This ensures that the controller gain learned via gradient descent in (17) results in a controller gain K0 ∈ K. -
The system can converge to the optimal policy at a linear rate, as proven by showing that
the one-step policy loss is gradient dominant,
which leads to the bound l lin φo(K) − l lin φo(K∗) = Tr[(((Suu(θ) + αI)−1∇Kl lin φo(K)(Ex∼D[xx⊤]))−1)⊤ × (Ru + α2I)(Suu(θ) + αI)—1∇Kl lin φo(K))]. -
The improved AI system can outperform model-based baselines like scenario-based MPC and optimal LQR controllers in simulation, performing
close to the optimal LQR controller
on tasks like controlling a linearized inverted pendulum from offline data.
Abstract
Online reinforcement learning (RL) often requires extensive interactions with the environment, which makes the training costly. Additionally, exploration can destabilise the dynamical system, which may be unsafe. On the other hand, offline learning from expert demonstrations avoids unsafe exploration but is limited by the quality and coverage of the offline dataset. In this paper, we propose a model-free RL approach that combines expert demonstrations with limited online interactions. Specifically, we optimise a policy objective consisting of an online performance term and a behavioural cloning regulariser computed from offline expert demonstrations. The regulariser encourages the learned policy to stay close to expert behaviour, reducing the need to explore unfamiliar actions. Expert demonstrations guide policy learning and reduce the need for many online interactions and exploration. For stochastic linear time-invariant (LTI) systems, we establish conditions for linear convergence of exact policy gradient updates and for all intermediate policies to remain stabilising. Although our theoretical analysis focuses on linear systems, we also evaluate the proposed approach on nonlinear systems to demonstrate its effectiveness beyond the linear setting. Numerical results show that the proposed method reaches comparable performance in fewer gradient steps and with less number of samples than a purely online RL baseline.
Sources
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Continuous control with deep reinforcement learning
- A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
Related papers
- Industrial Load Modeling and Optimization for Market-Based Interaction with Power Systems
- Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
- Certified Set Convergence for Piecewise Affine Systems
- Symbolic Control for Stochastic Systems via Finite Parity Games
- Multiobjective optimization-based design and dispatch of islanded, hybrid microgrids for remote, off-grid communities in sub-Saharan Africa
- Physiology-informed layered sensing for intelligent human-exoskeleton interaction