Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control".
Dev: Here's the gist of the paper. Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the "sim2real gap" by proposing an online algorithm that uses offline data from a similar system to improve performance when controlling an unknown linear quadratic regulator (LQR) The proposed algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps Problem Formulation The classical LQR design involves minimizing a cost function Jπ(θ∗) for a discrete-time linear time-invariant system with unknown matrices A∗ and B∗ The problem is framed by considering an auxiliary system characterized by equation (3) where the simulator is assumed to be different from the true system The offline data D represents a trajectory of length S of state-action pairs from this simulated system The performance is quantified by the cumulative Bayesian regret defined as R(T, π) = E
X T t=1 ct − J(θ∗): How the training signal is built The algorithm utilizes Thompson Sampling (TSOD-LQR) which samples a model (system matrices) from a belief distribution over unknown system matrices The sampling equation for the parameter ˜θt is given by ˜θt = SQˆθt + βt(δ2)V−1/2 t ηt This step augments the online data with data collected from the simulated system to characterize the prior for learning θ∗ The algorithm updates its belief distribution using equations (9) and (10) to compute Vt and ˆθt Where the gains come from The regret analysis decomposes the expected regret into four terms R(T, TSOD-LQR)1
ET: ≤ R0 + R1 + R2 + R3 The term I is upper bounded by O˜(E
αS(δ1): ) Term II is upper bounded by O˜(E
p λmax(US)Mδ: ) Term III is upper bounded by O˜(1) The final regret bound is established as O˜ r T S log(T) + E
αS(δ1) + p λmax(US)Mδ: Generalization and Results The framework can be extended to multiple offline sources by defining a new error metric e(θ) and updating the belief distribution accordingly By setting S = PN i=1 Si and Mδ = max Mδ,i, a similar upper bound on the cumulative regret can be obtained When offline data from the same system is available (Mδ = 0), the regret is bounded by O(log T) Furthermore, when data D is generated from Algorithm TSAC and θ sim∗ ∈ P, the regret is at most O˜ r T S log S + √SMδ!
Rosa: In this work, an online control problem of an LQR when an offline trajectory of length S of state-action pairs from a similar linear system is available is considered The algorithm achieves O˜(f(S, Mδ)√T) regret, where f(S, Mδ) is a decreasing function of S This work provides additional numerical insights by comparing the proposed approach with two other approaches The final result shows that utilizing the estimate and the uncertainty from dissimilar systems can be beneficial REFERENCES arXiv:2505.09057v1
eess.SY: 14 May 2025 1 Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta Fellow, IEEE Abstract—“Sim2real gap”, in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system.
Dev: In this work, we address this challenge in the linear quadratic regulator (LQR) setting Specifically, we consider an LQR problem for a system with unknown system matrices.
Rosa: Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained We establish that the algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps, where Mδ characterizes the dissimilarity between the two systems and f(S, Mδ) is a function of S and Mδ When Mδ is sufficiently small, the proposed algorithm achieves O˜(p T /S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control I.
Paper summary: Rosa: So this paper is about a problem called the sim-to-real gap in linear quadratic regulator control, and it looks at using offline data from a similar system to fix that gap when you don't know the true system matrices.
Dev: Exactly, it tackles the issue where controllers learned from simulations don't work well on the real system because of these unknown parameters. The main idea is they propose an online algorithm that uses this offline data to improve performance for controlling an unknown linear quadratic regulator.
Taro: It sounds like they're dealing with a situation where you have the actual system, but you have a trajectory of state-action pairs from a different, similar system available for training. How does that help the online control loop?
Rosa: Well, they frame this by saying the true system matrices A star and B star are different from some simulated matrices A sim and B sim. They then define an auxiliary system with equation three that represents this simulator, and they use the offline data D to characterize how different these two systems are.
Dev: It sets up a formal problem statement where you have the classical LQR design, which is minimizing a cost function J pi of theta star, but theta star is unknown. The core challenge is dealing with that uncertainty while trying to control the system using an online policy.
Taro: So they are not just trying to guess the parameters; they're using Thompson sampling, which samples a model from a belief distribution over those unknown system matrices to guide the online action selection.
Rosa: Right, and this sampling step involves augmenting your online data with information from the simulated system. They have an equation for sampling theta t that mixes the current estimate with information from the simulated system's uncertainty.
Dev: The key claim they make is that this approach achieves a Bayes regret of O f of S and M delta times T divided by S after T time steps, where S is the length of the offline data trajectory and M delta measures how different those two systems are.
Taro: That looks like a scaling factor involving the size of the offline data set and some measure of system mismatch, which is interesting because it suggests that having more similar offline data helps reduce that regret bound.
Rosa: And when that mismatch measure, M delta, is small enough, they find they get a better bound of O p T divided by S Bayes regret. That means if the systems are very similar, the performance scales nicely with the size of your offline dataset and time steps.
Paper summary: Dev: It also notes that when you have data from the same system available—meaning M delta is zero—the regret is bounded by O log T. That's a much better bound than what you get without any offline data at all, which usually involves square root of time or similar terms depending on the setup.
Taro: So, if you have perfect similarity between the simulated and real systems, the advantage of using that offline data is really just getting a logarithmic dependency on time steps. But what happens when they aren't perfectly similar?
Rosa: When they aren't perfectly similar, they still get a bound dependent on S and M delta, which is what the f of S and M delta term captures. They show numerically that as you increase the size of the offline trajectory, S, you can actually improve the regret with a fairly small number of measurements from your online system.
Dev: From an engineering standpoint, this means you can use a finite amount of offline data to get better guarantees than just relying on pure online exploration when dealing with these unknown system matrices. The safety case here is about controlling the error in that regret bound.
Taro: I wonder how robust this is when the world misbehaves unexpectedly during the online phase, since they are using a sample model to make decisions at each step based on that offline knowledge.
Rosa: They do use Thompson sampling to sample a parameter, compute the optimal control gain with that sample, and then transition to the next state and get feedback. The uncertainty from the simulated system is used as a prior for learning about the true system parameters.
Dev: It’s essentially using that knowledge of what's similar to help you make better guesses about what's different, which is how they characterize A star and B star by adding error terms from the simulator matrices.
Taro: So, the main implication for someone who only listens to this show is that when you face an unknown system and you have some data from a related system lying around, you can potentially achieve better long-term control performance bounds than if you were completely starting from scratch online.
Rosa: That's the gist of it. The paper, "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control," shows that using offline data to characterize the prior distribution for learning unknown system dynamics in an LQR setting can lead to a specific regret bound that depends on how similar the systems are and how much offline data you have.
Dev: It’s about bridging that sim-to-real gap by leveraging structural similarity in the system models. It's a method built on Thompson sampling that effectively uses the uncertainty from one system to inform your control strategy for another.
Conclusion: Rosa: So we've been looking at this paper from Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta, titled "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control."
Dev: It’s about how you can use data from a system that’s similar to the one you're actually trying to control when you don't know the exact system settings.
Rosa: Exactly. They tackle that sim-to-real gap in linear quadratic regulator control by using offline data from a different system to improve performance online.
Dev: The main takeaway is they prove an algorithm can achieve a certain regret bound, which means how much worse the online solution is compared to the absolute best possible solution if you knew everything upfront.
Rosa: They show that this regret scales in a way that depends on how much data you have and how different those two systems are.
Dev: The results suggest that having more similar offline data helps bring down that performance penalty, especially when the difference between the systems is small.
Rosa: And they find it gets even better bounds if the data comes from the exact same system, which is a pretty strong result for real-world applications.
Dev: It's about using structural similarity to get better control guarantees without needing infinite time to explore everything online.
Rosa: So, this moves us closer to having more reliable online controllers that can handle those tricky real-world mismatches.
Dev: But we still need to figure out the practical details of how this algorithm runs in a live loop with real latency.
Purdue University
eess.SY, cs.SY
Submitted: 2025-05-14
Updated: 2026-11-04
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 75/100
The gist: Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the "sim2real gap" by proposing an online algorithm that uses offline data from a similar system to improve…
Key concepts
- Sim2real Gap
- This refers to the performance loss that occurs when a controller trained using data from a simulated system is applied to a real system. The paper tackles this by using offline data from a similar simulator to inform the online learning process, aiming for better real-world control performance.
- Thompson Sampling (TSOD-LQR)
- This is the core online learning technique used. It works by sampling potential models (system matrices) from a belief distribution and taking an action based on that sample. It then updates this belief distribution using new online data, effectively balancing exploration and exploitation.
- Bayes Regret
- This metric measures the cumulative difference between the cost achieved by the learned controller over T time steps and the optimal cost achievable if all system parameters were known beforehand. A lower Bayes regret indicates better performance in controlling unknown LQR systems.
Terminology
Summary
Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the sim2real gap
by proposing an online algorithm that uses offline data from a similar system to improve performance when controlling an unknown linear quadratic regulator (LQR) The proposed algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps
Problem Formulation
The classical LQR design involves minimizing a cost function Jπ(θ∗) for a discrete-time linear time-invariant system with unknown matrices A∗ and B∗ The problem is framed by considering an auxiliary system characterized by equation (3) where the simulator is assumed to be different from the true system The offline data D represents a trajectory of length S of state-action pairs from this simulated system The performance is quantified by the cumulative Bayesian regret defined as R(T, π) = E [X T t=1 ct − J(θ∗)]
How the training signal is built
The algorithm utilizes Thompson Sampling (TSOD-LQR) which samples a model (system matrices) from a belief distribution over unknown system matrices The sampling equation for the parameter ˜θt is given by ˜θt = SQˆθt + βt(δ2)V−1/2 t ηt This step augments the online data with data collected from the simulated system to characterize the prior for learning θ∗ The algorithm updates its belief distribution using equations (9) and (10) to compute Vt and ˆθt
Where the gains come from
The regret analysis decomposes the expected regret into four terms R(T, TSOD-LQR)1[ET] ≤ R0 + R1 + R2 + R3 The term I is upper bounded by O˜(E[αS(δ1)]) Term II is upper bounded by O˜(E[p λmax(US)Mδ]) Term III is upper bounded by O˜(1) The final regret bound is established as O˜ r T S log(T) + E[αS(δ1) + p λmax(US)Mδ]
Generalization and Results
The framework can be extended to multiple offline sources by defining a new error metric e(θ) and updating the belief distribution accordingly By setting S = PN i=1 Si and Mδ = max Mδ,i, a similar upper bound on the cumulative regret can be obtained When offline data from the same system is available (Mδ = 0), the regret is bounded by O(log T) Furthermore, when data D is generated from Algorithm TSAC and θ sim∗ ∈ P, the regret is at most O˜ r T S log S + √SMδ!
Conclusion
In this work, an online control problem of an LQR when an offline trajectory of length S of state-action pairs from a similar linear system is available is considered The algorithm achieves O˜(f(S, Mδ)√T) regret, where f(S, Mδ) is a decreasing function of S This work provides additional numerical insights by comparing the proposed approach with two other approaches The final result shows that utilizing the estimate and the uncertainty from dissimilar systems can be beneficial
REFERENCES
arXiv:2505.09057v1 [eess. p. 0 arXiv:2505. 09057v1 [eess.Where the paper says it
1
Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta Fellow, IEEE Abstract—“Sim2real gap”, in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting Specifically, we consider an LQR problem for a system with unknown system matrices. p. 0 In this work, we address this challenge in
the linear quadratic regulator (LQR) setting. Specifically, we
consider an LQR problem for a system with unknown system
matrices.Where the paper says it
I. INTRODUCTION Online learning of linear quadratic regulators (LQRs) with unknown system matrices is a well-studied problem <ref:2505. p. 0 Index Terms—Identification for control, Sampled Data
Control, Autonomous Systems, Adaptive Control
I. INTRODUCTION
Online learning of linear quadratic regulators (LQRs)
with unknown system matrices is a well-studied problem. Many recent works have proposed novel algorithms with
performance guarantees on their (cumulative) regret, defined
as the difference between the cumulative cost with a controller
from the learning algorithm and the cost with an optimal
controller that knows the system matrices [1]–[4]. However,
these algorithms require long exploration times [5], which
impedes their usage in many practical applications. We propose an online
algorithm that leverages offline data to provably reduce the
exploration time, leading to lower regret. Leveraging offline data is not a new idea. Offline
reinforcement learning [6], for instance, uses offline data to
learn a policy which is used online. However, this leads to
the problem of sim-to-real gap or distribution shift since the
system parameters learned offline are different from the ones
encountered online [7]. However, this leads to
the problem of sim-to-real gap or distribution shift since the
system parameters learned offline are different from the ones
encountered online [7]. Although many methods have been
proposed in the literature to be robust to such issues, in
general, such policies are not optimal for the new system. Although many methods have been
proposed in the literature to be robust to such issues, in
general, such policies are not optimal for the new system. Another approach is to utilize the offline data to warm-start
Shivam Bajaj and Vijay Gupta are with the Department of Electrical
and Computer Engineering, Purdue University. Such strategies have been shown
to achieve an improved bound on the regret in multi-armed
bandits [8]–[11]. However, extending these algorithms to LQR
design and establishing their theoretical properties remains
unexplored, particularly for characterizing when they provide
benefits over learning the policy in a purely online fashion. Our algorithm provides a framework to incorporate offline
data from a similar linear system1
for online learning, which
provably achieves O˜(f(S, Mδ)
p
T /S) upper bound on the
regret, where S denotes the offline trajectory length and
Mδ quantifies the heterogeneity between the two systems. Our algorithm utilizes both the system matrices estimated
from offline data and the residual uncertainty. We show via
numerical simulations that as S increases, an improved regret
can be achieved with a fairly small number of measurements
from the online system. Our algorithm uses Thompson Sampling (TS) which
samples a model (system matrices) from a belief distribution
over unknown system matrices, takes the optimal action
based on the sample model, and subsequently updates the
belief distribution using the observed feedback (cost). Finally, our work
is also related to the growing literature on transfer learning
for linear systems [16]–[19]. However, unlike that stream, this
work focuses on determining regret guarantees on online LQR
control while leveraging offline data. However, unlike that stream, this
work focuses on determining regret guarantees on online LQR
control while leveraging offline data. This work is organized as follows. This work is organized as follows. Section II presents the
problem definition and a summary of background material. Section II presents the
problem definition and a summary of background material. Section III describes the offline data scheme. Section III describes the offline data scheme. Section IV
presents the proposed algorithm which is analyzed in Section
V. Section IV
presents the proposed algorithm which is analyzed in Section
V. In Section VI, we present additional numerical insights and
discuss how this work extends to when data from multiple
sources is available. Finally, Section VIII summarizes this
work and outlines directions for future work. Notation: ∥ · ∥, ∥ · ∥F , ∥ · ∥2, and Tr(·) denotes the
operator norm, Frobenius norm, spectral norm, and the trace,
respectively. Notation: ∥ · ∥, ∥ · ∥F , ∥ · ∥2, and Tr(·) denotes the
operator norm, Frobenius norm, spectral norm, and the trace,
respectively. For a positive definite matrix A (denoted as A ≻
0), λmax(A) and λmin(A) denote its maximum and minimum
eigenvalue, respectively.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
A1 B1 − A2 B2 ≤ Mδ.
II. PROBLEM FORMULATION We first review the classical LQR control problem and then describe our model followed by the formal problem statement <ref:2505. p. 1 Given a set P and a sample θ, SP represents a sampling
operator that ensures θ ∈ P. PROBLEM FORMULATION
We first review the classical LQR control problem and then
describe our model followed by the formal problem statement. PROBLEM FORMULATION
We first review the classical LQR control problem and then
describe our model followed by the formal problem statement. Classical LQR Design
Let xt ∈ R
n denote the state and ut ∈ R
m denote
the control at time t. Classical LQR Design
Let xt ∈ R
n denote the state and ut ∈ R
m denote
the control at time t. Let A∗ ∈ R
n×n and B∗ ∈ R
n×m
be the system matrices. arXiv:2505. 09057v1 [eess. Further, let θ
⊤
∗
:=
A∗ B∗
and
zt
:=
h
x
⊤
t u
⊤
t
i⊤
. Then, for t ≥ 1 and given matrices
Q ≻ 0, R ≻ 0, consider a discrete-time linear time-invariant
system with the dynamics and the cost function
xt+1 = θ
⊤
∗
zt + wt, (1)
ct = x
⊤
t Qxt + u
⊤
t Rut,
where wt ∼ N (0, I) is the system noise assumed to be white
and x1 = 0. Further, let θ
⊤
∗
:=
A∗ B∗
and
zt
:=
h
x
⊤
t u
⊤
t
i⊤
. Then, for t ≥ 1 and given matrices
Q ≻ 0, R ≻ 0, consider a discrete-time linear time-invariant
system with the dynamics and the cost function
xt+1 = θ
⊤
∗
zt + wt, (1)
ct = x
⊤
t Qxt + u
⊤
t Rut,
where wt ∼ N (0, I) is the system noise assumed to be white
and x1 = 0. The classical LQR control problem is to design
a closed-loop control π : R
n → R
m with ut = π(xt) that
minimizes the following cost:
Jπ(θ∗) = lim
T→∞
sup
1
T
X
T
t=1
E[ct(xt, ut)]. (2)
When θ∗ is known and under the assumption that (A∗, B∗)
is stabilizable, the optimal policy is ut = K(θ∗)xt and the
corresponding cost is J(θ∗) := Tr(P(θ∗)) where
K(θ∗) = −(R + B
⊤
∗ P(θ∗)B∗)
−1B
⊤
∗ P(θ∗)A∗
is the gain matrix and P(θ∗) is the unique positive definite
solution to the Riccati equation
P(θ∗) = Q + A
⊤
∗ P(θ∗)A∗ + A
⊤
∗ P(θ∗)B∗K(θ∗).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
B. Model and Problem Statement Consider a system characterized by equation (1) with unknown θ∗ and access to an offline dataset obtained through an approximated simulator <ref:2505. p. 1 (2)
When θ∗ is known and under the assumption that (A∗, B∗)
is stabilizable, the optimal policy is ut = K(θ∗)xt and the
corresponding cost is J(θ∗) := Tr(P(θ∗)) where
K(θ∗) = −(R + B
⊤
∗ P(θ∗)B∗)
−1B
⊤
∗ P(θ∗)A∗
is the gain matrix and P(θ∗) is the unique positive definite
solution to the Riccati equation
P(θ∗) = Q + A
⊤
∗ P(θ∗)A∗ + A
⊤
∗ P(θ∗)B∗K(θ∗). Model and Problem Statement
Consider a system characterized by equation (1) with
unknown θ∗ and access to an offline dataset obtained through
an approximated simulator. Model and Problem Statement
Consider a system characterized by equation (1) with
unknown θ∗ and access to an offline dataset obtained through
an approximated simulator. The simulator is assumed to be
characterized by the following auxiliary system which is
different than θ∗ and is also unknown. The simulator is assumed to be
characterized by the following auxiliary system which is
different than θ∗ and is also unknown. ξs+1 = θ
sim
∗
⊤
ys + w
sim
s
, (3)
where, ξs ∈ R
n and vs ∈ R
m denotes the state and the control,
respectively, at time instant s, θ
sim
∗
⊤
:=
h
Asim
∗ Bsim
∗
i
denotes
the system matrices, and ys :=
h
ξ
⊤
s
v
⊤
s
i⊤
.Where the paper says it
Where the paper says it
Where the paper says it
y1,..., yS represents a trajectory of length S of state-action pairs (ξs, vs), 1 ≤ s ≤ S <ref:2505. p. 1 The offline data
D = {y1, . , yS} represents a trajectory of length S of
state-action pairs (ξs, vs), 1 ≤ s ≤ S. , yS} represents a trajectory of length S of
state-action pairs (ξs, vs), 1 ≤ s ≤ S. We can characterize
A∗ and B∗ as A∗ = Asim
∗ + Aδ
∗
and B∗ = Bsim
∗ + Bδ
∗
,
respectively, where Aδ
∗
(resp. We can characterize
A∗ and B∗ as A∗ = Asim
∗ + Aδ
∗
and B∗ = Bsim
∗ + Bδ
∗
,
respectively, where Aδ
∗
(resp. Bδ
∗
) represents the change in the
system matrices A∗ (resp. , yS} represents a trajectory of length S of
state-action pairs (ξs, vs), 1 ≤ s ≤ S. We can characterize
A∗ and B∗ as A∗ = Asim
∗ + Aδ
∗
and B∗ = Bsim
∗ + Bδ
∗
,
respectively, where Aδ
∗
(resp. arXiv:2505. 09057v1 [eess. Thus,
the system characterized by equation (1) can be expressed as
xt+1 =
A
sim
∗ + A
δ
∗
xt +
B
sim
∗ + B
δ
∗
ut + wt. (4)
In this work, we assume that there exists a known constant
Mδ such that
θ
δ
∗
F
≤ Mδ, where θ
δ
∗
⊤
:=
h
Aδ
∗ Bδ
∗
i
. Thus,
the system characterized by equation (1) can be expressed as
xt+1 =
A
sim
∗ + A
δ
∗
xt +
B
sim
∗ + B
δ
∗
ut + wt. (4)
In this work, we assume that there exists a known constant
Mδ such that
θ
δ
∗
F
≤ Mδ, where θ
δ
∗
⊤
:=
h
Aδ
∗ Bδ
∗
i
. PROBLEM FORMULATION
We first review the classical LQR control problem and then
describe our model followed by the formal problem statement. Classical LQR Design
Let xt ∈ R
n denote the state and ut ∈ R
m denote
the control at time t. Let Ft
:= σ({x1, u1, . , xt, ut}) denote the filtration that
represents the knowledge up to time t during the online
process. , xt, ut}) denote the filtration that
represents the knowledge up to time t during the online
process. Similarly, let Fs := σ({ξ1, v1, . Similarly, let Fs := σ({ξ1, v1, . , ξS, vS}) denote
the filtration that represents the knowledge corresponding
to the offline data. , ξS, vS}) denote
the filtration that represents the knowledge corresponding
to the offline data. Then, we make the following standard
assumption on the noise process [20]. Assumption 1. There exists a filtration Ft and Fs such that
for any t ≥ 1 and 1 ≤ s ≤ S, zt, xt are Ft ∪ Fs-measurable
and ys, ξs are Fs-measurable. There exists a filtration Ft and Fs such that
for any t ≥ 1 and 1 ≤ s ≤ S, zt, xt are Ft ∪ Fs-measurable
and ys, ξs are Fs-measurable. Further, wt+1 and w
sim
s+1 are
individually martingale difference sequences. Further, wt+1 and w
sim
s+1 are
individually martingale difference sequences. Finally, for ease
of exposition, we assume that E[wt+1w
⊤
t+1|Ft ∪ Fs] = I and
E[w
sim
s+1w
sim
s+1
⊤
|Fs] = I. Finally, for ease
of exposition, we assume that E[wt+1w
⊤
t+1|Ft ∪ Fs] = I and
E[w
sim
s+1w
sim
s+1
⊤
|Fs] = I. Assuming that the parameter θ∗ is a random variable with
a known distribution µ, we quantify the performance of our
learning algorithm by comparing the cumulative cost to the
infinite-horizon cost attained by the LQR controller if the
system matrices defined by θ∗ were known a priori. Formally,
we quantify the performance of our algorithm through the
cumulative Bayesian regret defined as follows. R(T, π) = E
X
T
t=1
ct − J(θ∗)
, (5)
where the expectation is with respect to wt, µ, and any
randomization in the algorithms used to process the offline
and online data. R(T, π) = E
X
T
t=1
ct − J(θ∗)
, (5)
where the expectation is with respect to wt, µ, and any
randomization in the algorithms used to process the offline
and online data. This metric has been previously considered
for online control of LQR systems [3]. In this work, we address this challenge in
the linear quadratic regulator (LQR) setting. Specifically, we
consider an LQR problem for a system with unknown system
matrices. This metric has been previously considered
for online control of LQR systems [3]. The aim of this work is to find a control algorithm
that minimizes the expected regret defined in (5) while utilizing
the offline data D.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
III. OFFLINE DATA-GENERATION In this work, we do not consider a particular algorithm from which the offline data is generated <ref:2505. p. 1 The aim of this work is to find a control algorithm
that minimizes the expected regret defined in (5) while utilizing
the offline data D. OFFLINE DATA-GENERATION
In this work, we do not consider a particular algorithm from
which the offline data is generated. OFFLINE DATA-GENERATION
In this work, we do not consider a particular algorithm from
which the offline data is generated. As we will see later, any
algorithm that satisfies the following two properties can be
used to generate the offline dataset D. As we will see later, any
algorithm that satisfies the following two properties can be
used to generate the offline dataset D. Let Asim denote an
algorithm that is used to generate the offline data. Let Asim denote an
algorithm that is used to generate the offline data. Further, let
at time s, Us :=
Ps
k=0 ysy
⊤
s denote the precision matrix of
Algorithm Asim. Further, let
at time s, Us :=
Ps
k=0 ysy
⊤
s denote the precision matrix of
Algorithm Asim. We assume the following on algorithm Asim. We assume the following on algorithm Asim. Assumption 2 (Offline Algorithm). Assumption 2 (Offline Algorithm). For a given δ1 ∈ (0, 1),
with probability of at least 1 − δ1, Algorithm Asim satisfies
1) ∥U
0. arXiv:2505. 09057v1 [eess. 5
s
(
ˆθ
sim
s − θ
sim
∗
)∥F ≤ αs(δ1). 2) For s ≥ 200(n + m) log 12
δ1
, λmin(Us) ≥
s
40 .Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Leverage offline data from similar systems for online linear quadratic control to reduce exploration time, leading to lower regret in LQR problems with unknown system matrices. This allows an online learning algorithm to perform better than purely online methods by incorporating knowledge from a simulator or simpler model of the unknown system.
-
Implement Thompson Sampling with Learned Predictions (TSOD-LQR) as the proposed algorithm which samples a model from a belief distribution over unknown system matrices and utilizes both the mean and uncertainty of the dynamics of the system from which offline data is obtained. This approach achieves O(f(S, Mdelta) p T /S) Bayes regret after T time steps, where S denotes the offline trajectory length.
-
Design an improved control algorithm that incorporates data from multiple dissimilar offline sources by defining a new l2-least squares error and updating the belief distribution using aggregated information from all available trajectories. This generalization allows for online control of LQR when offline data from multiple systems is available, with a similar upper bound on cumulative regret.
Abstract
``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained. We establish that the algorithm achieves O tilde(f(S,M δ) sqrt T/S) Bayes regret after T time steps, where M δ characterizes the dissimilarity between the two systems and f(S,M δ) is a function of S and M δ. When M δ is sufficiently small, the proposed algorithm achieves O tilde(sqrt T/S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.
Sources
- Reinforcement Learning in Practice: Opportunities and Challenges
- Leveraging Offline Data in Linear Latent Contextual Bandits
- Transfer Learning for LQR Control
- Posterior Sampling for Reinforcement Learning Without Episodes
- Online Least Squares Estimation with Self-Normalized Processes: An Application to Bandit Problems
Related papers
- Industrial Load Modeling and Optimization for Market-Based Interaction with Power Systems
- Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
- Certified Set Convergence for Piecewise Affine Systems
- Symbolic Control for Stochastic Systems via Finite Parity Games
- Multiobjective optimization-based design and dispatch of islanded, hybrid microgrids for remote, off-grid communities in sub-Saharan Africa
- Physiology-informed layered sensing for intelligent human-exoskeleton interaction