Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control

arXiv:2505.09057 · eess.SY, cs.SY · Submitted 2025-05-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control".

Dev: Here's the gist of the paper. Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the "sim2real gap" by proposing an online algorithm that uses offline data from a similar system to improve performance when controlling an unknown linear quadratic regulator (LQR) The proposed algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps Problem Formulation The classical LQR design involves minimizing a cost function Jπ(θ∗) for a discrete-time linear time-invariant system with unknown matrices A∗ and B∗ The problem is framed by considering an auxiliary system characterized by equation (3) where the simulator is assumed to be different from the true system The offline data D represents a trajectory of length S of state-action pairs from this simulated system The performance is quantified by the cumulative Bayesian regret defined as R(T, π) = E

X T t=1 ct − J(θ∗): How the training signal is built The algorithm utilizes Thompson Sampling (TSOD-LQR) which samples a model (system matrices) from a belief distribution over unknown system matrices The sampling equation for the parameter ˜θt is given by ˜θt = SQˆθt + βt(δ2)V−1/2 t ηt This step augments the online data with data collected from the simulated system to characterize the prior for learning θ∗ The algorithm updates its belief distribution using equations (9) and (10) to compute Vt and ˆθt Where the gains come from The regret analysis decomposes the expected regret into four terms R(T, TSOD-LQR)1

ET: ≤ R0 + R1 + R2 + R3 The term I is upper bounded by O˜(E

αS(δ1): ) Term II is upper bounded by O˜(E

p λmax(US)Mδ: ) Term III is upper bounded by O˜(1) The final regret bound is established as O˜ r T S log(T) + E

αS(δ1) + p λmax(US)Mδ: Generalization and Results The framework can be extended to multiple offline sources by defining a new error metric e(θ) and updating the belief distribution accordingly By setting S = PN i=1 Si and Mδ = max Mδ,i, a similar upper bound on the cumulative regret can be obtained When offline data from the same system is available (Mδ = 0), the regret is bounded by O(log T) Furthermore, when data D is generated from Algorithm TSAC and θ sim∗ ∈ P, the regret is at most O˜ r T S log S + √SMδ!

Rosa: In this work, an online control problem of an LQR when an offline trajectory of length S of state-action pairs from a similar linear system is available is considered The algorithm achieves O˜(f(S, Mδ)√T) regret, where f(S, Mδ) is a decreasing function of S This work provides additional numerical insights by comparing the proposed approach with two other approaches The final result shows that utilizing the estimate and the uncertainty from dissimilar systems can be beneficial REFERENCES arXiv:2505.09057v1

eess.SY: 14 May 2025 1 Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta Fellow, IEEE Abstract—“Sim2real gap”, in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system.

Dev: In this work, we address this challenge in the linear quadratic regulator (LQR) setting Specifically, we consider an LQR problem for a system with unknown system matrices.

Rosa: Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained We establish that the algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps, where Mδ characterizes the dissimilarity between the two systems and f(S, Mδ) is a function of S and Mδ When Mδ is sufficiently small, the proposed algorithm achieves O˜(p T /S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control I.

Paper summary: Rosa: So this paper is about a problem called the sim-to-real gap in linear quadratic regulator control, and it looks at using offline data from a similar system to fix that gap when you don't know the true system matrices.

Dev: Exactly, it tackles the issue where controllers learned from simulations don't work well on the real system because of these unknown parameters. The main idea is they propose an online algorithm that uses this offline data to improve performance for controlling an unknown linear quadratic regulator.

Taro: It sounds like they're dealing with a situation where you have the actual system, but you have a trajectory of state-action pairs from a different, similar system available for training. How does that help the online control loop?

Rosa: Well, they frame this by saying the true system matrices A star and B star are different from some simulated matrices A sim and B sim. They then define an auxiliary system with equation three that represents this simulator, and they use the offline data D to characterize how different these two systems are.

Dev: It sets up a formal problem statement where you have the classical LQR design, which is minimizing a cost function J pi of theta star, but theta star is unknown. The core challenge is dealing with that uncertainty while trying to control the system using an online policy.

Taro: So they are not just trying to guess the parameters; they're using Thompson sampling, which samples a model from a belief distribution over those unknown system matrices to guide the online action selection.

Rosa: Right, and this sampling step involves augmenting your online data with information from the simulated system. They have an equation for sampling theta t that mixes the current estimate with information from the simulated system's uncertainty.

Dev: The key claim they make is that this approach achieves a Bayes regret of O f of S and M delta times T divided by S after T time steps, where S is the length of the offline data trajectory and M delta measures how different those two systems are.

Taro: That looks like a scaling factor involving the size of the offline data set and some measure of system mismatch, which is interesting because it suggests that having more similar offline data helps reduce that regret bound.

Rosa: And when that mismatch measure, M delta, is small enough, they find they get a better bound of O p T divided by S Bayes regret. That means if the systems are very similar, the performance scales nicely with the size of your offline dataset and time steps.

Paper summary: Dev: It also notes that when you have data from the same system available—meaning M delta is zero—the regret is bounded by O log T. That's a much better bound than what you get without any offline data at all, which usually involves square root of time or similar terms depending on the setup.

Taro: So, if you have perfect similarity between the simulated and real systems, the advantage of using that offline data is really just getting a logarithmic dependency on time steps. But what happens when they aren't perfectly similar?

Rosa: When they aren't perfectly similar, they still get a bound dependent on S and M delta, which is what the f of S and M delta term captures. They show numerically that as you increase the size of the offline trajectory, S, you can actually improve the regret with a fairly small number of measurements from your online system.

Dev: From an engineering standpoint, this means you can use a finite amount of offline data to get better guarantees than just relying on pure online exploration when dealing with these unknown system matrices. The safety case here is about controlling the error in that regret bound.

Taro: I wonder how robust this is when the world misbehaves unexpectedly during the online phase, since they are using a sample model to make decisions at each step based on that offline knowledge.

Rosa: They do use Thompson sampling to sample a parameter, compute the optimal control gain with that sample, and then transition to the next state and get feedback. The uncertainty from the simulated system is used as a prior for learning about the true system parameters.

Dev: It’s essentially using that knowledge of what's similar to help you make better guesses about what's different, which is how they characterize A star and B star by adding error terms from the simulator matrices.

Taro: So, the main implication for someone who only listens to this show is that when you face an unknown system and you have some data from a related system lying around, you can potentially achieve better long-term control performance bounds than if you were completely starting from scratch online.

Rosa: That's the gist of it. The paper, "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control," shows that using offline data to characterize the prior distribution for learning unknown system dynamics in an LQR setting can lead to a specific regret bound that depends on how similar the systems are and how much offline data you have.

Dev: It’s about bridging that sim-to-real gap by leveraging structural similarity in the system models. It's a method built on Thompson sampling that effectively uses the uncertainty from one system to inform your control strategy for another.

Conclusion: Rosa: So we've been looking at this paper from Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta, titled "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control."

Dev: It’s about how you can use data from a system that’s similar to the one you're actually trying to control when you don't know the exact system settings.

Rosa: Exactly. They tackle that sim-to-real gap in linear quadratic regulator control by using offline data from a different system to improve performance online.

Dev: The main takeaway is they prove an algorithm can achieve a certain regret bound, which means how much worse the online solution is compared to the absolute best possible solution if you knew everything upfront.

Rosa: They show that this regret scales in a way that depends on how much data you have and how different those two systems are.

Dev: The results suggest that having more similar offline data helps bring down that performance penalty, especially when the difference between the systems is small.

Rosa: And they find it gets even better bounds if the data comes from the exact same system, which is a pretty strong result for real-world applications.

Dev: It's about using structural similarity to get better control guarantees without needing infinite time to explore everything online.

Rosa: So, this moves us closer to having more reliable online controllers that can handle those tricky real-world mismatches.

Dev: But we still need to figure out the practical details of how this algorithm runs in a live loop with real latency.

Purdue University

eess.SY, cs.SY

Submitted: 2025-05-14

Updated: 2026-11-04

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 75/100

The gist: Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the "sim2real gap" by proposing an online algorithm that uses offline data from a similar system to improve…

Key concepts

Sim2real Gap
This refers to the performance loss that occurs when a controller trained using data from a simulated system is applied to a real system. The paper tackles this by using offline data from a similar simulator to inform the online learning process, aiming for better real-world control performance.
Thompson Sampling (TSOD-LQR)
This is the core online learning technique used. It works by sampling potential models (system matrices) from a belief distribution and taking an action based on that sample. It then updates this belief distribution using new online data, effectively balancing exploration and exploitation.
Bayes Regret
This metric measures the cumulative difference between the cost achieved by the learned controller over T time steps and the optimal cost achievable if all system parameters were known beforehand. A lower Bayes regret indicates better performance in controlling unknown LQR systems.

Terminology

Summary

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the sim2real gap by proposing an online algorithm that uses offline data from a similar system to improve performance when controlling an unknown linear quadratic regulator (LQR) The proposed algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps

Problem Formulation

The classical LQR design involves minimizing a cost function Jπ(θ∗) for a discrete-time linear time-invariant system with unknown matrices A∗ and B∗ The problem is framed by considering an auxiliary system characterized by equation (3) where the simulator is assumed to be different from the true system The offline data D represents a trajectory of length S of state-action pairs from this simulated system The performance is quantified by the cumulative Bayesian regret defined as R(T, π) = E [X T t=1 ct − J(θ∗)]

How the training signal is built

The algorithm utilizes Thompson Sampling (TSOD-LQR) which samples a model (system matrices) from a belief distribution over unknown system matrices The sampling equation for the parameter ˜θt is given by ˜θt = SQˆθt + βt(δ2)V−1/2 t ηt This step augments the online data with data collected from the simulated system to characterize the prior for learning θ∗ The algorithm updates its belief distribution using equations (9) and (10) to compute Vt and ˆθt

Where the gains come from

The regret analysis decomposes the expected regret into four terms R(T, TSOD-LQR)1[ET] ≤ R0 + R1 + R2 + R3 The term I is upper bounded by O˜(E[αS(δ1)]) Term II is upper bounded by O˜(E[p λmax(US)Mδ]) Term III is upper bounded by O˜(1) The final regret bound is established as O˜ r T S log(T) + E[αS(δ1) + p λmax(US)Mδ]

Generalization and Results

The framework can be extended to multiple offline sources by defining a new error metric e(θ) and updating the belief distribution accordingly By setting S = PN i=1 Si and Mδ = max Mδ,i, a similar upper bound on the cumulative regret can be obtained When offline data from the same system is available (Mδ = 0), the regret is bounded by O(log T) Furthermore, when data D is generated from Algorithm TSAC and θ sim∗ ∈ P, the regret is at most O˜ r T S log S + √SMδ!

Conclusion

In this work, an online control problem of an LQR when an offline trajectory of length S of state-action pairs from a similar linear system is available is considered The algorithm achieves O˜(f(S, Mδ)√T) regret, where f(S, Mδ) is a decreasing function of S This work provides additional numerical insights by comparing the proposed approach with two other approaches The final result shows that utilizing the estimate and the uncertainty from dissimilar systems can be beneficial

REFERENCES

arXiv:2505.09057v1 [eess. p. 0

Where the paper says it

arXiv:2505. 09057v1 [eess.

SY] 14 May 2025

1

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta Fellow, IEEE Abstract—“Sim2real gap”, in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting Specifically, we consider an LQR problem for a system with unknown system matrices. p. 0

Where the paper says it

In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices.

Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained We establish that the algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps, where Mδ characterizes the dissimilarity between the two systems and f(S, Mδ) is a function of S and Mδ When Mδ is sufficiently small, the proposed algorithm achieves O˜(p T /S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control

I. INTRODUCTION Online learning of linear quadratic regulators (LQRs) with unknown system matrices is a well-studied problem <ref:2505. p. 0

Where the paper says it

Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control I. INTRODUCTION Online learning of linear quadratic regulators (LQRs) with unknown system matrices is a well-studied problem.

09057#pg2> Many recent works have proposed novel algorithms with performance guarantees on their (cumulative) regret, defined as the difference between the cumulative cost with a controller from the learning algorithm and the cost with an optimal controller that knows the system matrices [1]–[4] However, these algorithms require long exploration times [5], which impedes their usage in many practical applications To aid these algorithms, we propose to use offline datasets of state-action pairs either from an approximate simulator or a simpler model of the unknown system We propose an online algorithm that leverages offline data to provably reduce the exploration time, leading to lower regret <ref:2505. p. 0
Where the paper says it

Many recent works have proposed novel algorithms with performance guarantees on their (cumulative) regret, defined as the difference between the cumulative cost with a controller from the learning algorithm and the cost with an optimal controller that knows the system matrices [1]–[4]. However, these algorithms require long exploration times [5], which impedes their usage in many practical applications.

09057#pg2> Leveraging offline data is not a new idea <ref:2505. p. 0
Where the paper says it

We propose an online algorithm that leverages offline data to provably reduce the exploration time, leading to lower regret. Leveraging offline data is not a new idea.

09057#pg2> Offline reinforcement learning [6], for instance, uses offline data to learn a policy which is used online However, this leads to the problem of sim-to-real gap or distribution shift since the system parameters learned offline are different from the ones encountered online [7] <ref:2505. p. 0
Where the paper says it

Offline reinforcement learning [6], for instance, uses offline data to learn a policy which is used online. However, this leads to the problem of sim-to-real gap or distribution shift since the system parameters learned offline are different from the ones encountered online [7].

09057#pg2> Although many methods have been proposed in the literature to be robust to such issues, in general, such policies are not optimal for the new system <ref:2505. p. 0
Where the paper says it

However, this leads to the problem of sim-to-real gap or distribution shift since the system parameters learned offline are different from the ones encountered online [7]. Although many methods have been proposed in the literature to be robust to such issues, in general, such policies are not optimal for the new system.

09057#pg2> Another approach is to utilize the offline data to warm-start an online learning algorithm <ref:2505. p. 0
Where the paper says it

Although many methods have been proposed in the literature to be robust to such issues, in general, such policies are not optimal for the new system. Another approach is to utilize the offline data to warm-start Shivam Bajaj and Vijay Gupta are with the Department of Electrical and Computer Engineering, Purdue University.

09057#pg2> Such strategies have been shown to achieve an improved bound on the regret in multi-armed bandits [8]–[11] However, extending these algorithms to LQR design and establishing their theoretical properties remains unexplored <ref:2505. p. 0
Where the paper says it

Such strategies have been shown to achieve an improved bound on the regret in multi-armed bandits [8]–[11]. However, extending these algorithms to LQR design and establishing their theoretical properties remains unexplored, particularly for characterizing when they provide benefits over learning the policy in a purely online fashion.

09057#pg2> Our algorithm provides a framework to incorporate offline data from a similar linear system for online learning, which provably achieves O˜(f(S, Mδ) p T /S) upper bound on the regret, where S denotes the offline trajectory length and Mδ quantifies the heterogeneity between the two systems <ref:2505. p. 0
Where the paper says it

Our algorithm provides a framework to incorporate offline data from a similar linear system1 for online learning, which provably achieves O˜(f(S, Mδ) p T /S) upper bound on the regret, where S denotes the offline trajectory length and Mδ quantifies the heterogeneity between the two systems. Our algorithm utilizes both the system matrices estimated from offline data and the residual uncertainty.

09057#pg2> Our algorithm utilizes both the system matrices estimated from offline data and the residual uncertainty We show via numerical simulations that as S increases, an improved regret can be achieved with a fairly small number of measurements from the online system Our algorithm uses Thompson Sampling (TS) which samples a model (system matrices) from a belief distribution over unknown system matrices, takes the optimal action based on the sample model, and subsequently updates the belief distribution using the observed feedback (cost) In the purely online setting, control of unknown linear dynamical systems using TS approach has been extensively studied [1], [12]–[14] Under the assumption that the distribution of the true parameters is known, [3] established a O˜(√T) (Bayes) regret bound Recently, the same O˜(√T) regret bound was established without that assumption [2], [15] Finally, our work is also related to the growing literature on transfer learning for linear systems [16]–[19] <ref:2505. p. 0
Where the paper says it

We show via numerical simulations that as S increases, an improved regret can be achieved with a fairly small number of measurements from the online system. Our algorithm uses Thompson Sampling (TS) which samples a model (system matrices) from a belief distribution over unknown system matrices, takes the optimal action based on the sample model, and subsequently updates the belief distribution using the observed feedback (cost).

09057#pg5> However, unlike that stream, this work focuses on determining regret guarantees on online LQR control while leveraging offline data <ref:2505. p. 0
Where the paper says it

Finally, our work is also related to the growing literature on transfer learning for linear systems [16]–[19]. However, unlike that stream, this work focuses on determining regret guarantees on online LQR control while leveraging offline data.

09057#pg5> This work is organized as follows. p. 0
Where the paper says it

However, unlike that stream, this work focuses on determining regret guarantees on online LQR control while leveraging offline data. This work is organized as follows.

Section II presents the problem definition and a summary of background material <ref:2505. p. 0
Where the paper says it

This work is organized as follows. Section II presents the problem definition and a summary of background material.

09057#pg5> Section III describes the offline data scheme <ref:2505. p. 0
Where the paper says it

Section II presents the problem definition and a summary of background material. Section III describes the offline data scheme.

09057#pg5> Section IV presents the proposed algorithm which is analyzed in Section V <ref:2505. p. 0
Where the paper says it

Section III describes the offline data scheme. Section IV presents the proposed algorithm which is analyzed in Section V.

09057#pg5> In Section VI, we present additional numerical insights and discuss how this work extends to when data from multiple sources is available <ref:2505. p. 0
Where the paper says it

Section IV presents the proposed algorithm which is analyzed in Section V. In Section VI, we present additional numerical insights and discuss how this work extends to when data from multiple sources is available.

09057#pg5> Notation:∥ · ∥,∥ · ∥F,∥ · ∥2, and Tr(·) denotes the operator norm, Frobenius norm, spectral norm, and the trace respectively <ref:2505. p. 0
Where the paper says it

Finally, Section VIII summarizes this work and outlines directions for future work. Notation: ∥ · ∥, ∥ · ∥F , ∥ · ∥2, and Tr(·) denotes the operator norm, Frobenius norm, spectral norm, and the trace, respectively.

09057#pg5> For a positive definite matrix A (denoted as A≻ 0), λmax(A) and λmin(A) denote its maximum and minimum eigenvalue respectively <ref:2505. p. 0
Where the paper says it

Notation: ∥ · ∥, ∥ · ∥F , ∥ · ∥2, and Tr(·) denotes the operator norm, Frobenius norm, spectral norm, and the trace, respectively. For a positive definite matrix A (denoted as A ≻ 0), λmax(A) and λmin(A) denote its maximum and minimum eigenvalue, respectively.

09057#pg5> I denotes the identity matrix and η1Two linear systems, characterized by system matrices Ai and Bi, i ∈

A1 B1 − A2 B2 ≤ Mδ.

II. PROBLEM FORMULATION We first review the classical LQR control problem and then describe our model followed by the formal problem statement <ref:2505. p. 1

Where the paper says it

Given a set P and a sample θ, SP represents a sampling operator that ensures θ ∈ P. PROBLEM FORMULATION We first review the classical LQR control problem and then describe our model followed by the formal problem statement.

09057#pg5> A. Classical LQR Design Let xt ∈ R n denote the state and ut ∈ R m denote the control at time t <ref:2505. p. 1
Where the paper says it

PROBLEM FORMULATION We first review the classical LQR control problem and then describe our model followed by the formal problem statement. Classical LQR Design Let xt ∈ R n denote the state and ut ∈ R m denote the control at time t.

09057#pg5> Let A∗ ∈ R n×n and B∗ ∈ R n×m be the system matrices <ref:2505. p. 1
Where the paper says it

Classical LQR Design Let xt ∈ R n denote the state and ut ∈ R m denote the control at time t. Let A∗ ∈ R n×n and B∗ ∈ R n×m be the system matrices.

09057#pg5> Further, let θ⊤∗:= A∗ B∗ and zt:= h x⊤ t u⊤ t i⊤ <ref:2505. p. 0
Where the paper says it

arXiv:2505. 09057v1 [eess.

09057#pg5> Then, for t ≥ 1 and given matrices Q≻ 0, R≻ 0, consider a discrete-time linear time-invariant system with the dynamics and the cost function xt+1 = θ⊤∗ zt + wt, (1) <ref:2505. p. 1
Where the paper says it

Further, let θ ⊤ ∗ := A∗ B∗ and zt := h x ⊤ t u ⊤ t i⊤ . Then, for t ≥ 1 and given matrices Q ≻ 0, R ≻ 0, consider a discrete-time linear time-invariant system with the dynamics and the cost function xt+1 = θ ⊤ ∗ zt + wt, (1) ct = x ⊤ t Qxt + u ⊤ t Rut, where wt ∼ N (0, I) is the system noise assumed to be white and x1 = 0.

09057#pg5> ct = x⊤ t Qxt + u⊤ t Rut, where wt ∼ N (0, I) is the system noise assumed to be white and x1 = 0 <ref:2505. p. 1
Where the paper says it

Further, let θ ⊤ ∗ := A∗ B∗ and zt := h x ⊤ t u ⊤ t i⊤ . Then, for t ≥ 1 and given matrices Q ≻ 0, R ≻ 0, consider a discrete-time linear time-invariant system with the dynamics and the cost function xt+1 = θ ⊤ ∗ zt + wt, (1) ct = x ⊤ t Qxt + u ⊤ t Rut, where wt ∼ N (0, I) is the system noise assumed to be white and x1 = 0.

09057#pg5> The classical LQR control problem is to design a closed-loop control π: R n → R m with ut = π(xt) that minimizes the following cost: Jπ(θ∗) = lim T→∞ sup 1 T X T t=1 E[ct(xt, ut)] (2) When θ∗ is known and under the assumption that (A∗, B∗) is stabilizable, the optimal policy is ut = K(θ∗)xt and the corresponding cost is J(θ∗):= Tr(P(θ∗)) where K(θ∗) = −(R + B⊤ ∗ P(θ∗)Bstar−1B⊤ ∗ Pstar Astar)−1B⊤ ∗ Pstar Astar is the gain matrix and P(θ∗) is the unique positive definite solution to the Riccati equation <ref:2505. p. 1
Where the paper says it

The classical LQR control problem is to design a closed-loop control π : R n → R m with ut = π(xt) that minimizes the following cost: Jπ(θ∗) = lim T→∞ sup 1 T X T t=1 E[ct(xt, ut)]. (2) When θ∗ is known and under the assumption that (A∗, B∗) is stabilizable, the optimal policy is ut = K(θ∗)xt and the corresponding cost is J(θ∗) := Tr(P(θ∗)) where K(θ∗) = −(R + B ⊤ ∗ P(θ∗)B∗) −1B ⊤ ∗ P(θ∗)A∗ is the gain matrix and P(θ∗) is the unique positive definite solution to the Riccati equation P(θ∗) = Q + A ⊤ ∗ P(θ∗)A∗ + A ⊤ ∗ P(θ∗)B∗K(θ∗).

09057#pg5>

B. Model and Problem Statement Consider a system characterized by equation (1) with unknown θ∗ and access to an offline dataset obtained through an approximated simulator <ref:2505. p. 1

Where the paper says it

(2) When θ∗ is known and under the assumption that (A∗, B∗) is stabilizable, the optimal policy is ut = K(θ∗)xt and the corresponding cost is J(θ∗) := Tr(P(θ∗)) where K(θ∗) = −(R + B ⊤ ∗ P(θ∗)B∗) −1B ⊤ ∗ P(θ∗)A∗ is the gain matrix and P(θ∗) is the unique positive definite solution to the Riccati equation P(θ∗) = Q + A ⊤ ∗ P(θ∗)A∗ + A ⊤ ∗ P(θ∗)B∗K(θ∗). Model and Problem Statement Consider a system characterized by equation (1) with unknown θ∗ and access to an offline dataset obtained through an approximated simulator.

09057#pg5> The simulator is assumed to be characterized by the following auxiliary system which is different than θ∗ and is also unknown <ref:2505. p. 1
Where the paper says it

Model and Problem Statement Consider a system characterized by equation (1) with unknown θ∗ and access to an offline dataset obtained through an approximated simulator. The simulator is assumed to be characterized by the following auxiliary system which is different than θ∗ and is also unknown.

09057#pg5> ξs+1 = θsim∗⊤ ys + wsim s, (3) where, ξs ∈ R n and vs ∈ R m denotes the state and the control, respectively, at time instant s, θsim∗⊤:= h Asim∗ Bsim∗ i denotes the system matrices <ref:2505. p. 1
Where the paper says it

The simulator is assumed to be characterized by the following auxiliary system which is different than θ∗ and is also unknown. ξs+1 = θ sim ∗ ⊤ ys + w sim s , (3) where, ξs ∈ R n and vs ∈ R m denotes the state and the control, respectively, at time instant s, θ sim ∗ ⊤ := h Asim ∗ Bsim ∗ i denotes the system matrices, and ys := h ξ ⊤ s v ⊤ s i⊤ .

09057#pg5> The offline data D =

y1,..., yS represents a trajectory of length S of state-action pairs (ξs, vs), 1 ≤ s ≤ S <ref:2505. p. 1

Where the paper says it

The offline data D = {y1, . , yS} represents a trajectory of length S of state-action pairs (ξs, vs), 1 ≤ s ≤ S.

09057#pg5> We can characterize A∗ and B∗ as A∗ = Asim∗ + Aδ∗ and B∗ = Bsim∗ + Bδ, respectively, where Aδ (resp. p. 1
Where the paper says it

, yS} represents a trajectory of length S of state-action pairs (ξs, vs), 1 ≤ s ≤ S. We can characterize A∗ and B∗ as A∗ = Asim ∗ + Aδ ∗ and B∗ = Bsim ∗ + Bδ ∗ , respectively, where Aδ ∗ (resp.

Bδ) represents the change in the system matrices A∗ (resp. p. 1
Where the paper says it

We can characterize A∗ and B∗ as A∗ = Asim ∗ + Aδ ∗ and B∗ = Bsim ∗ + Bδ ∗ , respectively, where Aδ ∗ (resp. Bδ ∗ ) represents the change in the system matrices A∗ (resp.

Bstar) from Asimstar (resp. p. 1
Where the paper says it

, yS} represents a trajectory of length S of state-action pairs (ξs, vs), 1 ≤ s ≤ S. We can characterize A∗ and B∗ as A∗ = Asim ∗ + Aδ ∗ and B∗ = Bsim ∗ + Bδ ∗ , respectively, where Aδ ∗ (resp.

Bsimstar) <ref:2505. p. 0
Where the paper says it

arXiv:2505. 09057v1 [eess.

09057#pg5> Thus, the system characterized by equation (1) can be expressed as xt+1 = Asim∗ + Aδ xt + Bsim∗ + Bδ ut + wt. p. 1
Where the paper says it

Thus, the system characterized by equation (1) can be expressed as xt+1 = A sim ∗ + A δ ∗ xt + B sim ∗ + B δ ∗ ut + wt. (4) In this work, we assume that there exists a known constant Mδ such that θ δ ∗ F ≤ Mδ, where θ δ ∗ ⊤ := h Aδ ∗ Bδ ∗ i .

(4) In this work, we assume that there exists a known constant Mδ such that θδ∗ F ≤ Mδ, where θδ∗⊤:= h A δ B δ i. p. 1
Where the paper says it

Thus, the system characterized by equation (1) can be expressed as xt+1 = A sim ∗ + A δ ∗ xt + B sim ∗ + B δ ∗ ut + wt. (4) In this work, we assume that there exists a known constant Mδ such that θ δ ∗ F ≤ Mδ, where θ δ ∗ ⊤ := h Aδ ∗ Bδ ∗ i .

Let Ft:= σ(x1, u1,. p. 1
Where the paper says it

PROBLEM FORMULATION We first review the classical LQR control problem and then describe our model followed by the formal problem statement. Classical LQR Design Let xt ∈ R n denote the state and ut ∈ R m denote the control at time t.

.., xt, ut) denote the filtration that represents the knowledge up to time t during the online process <ref:2505. p. 1
Where the paper says it

Let Ft := σ({x1, u1, . , xt, ut}) denote the filtration that represents the knowledge up to time t during the online process.

09057#pg5> Similarly, let Fs:= σ(ξ1, v1,. p. 1
Where the paper says it

, xt, ut}) denote the filtration that represents the knowledge up to time t during the online process. Similarly, let Fs := σ({ξ1, v1, .

.., ξS, vS) denote the filtration that represents the knowledge corresponding to the offline data <ref:2505. p. 1
Where the paper says it

Similarly, let Fs := σ({ξ1, v1, . , ξS, vS}) denote the filtration that represents the knowledge corresponding to the offline data.

09057#pg5> Then we make the following standard assumption on the noise process [20] Assumption 1. p. 1
Where the paper says it

, ξS, vS}) denote the filtration that represents the knowledge corresponding to the offline data. Then, we make the following standard assumption on the noise process [20].

There exists a filtration Ft and Fs such that for any t ≥ 1 and 1 ≤ s ≤ S, zt, xt are Ft∪ Fs-measurable and ys, ξs are Fs-measurable <ref:2505. p. 1
Where the paper says it

Assumption 1. There exists a filtration Ft and Fs such that for any t ≥ 1 and 1 ≤ s ≤ S, zt, xt are Ft ∪ Fs-measurable and ys, ξs are Fs-measurable.

09057#pg5> Further, wt+1 and wsim s+1 are individually martingale difference sequences. p. 1
Where the paper says it

There exists a filtration Ft and Fs such that for any t ≥ 1 and 1 ≤ s ≤ S, zt, xt are Ft ∪ Fs-measurable and ys, ξs are Fs-measurable. Further, wt+1 and w sim s+1 are individually martingale difference sequences.

Finally, for ease of exposition, we assume that E[wt+1w⊤ t+1Ft ∪ Fs] = I and E[w sim s+1w sim s+1⊤Fs] = I <ref:2505. p. 1
Where the paper says it

Further, wt+1 and w sim s+1 are individually martingale difference sequences. Finally, for ease of exposition, we assume that E[wt+1w ⊤ t+1|Ft ∪ Fs] = I and E[w sim s+1w sim s+1 ⊤ |Fs] = I.

09057#pg5> Assuming that the parameter θ∗ is a random variable with a known distribution µ, we quantify the performance of our learning algorithm by comparing the cumulative cost to the infinite-horizon cost attained by the LQR controller if the system matrices defined by θ∗ were known a priori <ref:2505. p. 1
Where the paper says it

Finally, for ease of exposition, we assume that E[wt+1w ⊤ t+1|Ft ∪ Fs] = I and E[w sim s+1w sim s+1 ⊤ |Fs] = I. Assuming that the parameter θ∗ is a random variable with a known distribution µ, we quantify the performance of our learning algorithm by comparing the cumulative cost to the infinite-horizon cost attained by the LQR controller if the system matrices defined by θ∗ were known a priori.

09057#pg5> Formally, we quantify the performance of our algorithm through the cumulative Bayesian regret defined as R(T, π) = E [X T t=1 ct − J(θ∗)], (5) where the expectation is with respect to wt, µ, and any randomization in the algorithms used to process the offline and online data <ref:2505. p. 1
Where the paper says it

Formally, we quantify the performance of our algorithm through the cumulative Bayesian regret defined as follows. R(T, π) = E   X T t=1 ct − J(θ∗)   , (5) where the expectation is with respect to wt, µ, and any randomization in the algorithms used to process the offline and online data.

09057#pg5> This metric has been previously considered for online control of LQR systems [3] <ref:2505. p. 1
Where the paper says it

R(T, π) = E   X T t=1 ct − J(θ∗)   , (5) where the expectation is with respect to wt, µ, and any randomization in the algorithms used to process the offline and online data. This metric has been previously considered for online control of LQR systems [3].

09057#pg5> Problem 1. p. 0
Where the paper says it

In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices.

The aim of this work is to find a control algorithm that minimizes the expected regret defined in (5) while utilizing the offline data D <ref:2505. p. 1
Where the paper says it

This metric has been previously considered for online control of LQR systems [3]. The aim of this work is to find a control algorithm that minimizes the expected regret defined in (5) while utilizing the offline data D.

09057#pg5>

III. OFFLINE DATA-GENERATION In this work, we do not consider a particular algorithm from which the offline data is generated <ref:2505. p. 1

Where the paper says it

The aim of this work is to find a control algorithm that minimizes the expected regret defined in (5) while utilizing the offline data D. OFFLINE DATA-GENERATION In this work, we do not consider a particular algorithm from which the offline data is generated.

09057#pg5> As we will see later, any algorithm that satisfies the following two properties can be used to generate the offline dataset D <ref:2505. p. 1
Where the paper says it

OFFLINE DATA-GENERATION In this work, we do not consider a particular algorithm from which the offline data is generated. As we will see later, any algorithm that satisfies the following two properties can be used to generate the offline dataset D.

09057#pg5> Let Asim denote an algorithm that is used to generate the offline data <ref:2505. p. 1
Where the paper says it

As we will see later, any algorithm that satisfies the following two properties can be used to generate the offline dataset D. Let Asim denote an algorithm that is used to generate the offline data.

09057#pg5> Further, let at time s, Us:= Ps k=0 ysy⊤ s denote the precision matrix of Algorithm Asim <ref:2505. p. 1
Where the paper says it

Let Asim denote an algorithm that is used to generate the offline data. Further, let at time s, Us := Ps k=0 ysy ⊤ s denote the precision matrix of Algorithm Asim.

09057#pg5> We assume the following on algorithm Asim. p. 1
Where the paper says it

Further, let at time s, Us := Ps k=0 ysy ⊤ s denote the precision matrix of Algorithm Asim. We assume the following on algorithm Asim.

Assumption 2 (Offline Algorithm). p. 1
Where the paper says it

We assume the following on algorithm Asim. Assumption 2 (Offline Algorithm).

For a given δ1 ∈ (0, 1), with probability of at least 1 − δ1, Algorithm Asim satisfies 1)∥U0. p. 1
Where the paper says it

Assumption 2 (Offline Algorithm). For a given δ1 ∈ (0, 1), with probability of at least 1 − δ1, Algorithm Asim satisfies 1) ∥U 0.

5 s(ˆθsim s − θsim∗)∥F ≤ αs(δ1) <ref:2505. p. 0
Where the paper says it

arXiv:2505. 09057v1 [eess.

09057#pg5> 2) For s ≥ 200(n + m) log 12 δ1, λmin(Us) ≥ s40 <ref:2505. p. 1
Where the paper says it

5 s ( ˆθ sim s − θ sim ∗ )∥F ≤ αs(δ1). 2) For s ≥ 200(n + m) log 12 δ1 , λmin(Us) ≥ s 40 .

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Leverage offline data from similar systems for online linear quadratic control to reduce exploration time, leading to lower regret in LQR problems with unknown system matrices. This allows an online learning algorithm to perform better than purely online methods by incorporating knowledge from a simulator or simpler model of the unknown system.

  2. Implement Thompson Sampling with Learned Predictions (TSOD-LQR) as the proposed algorithm which samples a model from a belief distribution over unknown system matrices and utilizes both the mean and uncertainty of the dynamics of the system from which offline data is obtained. This approach achieves O(f(S, Mdelta) p T /S) Bayes regret after T time steps, where S denotes the offline trajectory length.

  3. Design an improved control algorithm that incorporates data from multiple dissimilar offline sources by defining a new l2-least squares error and updating the belief distribution using aggregated information from all available trajectories. This generalization allows for online control of LQR when offline data from multiple systems is available, with a similar upper bound on cumulative regret.

Abstract

``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained. We establish that the algorithm achieves O tilde(f(S,M δ) sqrt T/S) Bayes regret after T time steps, where M δ characterizes the dissimilarity between the two systems and f(S,M δ) is a function of S and M δ. When M δ is sufficiently small, the proposed algorithm achieves O tilde(sqrt T/S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.

Sources

Related papers