Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control

summary

The gist

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the "sim2real gap" by proposing an online algorithm that uses offline data from a similar system to improve…

In short

The paper addresses the 'sim2real gap' in online Linear Quadratic Regulator (LQR) control by using offline data from a similar, simulated system. It proposes an online algorithm based on Thompson Sampling that leverages this offline data to improve performance when the true system matrices are unknown, achieving a regret bound of O(f(S, Mδ) p T /S). This method helps bridge the gap between simulation and real-world control.

Key concepts

Sim2real Gap
This refers to the performance loss that occurs when a controller trained using data from a simulated system is applied to a real system. The paper tackles this by using offline data from a similar simulator to inform the online learning process, aiming for better real-world control performance.
Thompson Sampling (TSOD-LQR)
This is the core online learning technique used. It works by sampling potential models (system matrices) from a belief distribution and taking an action based on that sample. It then updates this belief distribution using new online data, effectively balancing exploration and exploitation.
Bayes Regret
This metric measures the cumulative difference between the cost achieved by the learned controller over T time steps and the optimal cost achievable if all system parameters were known beforehand. A lower Bayes regret indicates better performance in controlling unknown LQR systems.

Terminology used across episodes

This episode discusses

The paper

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control · Read on arXiv

Purdue University

``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained. We establish that the algorithm achieves O tilde(f(S,M δ) sqrt T/S) Bayes regret after T time steps, where M δ characterizes the dissimilarity between the two systems and f(S,M δ) is a function of S and M δ. When M δ is sufficiently small, the proposed algorithm achieves O tilde(sqrt T/S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control".

Dev: Here's the gist of the paper. Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control addresses the "sim2real gap" by proposing an online algorithm that uses offline data from a similar system to improve performance when controlling an unknown linear quadratic regulator (LQR) The proposed algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps Problem Formulation The classical LQR design involves minimizing a cost function Jπ(θ∗) for a discrete-time linear time-invariant system with unknown matrices A∗ and B∗ The problem is framed by considering an auxiliary system characterized by equation (3) where the simulator is assumed to be different from the true system The offline data D represents a trajectory of length S of state-action pairs from this simulated system The performance is quantified by the cumulative Bayesian regret defined as R(T, π) = E

X T t=1 ct − J(θ∗): How the training signal is built The algorithm utilizes Thompson Sampling (TSOD-LQR) which samples a model (system matrices) from a belief distribution over unknown system matrices The sampling equation for the parameter ˜θt is given by ˜θt = SQˆθt + βt(δ2)V−1/2 t ηt This step augments the online data with data collected from the simulated system to characterize the prior for learning θ∗ The algorithm updates its belief distribution using equations (9) and (10) to compute Vt and ˆθt Where the gains come from The regret analysis decomposes the expected regret into four terms R(T, TSOD-LQR)1

ET: ≤ R0 + R1 + R2 + R3 The term I is upper bounded by O˜(E

αS(δ1): ) Term II is upper bounded by O˜(E

p λmax(US)Mδ: ) Term III is upper bounded by O˜(1) The final regret bound is established as O˜ r T S log(T) + E

αS(δ1) + p λmax(US)Mδ: Generalization and Results The framework can be extended to multiple offline sources by defining a new error metric e(θ) and updating the belief distribution accordingly By setting S = PN i=1 Si and Mδ = max Mδ,i, a similar upper bound on the cumulative regret can be obtained When offline data from the same system is available (Mδ = 0), the regret is bounded by O(log T) Furthermore, when data D is generated from Algorithm TSAC and θ sim∗ ∈ P, the regret is at most O˜ r T S log S + √SMδ!

Rosa: In this work, an online control problem of an LQR when an offline trajectory of length S of state-action pairs from a similar linear system is available is considered The algorithm achieves O˜(f(S, Mδ)√T) regret, where f(S, Mδ) is a decreasing function of S This work provides additional numerical insights by comparing the proposed approach with two other approaches The final result shows that utilizing the estimate and the uncertainty from dissimilar systems can be beneficial REFERENCES arXiv:2505.09057v1

eess.SY: 14 May 2025 1 Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta Fellow, IEEE Abstract—“Sim2real gap”, in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system.

Dev: In this work, we address this challenge in the linear quadratic regulator (LQR) setting Specifically, we consider an LQR problem for a system with unknown system matrices.

Rosa: Along with the state-action pairs from the system to be controlled, a trajectory of length S of state-action pairs from a different unknown system is available Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length S is obtained We establish that the algorithm achieves O˜(f(S, Mδ) p T /S) Bayes regret after T time steps, where Mδ characterizes the dissimilarity between the two systems and f(S, Mδ) is a function of S and Mδ When Mδ is sufficiently small, the proposed algorithm achieves O˜(p T /S) Bayes regret and outperforms a naive strategy which does not utilize the available trajectory Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control Index Terms—Identification for control, Sampled Data Control, Autonomous Systems, Adaptive Control I.

Paper summary: Rosa: So this paper is about a problem called the sim-to-real gap in linear quadratic regulator control, and it looks at using offline data from a similar system to fix that gap when you don't know the true system matrices.

Dev: Exactly, it tackles the issue where controllers learned from simulations don't work well on the real system because of these unknown parameters. The main idea is they propose an online algorithm that uses this offline data to improve performance for controlling an unknown linear quadratic regulator.

Taro: It sounds like they're dealing with a situation where you have the actual system, but you have a trajectory of state-action pairs from a different, similar system available for training. How does that help the online control loop?

Rosa: Well, they frame this by saying the true system matrices A star and B star are different from some simulated matrices A sim and B sim. They then define an auxiliary system with equation three that represents this simulator, and they use the offline data D to characterize how different these two systems are.

Dev: It sets up a formal problem statement where you have the classical LQR design, which is minimizing a cost function J pi of theta star, but theta star is unknown. The core challenge is dealing with that uncertainty while trying to control the system using an online policy.

Taro: So they are not just trying to guess the parameters; they're using Thompson sampling, which samples a model from a belief distribution over those unknown system matrices to guide the online action selection.

Rosa: Right, and this sampling step involves augmenting your online data with information from the simulated system. They have an equation for sampling theta t that mixes the current estimate with information from the simulated system's uncertainty.

Dev: The key claim they make is that this approach achieves a Bayes regret of O f of S and M delta times T divided by S after T time steps, where S is the length of the offline data trajectory and M delta measures how different those two systems are.

Taro: That looks like a scaling factor involving the size of the offline data set and some measure of system mismatch, which is interesting because it suggests that having more similar offline data helps reduce that regret bound.

Rosa: And when that mismatch measure, M delta, is small enough, they find they get a better bound of O p T divided by S Bayes regret. That means if the systems are very similar, the performance scales nicely with the size of your offline dataset and time steps.

Paper summary: Dev: It also notes that when you have data from the same system available—meaning M delta is zero—the regret is bounded by O log T. That's a much better bound than what you get without any offline data at all, which usually involves square root of time or similar terms depending on the setup.

Taro: So, if you have perfect similarity between the simulated and real systems, the advantage of using that offline data is really just getting a logarithmic dependency on time steps. But what happens when they aren't perfectly similar?

Rosa: When they aren't perfectly similar, they still get a bound dependent on S and M delta, which is what the f of S and M delta term captures. They show numerically that as you increase the size of the offline trajectory, S, you can actually improve the regret with a fairly small number of measurements from your online system.

Dev: From an engineering standpoint, this means you can use a finite amount of offline data to get better guarantees than just relying on pure online exploration when dealing with these unknown system matrices. The safety case here is about controlling the error in that regret bound.

Taro: I wonder how robust this is when the world misbehaves unexpectedly during the online phase, since they are using a sample model to make decisions at each step based on that offline knowledge.

Rosa: They do use Thompson sampling to sample a parameter, compute the optimal control gain with that sample, and then transition to the next state and get feedback. The uncertainty from the simulated system is used as a prior for learning about the true system parameters.

Dev: It’s essentially using that knowledge of what's similar to help you make better guesses about what's different, which is how they characterize A star and B star by adding error terms from the simulator matrices.

Taro: So, the main implication for someone who only listens to this show is that when you face an unknown system and you have some data from a related system lying around, you can potentially achieve better long-term control performance bounds than if you were completely starting from scratch online.

Rosa: That's the gist of it. The paper, "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control," shows that using offline data to characterize the prior distribution for learning unknown system dynamics in an LQR setting can lead to a specific regret bound that depends on how similar the systems are and how much offline data you have.

Dev: It’s about bridging that sim-to-real gap by leveraging structural similarity in the system models. It's a method built on Thompson sampling that effectively uses the uncertainty from one system to inform your control strategy for another.

Conclusion: Rosa: So we've been looking at this paper from Shivam Bajaj, Prateek Jaiswal, and Vijay Gupta, titled "Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control."

Dev: It’s about how you can use data from a system that’s similar to the one you're actually trying to control when you don't know the exact system settings.

Rosa: Exactly. They tackle that sim-to-real gap in linear quadratic regulator control by using offline data from a different system to improve performance online.

Dev: The main takeaway is they prove an algorithm can achieve a certain regret bound, which means how much worse the online solution is compared to the absolute best possible solution if you knew everything upfront.

Rosa: They show that this regret scales in a way that depends on how much data you have and how different those two systems are.

Dev: The results suggest that having more similar offline data helps bring down that performance penalty, especially when the difference between the systems is small.

Rosa: And they find it gets even better bounds if the data comes from the exact same system, which is a pretty strong result for real-world applications.

Dev: It's about using structural similarity to get better control guarantees without needing infinite time to explore everything online.

Rosa: So, this moves us closer to having more reliable online controllers that can handle those tricky real-world mismatches.

Dev: But we still need to figure out the practical details of how this algorithm runs in a live loop with real latency.

More episodes

← Home