Data-driven Linear Quadratic Integral Control: A Convex Formulation and Policy Gradient Approach

summary

The gist

This paper studies the data-driven synthesis of linear quadratic integral (LQI) controllers for continuous-time systems to achieve optimal state-feedback control with integral action for reference…

In short

This work develops a method to create optimal Linear Quadratic Integral (LQI) controllers for continuous-time systems using only measured data. It achieves this by creating a data-driven representation of the augmented system dynamics, formulating a convex optimization problem to find the best feedback gain without knowing system matrices, and using a policy gradient flow to compute the controller. This allows for optimal tracking control that adapts to changing conditions.

Key concepts

LQI Controller
An LQI controller is used for continuous-time systems that combines standard state feedback with an integral action. The integral action accumulates the error between the system's output and a desired reference signal, ensuring the system eventually tracks that reference perfectly. This combination provides both fast response and zero steady-state error.
Data-Driven Parameterization
This involves using measured input, state, and output data to create a mathematical model of the closed-loop system dynamics. Instead of needing exact system matrices beforehand, this method uses sampled data (like $y(0), y( riangle), ext{etc.}$) to define equivalent representations that describe how the system behaves based on what we have actually measured.
Convex Optimization
The paper sets up a mathematical problem that is convex, meaning it has a single best solution that can be found efficiently. This optimization problem is used to determine the optimal LQR feedback gain ($K^*$) for the augmented system. Because it's convex, we can solve it reliably using standard computational techniques.
Policy Gradient Flow
This is a technique used to compute the actual controller within a set of possible stabilizing gains. It starts with an initial guess and iteratively adjusts the gain in a direction that improves performance, guided by the optimization problem. This flow converges to the optimal control law, allowing for dynamic adaptation based on real-time system behavior.

Terminology used across episodes

This episode discusses

The paper

Data-driven Linear Quadratic Integral Control: A Convex Formulation and Policy Gradient Approach · Read on arXiv

Karlsruhe Institute of Technology (KIT) · ABB Corporate Research Center

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Data-driven Linear Quadratic Integral Control".

Dev: Here's the gist of the paper. This paper studies the data-driven synthesis of linear quadratic integral (LQI) controllers for continuous-time systems to achieve optimal state-feedback control with integral action for reference tracking using only measured data.

Rosa: Core Methodology The objective is to derive a data-driven closedloop parameterization of the augmented dynamics that incorporates the integral state while relying solely on input-stateoutput measurements.

Dev: Based on this parameterization, a data-driven convex optimization problem is formulated whose solution yields the optimal linear quadratic regulator (LQR) feedback gain for the augmented system without explicit knowledge of the system matrices.

Rosa: In addition, a policy gradient flow is derived to compute the optimal controller within the space of stabilizing gains.

Title and authors: Rosa: So this paper is titled "Data-driven Linear Quadratic Integral Control: A Convex Formulation and Policy Gradient Approach." It's about making these linear quadratic integral controllers for continuous-time systems using only measured data.

Dev: That sounds like they're tackling the problem of getting optimal state feedback with integral action, which is usually hard without knowing the system matrices beforehand.

Rosa: Exactly. The core idea here is to get that optimal control using just input, state, and output measurements. It’s about synthesizing a controller without needing explicit knowledge of the underlying system dynamics for that part of the calculation.

Taro: So if I'm hearing this right, we're trying to build a feedback mechanism that has integral action for tracking but doesn't require us to model every single parameter of the plant upfront?

Dev: That’s right. They propose a way to do that by first creating a data-driven closed-loop parameterization of the augmented system, which includes that integral state.

The paper's summary: Rosa: So what they're actually doing is deriving this representation from measured input, state, and output trajectories. They set up a convex optimization problem based on that representation to find the optimal linear quadratic regulator feedback gain.

Dev: That’s the crucial part for me—using a convex optimization framework so we don't get stuck with something non-convex where finding the solution is tough. And they solve it without knowing A or B explicitly, which is a big deal.

Rosa: They also introduce this policy gradient flow to compute the optimal controller within the set of stabilizing gains. So it’s not just a static gain; it's something that can adapt based on what the data shows them during operation.

Taro: I wonder how robust this adaptive part is when things get messy, like if there's some noise or if the system starts behaving unexpectedly outside the tested conditions?

Dev: The paper shows it works by validating it on a distributed generation unit in a DC microgrid. They found that the feedback gain K converges to an optimal value of zero point four zero nine, which is pretty specific for that setup.

The paper's improvements: Rosa: One thing they highlight as an improvement is how they handle the parameterization itself. They use a data-driven closed-loop parameterization based on sampled covariance, and the resulting data matrices don't actually depend on the sample size T.

Dev: That’s significant because it means we don't have to keep collecting more and more samples just to make the representation accurate for a bigger system. The structure of those data matrices is independent of how long you measure, as long as you have enough measurements.

Rosa: And then they show how the optimal LQR feedback gain is recovered through this convex program. They find that the optimal linear quadratic regulator feedback gain K equals U times Z minus one, where Z and W are matrices derived from minimizing that program based on the data matrices.

Taro: So, it’s not just finding *a* controller; they show a way to find the *optimal* one by framing it as this constrained optimization problem over the data. Does that mean we can actually get a better result than just guessing?

Dev: It means mathematically, yes. The formulation allows them to compute that optimal LQR gain for the augmented system directly from those measured data matrices without having to solve the original state-space equations analytically first.

Conclusion: Rosa: So to wrap up, this paper introduces a data-driven approach for synthesizing linear quadratic integral controllers for continuous-time systems using only input, state, and output measurements. They achieve optimal state feedback control with integral action for reference tracking just by relying on measured data.

Dev: The main contribution is deriving the convex optimization problem that lets you compute the optimal LQR feedback gain without needing explicit knowledge of the system matrices, and then using a policy gradient flow to compute that optimal controller within the space of stabilizing gains.

Taro: For someone listening who just cares about real-world application, this means you can build a tracking controller for something like a microgrid or some other continuous system where you don't have perfect models—you just feed it the sensor readings and it figures out how to control it optimally.

Rosa: It’s moving toward systems that adapt their control law based on what they are actually measuring in real time. We’re looking at a data-driven approach for LQI controllers, and this paper shows how to frame that as a convex optimization problem solvable with policy gradient flows.

Dev: The implication is that we can move away from relying entirely on perfect system identification for control synthesis and instead use data collection to guide the optimization process itself.

Taro: I’m just thinking about the next steps, though, because they mention limitations. They say they didn't fully address scenarios with process and measurement noise or time-varying system dynamics yet. That’s where the real test will be for any practical deployment.

Rosa: Right now, it’s a proof of concept on a DGU in a DC microgrid, showing convergence to that optimal gain of zero point four zero nine for that example. It proves the concept works under certain conditions.

Dev: So the next paper we'll be looking at might be how this handles those noise issues you mentioned, because if it can’t handle noise gracefully, it doesn't really help us in a noisy power grid environment.

More episodes

← Home