Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Mathematics. The day in mathematics: new theorems, new proofs, and what they open.
Ada: I'm Ada.
Emmy: And I'm Emmy, here with Kolya.
Ada: We're looking at the paper: "Policy Optimization in the Linear Quadratic Gaussian Problem".
Emmy: Here's the gist of the paper. The paper addresses the challenges in policy optimization for Linear Quadratic Gaussian (LQG) control by deriving a necessary and sufficient condition for global optimality and developing a gradient-based algorithm in an infinite-dimensional space, which provides rigorous guarantees.
Ada: Theoretical Insights on Optimality The authors derive a necessary and sufficient condition for the global optimality of stationary points in parameterized LQG problems, which reduces verification to a test of controllability and observability for a novel transfer function.
Emmy: This condition provides a rigorous explanation for why traditional parameterizations can lead to suboptimal stationary points by showing that direct parameterization acts as a projection that destroys the benign structure of the objective functional.
Ada: Algorithmic Developments in Infinite Dimensions The work elevates the controller parameter space from finite-dimensional settings to the infinite-dimensional RH∞ space and develops a gradient-based algorithm for this setting, establishing a theoretical analysis that proves global convergence.
Paper summary: Ada: We're starting by looking at "Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective." The paper addresses two major gaps: first, there is currently no general method to certify the global optimality of candidate controller parameters in parameterized LQG problems. And second, most existing numerical methods for these problems lack rigorous guarantees of global convergence.
Emmy: They claim they fill those gaps by deriving a necessary and sufficient condition for the global optimality of stationary points in parameterized LQG problems. This condition simplifies the verification step down to testing controllability and observability for a new transfer function, which is much easier than what was previously required.
Kolya: So what makes this matter beyond just simplifying the verification? The paper suggests that traditional parameterizations inherently lead to suboptimal stationary points because direct parameterization acts like a projection that destroys the benign structure of the objective functional.
Ada: That's right. It shows that you can get stuck at a local optimum simply because of how you choose to write down your controller parameters; it’s a structural issue with the parameterization itself, not just a numerical fluke.
Emmy: It connects back to the LQR problem, where stationary points are always optimal, but for LQG problems, this gap between stationarity and optimality is much wider because of those noise terms.
Kolya: So if we think about what this means for the control engineer on the ground? They're dealing with a system that has both process and measurement noise. How does this paper change their approach to finding a controller?
Ada: It suggests that instead of just blindly optimizing over every possible parameter set, you can use this new condition to filter out the bad ones much more effectively. It provides a theoretical foundation for policy optimization methods in more challenging problems like LQG and dynamic filtering forty-one thirty-two <ref:2508.17252#pg2>.
Emmy: They are also drawing on prior work where they showed that the optimal dynamic filtering process satisfies the weak Polyak-Lojasiewicz condition and proposed a transformation technique to make the cost functional coercive. That background theory helps them set up their new framework for tackling these non-convex problems.
Kolya: Building on that, they’re combining policy optimization methods with convex parameterization theory to leverage the strengths of both approaches, which is a way of trying to manage the complexity inherent in LQG and H-infinity controller optimization forty-one forty-two <ref:2508.17252#pg2>.
Conclusion: Ada: So to wrap up, this paper, "Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective," by Li, Li, Ni, and Zhang addresses a long-standing theoretical issue about finding globally optimal solutions for controllers under noise.
Emmy: Essentially, they provide a necessary and sufficient condition that turns the problem of proving optimality into a much more tractable test involving testing controllability and observability for a novel transfer function. This gives researchers much better tools to certify solutions in these complex settings.
Kolya: What does this mean practically for us? It means that when we can't rely on perfect models, we have a rigorous way to find good control policies even without perfect system knowledge, provided we are using these new gradient-based algorithms correctly.
Ada: Exactly. They’re showing that direct policy optimization is viable in a mathematically sound way in the infinite-dimensional space, and they've shown global convergence for the algorithm itself under those conditions.
Emmy: The implication is that model-free implementations might become more feasible because they have a theoretical backbone to back up the idea of using data instead of just relying on perfect system identification.
Kolya: So, in short, this work provides a path forward for applying policy optimization methods to real-world problems where the system models are inherently imperfect.
Haoran Li, Xun Li, Yuan-Hua Ni, Xuebo Zhang
The Hong Kong Polytechnic University · Nankai University
math.OC
Submitted: 2025-08-24
Updated: 2026-11-10
Comments: This version corrects several typos and notational errors in both the derivations and the statements of theorems
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 76/100
The gist: The paper addresses the challenges in policy optimization for Linear Quadratic Gaussian (LQG) control by deriving a necessary and sufficient condition for global optimality and developing a…
Key concepts
- Necessary and Sufficient Condition
- This is a mathematical test that determines if a control solution is truly the best possible one globally. The authors derive this condition by relating it to controllability and observability of a specific transfer function, simplifying the complex search for optimal parameters.
- Infinite-Dimensional Space (RH∞)
- Instead of searching in finite dimensions, this method operates in an infinite space where the controller parameters reside. This allows the algorithm to handle complex control structures without needing to truncate them into simpler models, which avoids approximation errors.
- Youla Parameterization
- This is a specific way to represent controllers in LQG problems using transfer functions. The paper reformulates the optimization problem by working directly with these parameters in an infinite-dimensional space, transforming a difficult non-convex search into a more manageable convex optimization.
- Zeroth-Order Gradient Estimation
- Since calculating the true gradient is hard, this method estimates it using only system input/output data (zeroth-order methods). It uses Monte Carlo sampling to approximate the derivative of the cost function without needing complex model derivatives, which is useful when system models are unknown.
Terminology
Summary
The paper addresses the challenges in policy optimization for Linear Quadratic Gaussian (LQG) control by deriving a necessary and sufficient condition for global optimality and developing a gradient-based algorithm in an infinite-dimensional space, which provides rigorous guarantees. p. 0
Theoretical Insights on Optimality
The authors derive a necessary and sufficient condition for the global optimality of stationary points in parameterized LQG problems, which reduces verification to a test of controllability and observability for a novel transfer function. This condition provides a rigorous explanation for why traditional parameterizations can lead to suboptimal stationary points by showing that direct parameterization acts as a projection that destroys the benign structure of the objective functional. p. 2
Algorithmic Developments in Infinite Dimensions
The work elevates the controller parameter space from finite-dimensional settings to the infinite-dimensional RH∞ space and develops a gradient-based algorithm for this setting, establishing a theoretical analysis that proves global convergence. This approach bypasses the need for initial truncation associated with fixed-order model approximations, avoiding the intrinsic approximation error of such methods. p. 3
Structural Differences Between LQR and LQG
The paper provides a structural explanation for suboptimal stationary points by contrasting the LQG problem with the LQR problem, noting that in LQR, any stationary point is globally optimal. For LQG, the gap between stationarity and optimality arises because the gradient is projected onto a subspace of structurally admissible controllers, and a zero residue does not guarantee global optimality. p. 2
Optimization Method Based on Optimality Condition
A new controller synthesis framework is developed that operates directly on the Youla parameterization in an infinite-dimensional space U, transforming the non-convex synthesis problem into a convex optimization over a new parameter space. The gradient of this reformulated cost functional contains a transfer function matrix that precisely recovers the frequency-domain optimality condition established earlier. p. 7
Convergence Analysis and Data-Driven Extensions
The proposed gradient-based algorithm is analyzed for convergence, proving sublinear convergence under specific step size conditions. Furthermore, the method is shown to be extensible to a data-driven setting where key system operators can be estimated from input-output data using zeroth-order methods. p. 2 This includes estimating the transfer function matrix MK0 and the residue terms via Monte Carlo based estimation procedures. p. 3
Numerical Validation of Findings
Numerical experiments validate the theoretical findings, demonstrating the practical viability of the proposed approach by showing that Algorithm 1 can successfully navigate away from suboptimal stationary points in complex optimization landscapes compared to vanilla policy gradient methods. The data-driven estimation methods are also validated through simulations where they accurately estimate system operators like MK0 and Sk(s) from simulated noise-free scenarios. p. 2
Conclusion
This study investigates the direct policy optimization of Linear Quadratic Gaussian (LQG) control, proposing a verifiable necessary and sufficient condition for global optimality and developing a gradient-based algorithm in the infinite-dimensional RH∞ space with proven global convergence. Future work should focus on scalability to large systems and robustness against unmodeled dynamics. The preliminary data-driven extension suggests a path toward model-free implementations for applications where system models are unavailable or inaccurate. p. 2
Appendix A Data-Driven Estimation Procedures for Key Operators
This subsection briefly describes the estimation methods for the interconnection operator MK0, the closed-loop sensitivity operator Sk(s) in Algorithm 1, and the residue terms.
A.0.1 Estimation of MK0
To ensure sufficient excitation of the internal filter dynamics during this process, we augment the filter equation of the overall system (8) by introducing an explicit auxiliary input uxt ˆ. By taking the Laplace transform of the system equations (8) (with the auxiliary input uxt ˆ included), we obtain the following input-output relationship y(s) = MK0,22(s) uω(s) + w(s), where y(s) is the Laplace transform of output signal (yt, xˆt), uω(s) is the Laplace transform of the composite input (ut, uxt ˆ), MK0,22(s) denotes the noise-free transfer operator of interest, and w(s) aggregates the effects of all the noise sources.
A.0.1 Estimation of MK0
We sequentially apply sinusoidal excitations to each input channel of (ut, uxt ˆ) for system identification. By taking the Laplace transform of the system equations (8) (with the auxiliary input uxt ˆ included), we obtain the following input-output relationship y(s) = MK0,22(s) uω(s) + w(s), where y(s) is the Laplace transform of output signal (yt, xˆt), uω(s) is the Laplace transform of the composite input (ut, uxt ˆ), MK0,22(s) denotes the noise-free transfer operator of interest, and w(s) aggregates the effects of all the noise sources.
A.0.1 Estimation of MK0
Once the empirical frequency response data have been collected, we fit each entry of the transfer function matrix MK0,22(s) individually by parameterizing it as a rational function. This problem is nonlinear in the polynomial coefficients due to the rational form. p. 0 However, in the absence of noise, both the empirical output data and the target transfer function can be matched exactly. p. 2 Hence, the global optimum achieves zero objective value (i. p. 6e., the rational function equals the actual transfer function at all frequencies). p. 8
A.0.3 Estimation of Residue Terms
To estimate the residue term e(q+m1)×(q+m2)ResR(s)<0(Sk(s)), m1, m2, we employ zeroth-order (gradient-free) methods. Using zeroth-order methods (see, e. p. 1g., Sec. D.3 in [10]), the gradient can be estimated as ∇ˆ Qk JK0(Qk, Qk) = 1/m Xm i=1 d 2r 2 EU∼Unif(Sr) [(JK0(Qk, Qk + U) - JK0(Qk, Qk - U))U]. p. 0
B Second-Order Optimality Condition
By combining Theorem 3.4 in this paper with Theorem 2 in [43], we characterize the optimality condition in terms of the second-order derivatives of the objective function in (8). p. 4 This result is formalized in Corollary B. p. 01, which states that K is a global optimal solution if and only if the second-order directional derivative of JKe(λ) with respect to ∆ is identically zero, i. p. 4e., D2∆JKe(λ) = 0, ∀λ < 0, ∆ ∈ Mp.
D2∆JKe(λ) = 4 tr (ΣK 0 0 P'K,∆ [0 0 B∆C 0] + 2P'K,∆ [BKV∆⊤ B 0] + M1 (PK, ∆).
Corollary B.1
Let K ∈ Kq and p be a positive integer. Then, K is a global optimal solution to the problem (8) if and only if the second-order directional derivative of JKe(λ) with respect to ∆ is identically zero, i. p. 4e., D2∆JKe(λ) = 0, ∀λ < 0, ∆ ∈ Mp.
Corollary B.1
If K is not a global optimal solution to problem (8), there exist ∆ ∈ Mp and λ < 0 such that D2∆J(Ke(λ)) < 0.
D JKe(λ) = 4 tr (ΣK 0 0 P'K,∆ [BKV∆⊤ B 0] + M1 (PK, ∆).
Corollary B.1
In conclusion, D2∆JKe(λ) = 0 for any λ < 0, ∆ ∈ Mp if and only if the condition (27) holds.
References
[1] Osvaldo Agamennoni, Eduardo Paolini, and Alfredo Desages. On robust stability analysis of a control system using laguerre series. p. 1
[3] Dimitri Bertsekas. Dynamic Programming and Optimal Control, volume 1. p. 0
[4] Jean-Pierre Corriou. Optimal Control, pages 493–554. p. 0
[6] Jingliang Duan, Wenhan Cao, Yang Zheng, and Lin Zhao. p. 1 On the optimization landscape of dynamic output feedback linear quadratic control. p. 0
[9] Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. p. 1 Global convergence of policy gradient methods for the linear quadratic regulator. p. 0 p. 0
[10] Maryam Fazel, Rong Ge, Sham M. p. 1 Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator.
[16] Bin Hu, Kaiqing Zhang, Na. p. 0 Li, Mehran Mesbahi, Maryam Fazel, and Tamer Bacsar. p. 1 Towards a theoretical foundation of policy optimization for learning control policies. p. 0
[17] Joao Paulo Jansch-Porto, Bin Hu, and Geir E. Dullerud. Policy optimization for markovian jump linear quadratic control: Gradient method and global convergence. p. 0
[18] Imane Mahtout, Francisco Navas, Vicente Milanes, and Fawzi Nashashibi. Advances in youla-kucera parametrization: A review. p. 0
[19] Volodymyr Mnih et al. Human-level control through deep reinforcement learning. p. 0
[34] Jack Umenberger, Max Simchowitz, Juan Perdomo, Kaiqing Zhang, and Russ Tedrake. p. 0 Globally convergent policy search over dynamic filters for output estimation. p. 2
[36] Dante Youla and Joseph Bongiorno. p. 2 Modern wiener-hopf design of optimal controllers–part II: The multivariable case.
[43] Yang Zheng, Yue Sun, Maryam Fazel, and Na Li. p. 2 Escaping high-order saddles in policy optimization for linear quadratic gaussian (LQG) control. p. 0
Table 1: Relative error between real and estimated coefficients for non-zero elements of MK0
TF11(s) TF13(s) TF22(s) TF31(s) TF33(s).
Figure 1: Convergence comparison between vanilla policy gradient and Algorithm 1 when initialized (a) near and (b) exactly at a sub-optimal stationary point. p. 6 p. 6
Table 2: Quantitative relative error of the zeroth-order gradient estimation under different sample sizes. p. 1 p. 1
Figure 3: Qualitative convergence of the zeroth-order gradient estimate as the number of Monte Carlo samples (m) increases. p. 1 p. 1
Table 1: Relative error between real and estimated coefficients for non-zero elements of MK0,22(s). p. 3
Figure 1: Convergence comparison between vanilla policy gradient and Algorithm 1 when initialized (a) near and (b) exactly at a sub-optimal stationary point.
Table 2: Quantitative relative error of the zeroth-order gradient estimation under different sample sizes.
Figure 3: Qualitative convergence of the zeroth-order gradient estimate as the number of Monte Carlo samples (m) increases.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Certified Global Optimality for LQG Controllers: The system can now rigorously verify if a candidate controller parameter set is globally optimal by testing controllability and observability of a novel transfer function, providing a
precise and computationally tractable certificate.
-
Globally Convergent Gradient-Based Optimization in Infinite Dimensions: The algorithm can now operate directly in the infinite-dimensional RH infinity space, bypassing the need for initial truncation and providing a theoretical analysis establishing global convergence for policy optimization.
-
Structural Explanation of Suboptimal Stationary Points: The system gains the ability to explain why traditional parameterizations lead to suboptimal stationary points by revealing that
the direct parameterization acts as a projection, while preserving the unique minimum of the optimization landscape for LQR problem, destroys the benign structure of the objective functional of LQG problem.
-
Robustness to Initial Conditions: The algorithm can reliably bypass local optima and saddle points in complex landscapes; for instance, when initialized exactly at a stationary point where
the gradient ∇J(K0) is identically zero,
it identifies an efficient search direction and achieves cost reduction within just 14 iterations. -
Data-Driven Model-Free Policy Optimization: The system can be extended to a model-free setting by outlining a parameter estimation scheme that relies on input/output data to estimate key operators like the transfer function matrix MK022(s) and residue terms, enabling practical viability in applications where system models are unavailable or inaccurate.
-
Accurate Parameter Estimation from Data: The AI can accurately estimate the necessary components of the optimization process using zeroth-order methods, such as Monte Carlo based estimation for residue terms, with quantitative results showing relative errors decreasing significantly as sample size increases up to 10,000 samples.
Sources
- On the Gradient Domination of the LQG Problem
- Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies
- Globally Convergent Policy Search over Dynamic Filters for Output Estimation
- Benign Nonconvex Landscapes in Optimal and Robust Control, Part I: Global Optimality
- Benign Nonconvex Landscapes in Optimal and Robust Control, Part II: Extended Convex Lifting
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise