Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces

The gist: This work proposes a fully data-driven approach for optimal control of nonlinear controlaffine systems represented by a stochastic diffusion by embedding state probability densities into a reproducing kernel Hilbert space to identify Markov transition operators.

Problem Formulation and Context

The paper addresses the challenge of finding an optimal feedback control law for nonlinear systems where both the dynamics and stage cost functions are unknown, relying only on a control penalty function and constraints. The goal is to find a feedback control law π solving the stochastic optimal control problem on the domain X = R nx by minimizing the expected cost over time T0 to infinity. This involves navigating a system described by an SDE (1b) with unknown functions (f, G, l).

Operator Theory and Reformulation

The paper leverages operator-theoretic approaches to learn nonlinear system dynamics from data. Specifically, it focuses on learning finite-rank approximations of the Markov transition operators of stochastic control systems using kernel regression techniques. The evolution of the probability density ρ in the parabolic Fokker-Planck equation is given by L(π)ρ = -div(ρ(f + Gπ)) + ϵ∆ρ.

The original stochastic optimal control problem (1) is cast into a convex PDE-constrained optimization problem via a change of variables, defining J (T, ρ0) as the minimum of an integral involving the cost and penalty terms subject to constraints on the density evolution. This reformulation admits a unique minimizer (ρ⋆, ν⋆) with associated optimal feedback law π⋆ = ν⋆ρ⋆.

Data-Driven Operator Learning via RKHS

The core of the data-driven method involves embedding probability densities into a Reproducing Kernel Hilbert Space (RKHS) to approximate the dynamics. The embedding EX maps a probability density ρ into an RKHS as EXρ = Z X k(·, x)ρ(x) dx. Similarly, controls are embedded into R nu⊗HX via EX,U ν = Z X ν(x)⊗k(·, x) dx.

The empirical embeddings EbXw and EbX,Uw are constructed from finite training samples Dn=(x(i), u(i), x(i)+, l(x(i))). The operator T in the Hilbert space HZ is learned by minimizing the squared RKHS norm error over residuals by solving Tb = arg min T ∈ HS(HZ,HX) Xn i=1 ˆri(T) 2 HX + γT 2 HS. This leads to matrix representations for the operators A and B in terms of the Gram matrices (KX) and (K+) derived from the kernel evaluations.

Kernel HJB Recursion and Control Law Recovery

The paper develops a data-driven approximation of the global minimizer by substituting these learned operator estimators into the discretized dynamic programming recursion (9). This results in an equivalent Kernel HJB (KHJB) recursion vk = Avk+1 + lh + Dr(Bvk+1) s.t. vH = 0.

The recursion is solved using matrix representations derived from the learned operators A and B, leading to a tractable recursion in an n-dimensional subspace. The approximate optimal feedback control law is then obtained as πb⋆(x) = u⋆⟨Inu ⊗ k(x), Bv⋆⟩ Rnnu. Numerical examples demonstrate the method's ability to recover known optimal control laws with low Root Mean Squared Error (RMSE).

Conclusion

The proposed framework scales linearly with states, enabling polynomial runtime complexity for both identification and control. The Kernel HJB recursion provides a promising avenue for addressing high-dimensional and uncertain optimal control problems. The method successfully recovers optimal control laws reliably across various nonlinear benchmarks. This approach offers a data-driven solution that exploits the full nonlinear dynamics without requiring prior knowledge of system parameters or models.

--- Page 1 ---

This work has been submitted to IFAC for possible publication<ref:2407.16407#pg2> Data-Driven Stochastic Optimal Control in Reproducing Kernel Hilbert Spaces⋆ Nicolas Hoischena, Petar Bevandaa, Stefan Sosnowskia, Sandra Hirchea, Boris Houskab aChair of Information-oriented Control, TU Munich, Germany bSchool of Information Science and Technology, ShanghaiTech University, China Abstract This paper proposes a fully data-driven approach for optimal control of nonlinear controlaffine systems represented by a stochastic diffusion. The focus is on the scenario where both the nonlinear dynamics and stage cost functions are unknown, while only a control penalty function and constraints are provided. To this end, we embed state probability densities into a reproducing kernel Hilbert space (RKHS) to leverage recent advances in operator regression, thereby identifying Markov transition operators associated with controlled diffusion processes. This operator learning approach integrates naturally with convex operator-theoretic Hamilton–Jacobi–Bellman recursions that scale linearly with state dimensionality, effectively solving a wide range of nonlinear optimal control problems. Numerical results demonstrate its ability to address diverse nonlinear control tasks, including the depth regulation of an autonomous underwater vehicle<ref:2407.16407#pg2> Keywords: stochastic optimal control problems, learning methods for optimal control, operator theory, data-driven control theory, kernel methods

--- Page 2 ---

parameter estimation to purely data-driven policies, reflects a fundamental shift in which data becomes the primary source of insight as complexity increases. In recent years, operator-theoretic approaches have garnered substantial interest for learning nonlinear system dynamics from data (Kostic et al., 2022; Klus et al., 2020). Unlike conventional nonlinear representations in the physical state space, these methods describe the dynamics through a linear operator, most notably, the Koopman operator (Koopman, 1931), which fully characterizes the system’s evolution. This representational simplicity has enabled diverse applications including system identification (Li et al., 2017), soft robotics (Bruder et al., 2020), robust control (Strässer et al., 2023) and optimal control (Vaidya and Tellez-Castro, 2023; Caldarelli et al., 2025; Korda and Mezić, 18). Optimal control in state space traditionally relies on Bellman’s principle of optimality, but is severely limited by the curse of dimensionality, which renders Hamilton-Jacobi Bellman equations intractable even for moderate-scale problems (Bertsekas and Shreve, 1996). In such scenarios, local optimal control approaches, such as direct methods combined with sequential quadratic programming or interior points solvers (Diehl et al., 2009) have demonstrated superior performance over dynamic programming approaches. However, these local approaches require accurate initialization, are challenged by nonconvexity, and may result in suboptimal local solutions. In contrast, convex reformulations offer a promising route to globally optimal solutions in nonlinear control via linear embedding methods (Vinter, 1993), sum-of-squares dual approaches (Henrion et al., 2008), or operator-theoretic system representations (Houska, 2025; Guo et al., 2022), each entailing fundamentally different algorithmic frameworks. Moreover, convex reformulations circumvent the linearization trade-offs inherent to traditional local methods. Operator-theoretic representations are particularly attractive as they can be learned from a single dataset by modern nonparametric operator-theoretic learning methods (Kostic et al., 2022, 2023; Bevanda et al., 2025a) and integrate naturally with stochastic control theory for diffusion processes (Fleming and Soner, 1993; Bogachev et al., 18; Krylov, 18). We therefore build on these techniques to address stochastic optimal control design in flexible, infinite-dimensional, Reproducing Kernel Hilbert Spaces (RKHSs), which have a rich history in machine learning (Steinwart and Christmann, 2008). Contributions: This paper proposes a novel algorithm for solving data-driven stochastic optimal control problems. Our main contributions are: 1. We develop a nonparametric framework to learn finite-rank approximations of the Markov transition operators of stochastic control systems by using kernel regression techniques. The sample and time complexity of our kernel-based system identification method scales linearly with the number of states and controls<ref:2407.16407#pg3>. 2. We propose an RKHS embedded dynamic programming recursion, which leverages the data-driven Markov operator estimates to compute an associated approximation of the global minimizer of stochastic optimal control problems. The estimation and control algorithm is established under minimal system assumptions, without relying on stability conditions or detailed model knowledge, thereby relaxing common requirements in related existing works (Caldarelli et al., 2025; Vaidya and Tellez-Castro, 2023). In contrast to (Bevanda et al.

1. Data-driven approximation of Markov transition operators for stochastic control systems can be learned by using kernel regression techniques, allowing for a nonparametric framework to learn finite-rank approximations of the Markov transition operators of stochastic control systems. This enables the system to identify unknown nonlinear dynamics and stage cost functions directly from sampled data, bypassing the need for firstprinciple models.

2. Solving complex optimal control problems can be achieved via an RKHS embedded dynamic programming recursion, which leverages the data-driven Markov operator estimates to compute an associated approximation of the global minimizer of stochastic optimal control problems. This allows for computing a data-driven approximation of the global minimizer in a manner that is computationally simple.

3. The resulting system can compute an approximate feedback control law using the formula πb⋆(x) = u⋆⟨Inu ⊗ k(x), Bv⋆⟩Rnnu, which provides a practical solution for nonlinear systems like autonomous underwater vehicles, achieving a mean cost reduction of 5.09% and a standard deviation reduction of 30.11% compared to the LQR baseline.

This paper proposes a fully data-driven approach for optimal control of nonlinear control-affine systems represented by a stochastic diffusion. The focus is on the scenario where both the nonlinear dynamics and stage cost functions are unknown, while only a control penalty function and constraints are provided. To this end, we embed state probability densities into a reproducing kernel Hilbert space (RKHS) to leverage recent advances in operator regression, thereby identifying Markov transition operators associated with controlled diffusion processes. This operator learning approach integrates naturally with convex operator-theoretic Hamilton-Jacobi-Bellman recursions that scale linearly with state dimensionality, effectively solving a wide range of nonlinear optimal control problems. Numerical results demonstrate its ability to address diverse nonlinear control tasks, including the depth regulation of an autonomous underwater vehicle.