Riemannian Lyapunov Framework: Optimization as Closed Loop Control on Riemannian Manifolds

arXiv:2601.22284 · cs.LG · Submitted 2026-01-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Riemannian Lyapunov Framework".

Jane: Here's the gist of the paper. Riemannian Lyapunov Optimizers (RLOs) introduce a unified geometric framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold This framework allows for the systematic design of stable and effective optimizers by grounding them in control theory, bridging control theory and modern machine learning optimization.

Tom: Unified Geometric Foundation The paper establishes a unified geometric principle where diverse optimization components are equivalent to fundamental geometric objects Specifically, it demonstrates that preconditioning corresponds to selecting a Riemannian metric and momentum represents an extended velocity state in the tangent bundle.

Jane: This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs.

Tom: NAIM Mechanism and Control Synthesis The core structure organizing training dynamics is the Normally Attracting Invariant Manifold (NAIM) This mechanism organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it.

Paper summary: Tom: So to wrap up this first part of our talk, we’ve covered that this paper introduces Riemannian Lyapunov Optimizers, or RLOs. The central thesis is that they unify diverse optimization algorithms within a single geometric framework <ref:2601.22284#pg1>.

Jane: They claim these optimizers aren't just heuristic tweaks to existing methods; they are systematically derived from control theory by reinterpreting optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold Summary.

Lu: The key idea is the identification of a Normally Attracting Invariant Manifold, or NAIM. This NAIM organizes the training dynamics into two stages: rapid alignment of the speed state to a target graph, and then controlled evolution within it <ref:2601.22284#pg3>.

Meng: And they formalize this by constructing a strict Lyapunov function that certifies convergence to that target manifold Summary. This is what gives them mathematical proof of stability.

Tom: And the payoff for us is a constructive "optimizer generator" <ref:2601.22284#pg1>. This generator doesn't just recover classic algorithms; it enables the principled design of new RLOs Summary.

Jane: It means we can move beyond trial and error with hyperparameters and start designing optimizers based on a systematic synthesis of Lyapunov based feedback laws Page four.

Lu: They demonstrate that diverse components, like preconditioning or momentum, are equivalent to fundamental geometric objects—preconditioning is just selecting a Riemannian metric, and momentum is an extended velocity state in the tangent bundle <ref:2601.22284#pg3>.

Meng: That's a lot of abstraction. But if we look at it from a deployment standpoint, this means we can define the structure of the optimizer upfront rather than just tweaking coefficients after training starts Summary.

Tom: Exactly. It takes optimization from an empirical tuning exercise into a principled design problem based on control theory.

Jane: And they even offer validation through geometric diagnostics and show that grounding optimization in control theory yields state-of-the-art performance in large scale benchmarks Summary.

Conclusion: Tom: We’ve seen how this framework uses control theory to structure optimization via the NAIM, leading to a unified generator <ref:2601.22284#pg1>. Now, let's talk about the implications of this work overall.

Jane: The authors are Yixuan Wang, Omkar Sudhir Patil, and Warren E. Dixon from the University of Florida pg1. This work shows that optimization is fundamentally a control problem on a curved space.

Lu: It provides a generative grammar for optimization algorithms, meaning it’s not just a new tool; it’s a way to think about building them from first principles Summary.

Meng: For me, the biggest implication is shifting our focus in R andD. Instead of just chasing the next Adam or SGD tweak, we should be designing the underlying geometric structure—the metric and the state transitions <ref:2601.22284#pg3>.

Tom: That's right. If we treat it like a control system, then stability and convergence become properties of the controller design itself, not just luck with initialization.

Jane: It gives us a systematic toolkit for designing optimizers that are inherently stable and effective Summary. It moves the needle from heuristic experimentation to principled design based on Lyapunov feedback laws Page four.

Lu: The framework successfully demonstrates that adaptive preconditioning within RLO-Λ provides robust benefits across different model families Summary. It's about making the optimization process more predictable.

Meng: Predictability is what we need when we're trying to deploy models that run reliably on real traffic Summary. I’m cautious, though, because of those architecture-specific insights regarding the lifting parameter eta Summary.

Tom: So the final word is that the NAIM-Lyapunov framework offers a mathematical language for optimization. It’s a systematic toolkit for building stable and effective optimizers across different model types.

University of Florida

cs.LG

Submitted: 2026-01-29

Updated: 2026-10-29

Comments: 33 pages, 4 figures. Accepted at Neurips 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Riemannian Lyapunov Optimizers (RLOs) introduce a unified geometric framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter…

Key concepts

Riemannian Parameter Manifold
This is the geometric space where optimization happens. It's not a flat space but a curved one where different optimization components like preconditioning and momentum are mapped onto specific geometric objects. This allows for a unified mathematical description of how the learning process evolves.
Normally Attracting Invariant Manifold (NAIM)
This is the geometric skeleton that organizes training dynamics into two phases: first, rapidly aligning the system's speed state to a target direction, and second, evolving within that target structure. It defines the desired relationship between update speed and the gradient field.
Strict Lyapunov Function
This function mathematically encodes both how well the optimization objective is being minimized and how far the system is from its desired target manifold (NAIM). By using this function, researchers can derive a controller that actively steers the optimization trajectory toward stability.
Optimizer Generator
This is a constructive framework that systematically generates optimizers. Instead of guessing new algorithms, you select geometric components like the Riemannian metric and momentum to build an optimizer based on control theory principles.

Terminology

Summary

Riemannian Lyapunov Optimizers (RLOs) introduce a unified geometric framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold This framework allows for the systematic design of stable and effective optimizers by grounding them in control theory, bridging control theory and modern machine learning optimization. p. 0

Where the paper says it

edu ABSTRACT We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework. Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold.

Unified Geometric Foundation

The paper establishes a unified geometric principle where diverse optimization components are equivalent to fundamental geometric objects Specifically, it demonstrates that preconditioning corresponds to selecting a Riemannian metric and momentum represents an extended velocity state in the tangent bundle. This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs.

NAIM Mechanism and Control Synthesis

The core structure organizing training dynamics is the Normally Attracting Invariant Manifold (NAIM) This mechanism organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. The framework is constructed using a strict Lyapunov function that encodes both the objective value and the distance to the NAIM. p. 1

Where the paper says it

We formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric skeleton relating the update speed to the target direction field. Drawing inspiration from the idea of backstepping in nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and the distance to the NAIM.

This approach allows for deriving a controller that actively forces the system to track the manifold, transforming optimizer design from heuristic experimentation into a principled synthesis of Lyapunov based feedback laws. p. 1
Where the paper says it

Drawing inspiration from the idea of backstepping in nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and the distance to the NAIM. This approach allows us to derive a controller that actively forces the system to track the manifold, transforming optimizer design from heuristic experimentation into a principled synthesis of Lyapunov based feedback laws.

Lyapunov Stability and Convergence Guarantees

The analysis provides rigorous guarantees for stability under disturbances through Theorem 3.1 This theorem shows that the discrete Lyapunov function candidate satisfies a difference inequality, confining the trajectory to a “thick tube" around the target manifold. Furthermore, under the Polyak-Lojasiewicz condition, this inequality implies that the optimization error is ultimately bounded by the magnitude of the geometric forcing δk. p. 4

Where the paper says it

There exists a weighting constant α > 0 such that for sufficiently small hk, the discrete Lyapunov function candidate Vk(θk, zk) ≜ f(θk) − f ⋆ + α hk ∥zk∥ 2 gk satisfies the difference inequality: Vk+1 − Vk ≤ −c1hk∥ grad f(θk)∥ 2 g − c2 ηk hk ∥zk∥ 2 g + c3 1 ηkhk ∥δk∥ 2 g , where c1, c2, c3 > 0 are constants. This implies that the residual zk is uniformly bounded by a region proportional to the forcing magnitude ∥δk∥g, effectively confining the trajectory to a “thick tube" around Λ.

Empirical Validation and Algorithm Unification

The framework is validated through systematic ablation experiments showing that smooth direction fields yield more robust optimization when global normalization is absent. The RLO family, including RLO-Λ, achieves the best average performance over ImageNet classification benchmarks. p. 9

Where the paper says it

This synthesis ensures that resulting algorithms inherit stability properties from the continuous-time analysis. Systematic ablation validates that the NAIM geometric structure provides optimization benefits independent of auxiliary mechanisms, while large-scale experiments demonstrate that RLO-Λ achieves the best average performance over ImageNet classification benchmarks.

The algorithm template (Algorithm 1) serves as a universal computational template determined by four geometric components: the metric g, the internal state transition Ψ, the direction field generator Φ, and the lifting parameter η. p. 4
Where the paper says it

We formally encapsulate this control logic into the RLO, presented in Algorithm 1. This algorithm serves not merely as a new method, but as a universal computational template determined by four geometric components: the metric g, the internal state transition Ψ, the direction field generator Φ, and the lifting parameter η.

This structure recovers standard optimizers like AdamW and Lion through specific instantiations of these components. p. 9
Where the paper says it

This NAIM-Lyapunov framework characterizes optimizers through three geometric objects: a Riemannian metric M, a direction field generator Φ, and a contraction rate η. This abstraction recovers existing methods as special cases: SGD with momentum, Adam, and Lion all emerge from specific instantiations of these components.

Architecture-Specific Insights

The study reveals architecture-specific behaviors, such as the performance difference between RLO-Lifted on ViT-S/16 and ViT-B/16, suggesting that the optimal lifting parameter η may depend on model capacity. The explicit velocity state in RLO-Lifted introduces temporal smoothing, which may impede necessary rapid adaptation on smaller models like ViT-S/16, while larger models benefit from this smoothing. p. 9

Where the paper says it

The scale-dependent behavior of RLO-Lifted, which performs well on ResNet-50 and ViT-B/16 but poorly on ViT-S/16, suggests that the optimal lifting parameter η may depend on model capacity in ways that warrant further investigation. We conjecture that smaller models require faster adaptation to rapid landscape changes, which the explicit velocity state in RLO-Lifted may impede.

This suggests that for smaller transformer architectures, the explicit velocity state should be configured for rapid tracking (η close to 1. p. 9
Where the paper says it

The scale-dependent behavior of RLO-Lifted, which performs well on ResNet-50 and ViT-B/16 but poorly on ViT-S/16, suggests that the optimal lifting parameter η may depend on model capacity in ways that warrant further investigation. We conjecture that smaller models require faster adaptation to rapid landscape changes, which the explicit velocity state in RLO-Lifted may impede.

0) rather than strong smoothing (η closer to 0). p. 9
Where the paper says it

strong inductive biases of convolution constrain the optimization landscape to be relatively benign regardless of the descent direction. Vision Transformers.

The final results show that RLO-Λ consistently achieves the highest final accuracy across all tested architectures. p. 8
Where the paper says it

33 Figure 3: Validation accuracy curves for ImageNet classification using ResNet-50 (left), ViT-B/16 (center), and ViT-S/16 (right). RLO-Λ consistently achieves the highest final accuracy across all architectures (73.

Conclusion and Practical Guidance

The paper concludes that the NAIM-Lyapunov framework provides a generative grammar for optimization algorithms rather than merely a post-hoc taxonomy. The Lyapunov-based construction methodology transforms optimizer design from heuristic experimentation into strict Lyapunov controller design. p. 9

Where the paper says it

The framework thus provides a generative grammar for optimization algorithms rather than merely a post-hoc taxonomy. The Lyapunov-based construction methodology transforms optimizer design from heuristic experimentation into strict Lyapunov controller design.

It is recommended that practitioners use η = 1 as a reasonable default for the lifting parameter, while reducing η to values in the range [0. p. 6
Where the paper says it

Our theoretical analysis in Section 3 predicts that smoother direction field generators Φ should yield more stable optimization by reducing the drift term δk in the Lyapunov bound. To test this prediction, we modify RLO-Lifted to compare its default smooth hyperbolic tangent Φ = tanh(γ·) against the discontinuous sign function Φ = sign(·) used in Lion and the base RLO variant.

3, 0.7] may provide beneficial smoothing at minimal accuracy cost. p. 6
Where the paper says it

Rather, it serves as a convenient mechanism for automatic learning rate adaptation that can be substituted by manual tuning without fundamental loss of performance. Third, the comparison between RLO-Lifted and Nolifted (η = 1) reveals minimal difference in accuracy.

The framework successfully demonstrates that adaptive preconditioning in RLO-Λ provides robust benefits across different model families. p. 9
Where the paper says it

The consistent strong performance of RLO-Λ across all three architectures supports the theoretical framework developed in Sections 3 and 4. The adaptive preconditioning in RLO-Λ, which constructs the target manifold Λ using secondmoment information, appears to provide robust benefits across different model families.

This research is based on work supported in part by AFOSR grant FA9550-19-1-0169, AFRL grant FA8651-21-F-1027, and Office of Naval Research Grant N00014-13-1-0151. p. 9
Where the paper says it

Systematic ablation validates that the NAIM geometric structure provides optimization benefits independent of auxiliary mechanisms, while large-scale experiments demonstrate that RLO-Λ achieves the best average performance over ImageNet classification benchmarks. 8 Acknowledgment This research is based on work supported in part by AFOSR grant FA9550-19-1-0169, AFRL grant FA8651-21-F-1027, and Office of Naval Research Grant N00014-13-1-0151.

The references are listed in the final section. p. 9
Where the paper says it

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsoring agency. References [1] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.

The paper is available on arXiv as 2601. p. 0
Where the paper says it

We introduce a unified framework that reinterprets optimization as a closed loop ∗Corresponding author. arXiv:2601.

22284v1 [cs. p. 0
Where the paper says it

arXiv:2601. 22284v1 [cs.

LG] 3 Jan 9, 50 pages copied word for word>

--- Page 1 ---

RIEMANNIAN LYAPUNOV OPTIMIZER: A UNIFIED FRAMEWORK FOR OPTIMIZATION Yixuan Wang, Omkar Sudhir Patil, Warren E. Dixon Department of Mechanical and Aerospace Engineering University of Florida

--- Page 2 ---

ABSTRACT We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. p. 0

Where the paper says it

Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it.

We formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold. p. 0
Where the paper says it

Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. We formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold.

This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs. We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks. p. 0
Where the paper says it

This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs. We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks.

Overall, RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit for designing stable, effective optimizers. p. 0
Where the paper says it

We validate our theory via geometric diagnostics and demonstrate that grounding optimizer design in control theory yields state-of-the-art performance in large-scale benchmarks. Overall, RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit for designing stable, effective optimizers.

--- Page 3 ---

1 Introduction The evolution of deep learning optimization has produced a vast ecosystem of algorithms designed to navigate complex loss landscapes [1, 2, 3] This lineage includes Stochastic Gradient Descent (SGD) [4, 5], momentum methods [6, 7], Nesterov’s Accelerated Gradient (NAG) [8], and a diverse array of adaptive methods such as AdaGrad [9], RMSProp [10], Adam [11], AdamW [12], and Adafactor [13]. While these methods are empirically excellent and form the backbone of modern machine learning, they are largely treated as a collection of distinct heuristics. p. 0

Where the paper says it

Recently, specialized techniques like Lion [14], Shampoo [15], and Sophia [16] have pushed the boundaries of efficiency in large scale training by utilizing sign based updates [17, 18] or second order information [19]. While these methods are empirically excellent and form the backbone of modern machine learning, they are largely treated as a collection of distinct heuristics.

Theoretical analyses of these optimizers remain fragmented, often focusing on narrow properties or specific algorithmic artifacts [20, 2601. p. 0
Where the paper says it

While these methods are empirically excellent and form the backbone of modern machine learning, they are largely treated as a collection of distinct heuristics. Theoretical analyses of these optimizers remain fragmented, often focusing on narrow properties or specific algorithmic artifacts [20, 21] while failing to provide a global, principled explanation for their shared success.

22284] while failing to provide a global, principled explanation for their shared success. p. 0
Where the paper says it

While these methods are empirically excellent and form the backbone of modern machine learning, they are largely treated as a collection of distinct heuristics. Theoretical analyses of these optimizers remain fragmented, often focusing on narrow properties or specific algorithmic artifacts [20, 21] while failing to provide a global, principled explanation for their shared success.

--- Page 4 ---

controlled dynamical system on a Riemannian manifold. Our motivation is grounded in the observation of two timescale dynamics where the system state must track a specific relationship between its velocity and the gradient field We formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric skeleton relating the update speed to the target direction field. p. 1

Where the paper says it

Our motivation is grounded in the observation of two timescale dynamics where the system state must track a specific relationship between its velocity and the gradient field. We formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric skeleton relating the update speed to the target direction field.

Drawing inspiration from the idea of backstepping in nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and the distance to the NAIM. p. 1
Where the paper says it

We formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric skeleton relating the update speed to the target direction field. Drawing inspiration from the idea of backstepping in nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and the distance to the NAIM.

This approach allows us to derive a controller that actively forces the system to track the manifold, transforming optimizer design from heuristic experimentation into a principled synthesis of Lyapunov based feedback laws. p. 1
Where the paper says it

Drawing inspiration from the idea of backstepping in nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and the distance to the NAIM. This approach allows us to derive a controller that actively forces the system to track the manifold, transforming optimizer design from heuristic experimentation into a principled synthesis of Lyapunov based feedback laws.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Accelerated convergence for deep learning optimization by implementing Riemannian Lyapunov Optimizers (RLOs) which unify classic optimizers within a geometric framework to systematically design stable and effective optimizers.

  2. Enables the principled design of new optimizers by using a constructive optimizer generator derived from control theory, allowing researchers to synthesize feedback laws that explicitly regulate auxiliary optimizer states.

  3. Provides provable convergence guarantees for optimization algorithms by constructing a strict Lyapunov function that certifies convergence to a target manifold, ensuring robustness even under stochastic gradient disturbances.

  4. Improves parameter adaptation in models like Adam and RMSProp by reinterpreting adaptive preconditioning matrices as Riemannian metrics, allowing the system to learn local geometry rather than relying on heuristic scaling.

  5. Enhances optimization stability on vision transformers by using RLO-Lifted with an optimal lifting parameter eta close to 1.0 for smaller architectures, achieving accuracy competitive with or exceeding the RLO-Lambda variant.

Abstract

We introduce the Riemannian Lyapunov Framework (RLF), a unified framework that represents first order optimization as a closed loop controlled dynamical system on a Riemannian parameter manifold. Within this framework, optimizers are specified by a quadruple (M, g, Φ, η) consisting of a Riemannian metric, an internal state transition map, a direction field generator, and a lifting parameter. Classical and modern methods including SGD, Adam, AdamW, Lion and Shampoo are recovered as specific instantiations of this quadruple. The central structural object of the framework is a time indexed family of target submanifolds in the extended state space of parameters, velocities, and internal memory, which we call the Normally Attracting Invariant Manifold (NAIM) family. These submanifolds organize training dynamics into two timescales: a fast contraction of the velocity state onto the current target submanifold, followed by a slow descent along it. Using Riemannian backstepping, we construct a strict Lyapunov function on the extended state space and prove Uniform Ultimate Boundedness (UUB) of the optimization trajectory under a descent alignment assumption and bounded stochastic disturbances. Under the additional Polyak-Lojasiewicz (PL) condition, the Lyapunov function converges linearly to a bounded neighborhood of the optimum, defined as the drift of the target direction field between consecutive iterates after parallel transport. To validate the framework's generativity, we derive three concrete instantiations of the quadruple, each exercising a distinct design axis, and show on geometric diagnostics, image classification benchmarks, and language modeling that their empirical behavior matches the theoretical predictions. The framework casts optimizer design as controller synthesis, complementing heuristic perspectives by providing a constructive template with provable guarantees.

Sources

Related papers