Riemannian Lyapunov Framework: Optimization as Closed Loop Control on Riemannian Manifolds
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Riemannian Lyapunov Framework".
Jane: Here's the gist of the paper. Riemannian Lyapunov Optimizers (RLOs) introduce a unified geometric framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold This framework allows for the systematic design of stable and effective optimizers by grounding them in control theory, bridging control theory and modern machine learning optimization.
Tom: Unified Geometric Foundation The paper establishes a unified geometric principle where diverse optimization components are equivalent to fundamental geometric objects Specifically, it demonstrates that preconditioning corresponds to selecting a Riemannian metric and momentum represents an extended velocity state in the tangent bundle.
Jane: This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs.
Tom: NAIM Mechanism and Control Synthesis The core structure organizing training dynamics is the Normally Attracting Invariant Manifold (NAIM) This mechanism organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it.
Paper summary: Tom: So to wrap up this first part of our talk, we’ve covered that this paper introduces Riemannian Lyapunov Optimizers, or RLOs. The central thesis is that they unify diverse optimization algorithms within a single geometric framework <ref:2601.22284#pg1>.
Jane: They claim these optimizers aren't just heuristic tweaks to existing methods; they are systematically derived from control theory by reinterpreting optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold Summary.
Lu: The key idea is the identification of a Normally Attracting Invariant Manifold, or NAIM. This NAIM organizes the training dynamics into two stages: rapid alignment of the speed state to a target graph, and then controlled evolution within it <ref:2601.22284#pg3>.
Meng: And they formalize this by constructing a strict Lyapunov function that certifies convergence to that target manifold Summary. This is what gives them mathematical proof of stability.
Tom: And the payoff for us is a constructive "optimizer generator" <ref:2601.22284#pg1>. This generator doesn't just recover classic algorithms; it enables the principled design of new RLOs Summary.
Jane: It means we can move beyond trial and error with hyperparameters and start designing optimizers based on a systematic synthesis of Lyapunov based feedback laws Page four.
Lu: They demonstrate that diverse components, like preconditioning or momentum, are equivalent to fundamental geometric objects—preconditioning is just selecting a Riemannian metric, and momentum is an extended velocity state in the tangent bundle <ref:2601.22284#pg3>.
Meng: That's a lot of abstraction. But if we look at it from a deployment standpoint, this means we can define the structure of the optimizer upfront rather than just tweaking coefficients after training starts Summary.
Tom: Exactly. It takes optimization from an empirical tuning exercise into a principled design problem based on control theory.
Jane: And they even offer validation through geometric diagnostics and show that grounding optimization in control theory yields state-of-the-art performance in large scale benchmarks Summary.
Conclusion: Tom: We’ve seen how this framework uses control theory to structure optimization via the NAIM, leading to a unified generator <ref:2601.22284#pg1>. Now, let's talk about the implications of this work overall.
Jane: The authors are Yixuan Wang, Omkar Sudhir Patil, and Warren E. Dixon from the University of Florida pg1. This work shows that optimization is fundamentally a control problem on a curved space.
Lu: It provides a generative grammar for optimization algorithms, meaning it’s not just a new tool; it’s a way to think about building them from first principles Summary.
Meng: For me, the biggest implication is shifting our focus in R andD. Instead of just chasing the next Adam or SGD tweak, we should be designing the underlying geometric structure—the metric and the state transitions <ref:2601.22284#pg3>.
Tom: That's right. If we treat it like a control system, then stability and convergence become properties of the controller design itself, not just luck with initialization.
Jane: It gives us a systematic toolkit for designing optimizers that are inherently stable and effective Summary. It moves the needle from heuristic experimentation to principled design based on Lyapunov feedback laws Page four.
Lu: The framework successfully demonstrates that adaptive preconditioning within RLO-Λ provides robust benefits across different model families Summary. It's about making the optimization process more predictable.
Meng: Predictability is what we need when we're trying to deploy models that run reliably on real traffic Summary. I’m cautious, though, because of those architecture-specific insights regarding the lifting parameter eta Summary.
Tom: So the final word is that the NAIM-Lyapunov framework offers a mathematical language for optimization. It’s a systematic toolkit for building stable and effective optimizers across different model types.
University of Florida
cs.LG
Submitted: 2026-01-29
Updated: 2026-10-29
Comments: 33 pages, 4 figures. Accepted at Neurips 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: Riemannian Lyapunov Optimizers (RLOs) introduce a unified geometric framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter…
Key concepts
- Riemannian Parameter Manifold
- This is the geometric space where optimization happens. It's not a flat space but a curved one where different optimization components like preconditioning and momentum are mapped onto specific geometric objects. This allows for a unified mathematical description of how the learning process evolves.
- Normally Attracting Invariant Manifold (NAIM)
- This is the geometric skeleton that organizes training dynamics into two phases: first, rapidly aligning the system's speed state to a target direction, and second, evolving within that target structure. It defines the desired relationship between update speed and the gradient field.
- Strict Lyapunov Function
- This function mathematically encodes both how well the optimization objective is being minimized and how far the system is from its desired target manifold (NAIM). By using this function, researchers can derive a controller that actively steers the optimization trajectory toward stability.
- Optimizer Generator
- This is a constructive framework that systematically generates optimizers. Instead of guessing new algorithms, you select geometric components like the Riemannian metric and momentum to build an optimizer based on control theory principles.
Terminology
Summary
Riemannian Lyapunov Optimizers (RLOs) introduce a unified geometric framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold This framework allows for the systematic design of stable and effective optimizers by grounding them in control theory, bridging control theory and modern machine learning optimization. p. 0 edu
ABSTRACT
We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that
unifies classic optimizers within one geometric framework. Unlike heuristic improvements to
existing optimizers, RLOs are systematically derived from a novel control-theoretic framework
that reinterprets optimization as an extended state discrete-time controlled dynamical system on
a Riemannian parameter manifold.Where the paper says it
Unified Geometric Foundation
The paper establishes a unified geometric principle where diverse optimization components are equivalent to fundamental geometric objects Specifically, it demonstrates that preconditioning corresponds to selecting a Riemannian metric and momentum represents an extended velocity state in the tangent bundle. This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms but enables the principled design of RLOs.
NAIM Mechanism and Control Synthesis
The core structure organizing training dynamics is the Normally Attracting Invariant Manifold (NAIM) This mechanism organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. The framework is constructed using a strict Lyapunov function that encodes both the objective value and the distance to the NAIM. p. 1 We
formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric
skeleton relating the update speed to the target direction field. Drawing inspiration from the idea of backstepping in
nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and
the distance to the NAIM. Drawing inspiration from the idea of backstepping in
nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and
the distance to the NAIM. This approach allows us to derive a controller that actively forces the system to track the
manifold, transforming optimizer design from heuristic experimentation into a principled synthesis of Lyapunov based
feedback laws.Where the paper says it
Where the paper says it
Lyapunov Stability and Convergence Guarantees
The analysis provides rigorous guarantees for stability under disturbances through Theorem 3.1 This theorem shows that the discrete Lyapunov function candidate satisfies a difference inequality, confining the trajectory to a “thick tube" around the target manifold. Furthermore, under the Polyak-Lojasiewicz condition, this inequality implies that the optimization error is ultimately bounded by the magnitude of the geometric forcing δk. p. 4 There exists a weighting constant α > 0 such that for sufficiently small hk, the discrete Lyapunov function
candidate
Vk(θk, zk) ≜ f(θk) − f
⋆ +
α
hk
∥zk∥
2
gk
satisfies the difference inequality:
Vk+1 − Vk ≤ −c1hk∥ grad f(θk)∥
2
g
− c2
ηk
hk
∥zk∥
2
g + c3
1
ηkhk
∥δk∥
2
g
,
where c1, c2, c3 > 0 are constants. This implies that the residual zk is uniformly bounded by a region proportional to
the forcing magnitude ∥δk∥g, effectively confining the trajectory to a “thick tube" around Λ.Where the paper says it
Empirical Validation and Algorithm Unification
The framework is validated through systematic ablation experiments showing that smooth direction fields yield more robust optimization when global normalization is absent. The RLO family, including RLO-Λ, achieves the best average performance over ImageNet classification benchmarks. p. 9 This synthesis ensures that resulting algorithms inherit stability properties from the continuous-time
analysis. Systematic ablation validates that the NAIM geometric structure provides optimization benefits independent of auxiliary
mechanisms, while large-scale experiments demonstrate that RLO-Λ achieves the best average performance over
ImageNet classification benchmarks. We formally encapsulate this control logic into the RLO, presented in Algorithm 1. This algorithm serves not
merely as a new method, but as a universal computational template determined by four geometric components: the
metric g, the internal state transition Ψ, the direction field generator Φ, and the lifting parameter η. This NAIM-Lyapunov framework characterizes optimizers through three geometric objects: a Riemannian metric
M, a direction field generator Φ, and a contraction rate η. This abstraction recovers existing methods as special
cases: SGD with momentum, Adam, and Lion all emerge from specific instantiations of these components.Where the paper says it
Where the paper says it
Where the paper says it
Architecture-Specific Insights
The study reveals architecture-specific behaviors, such as the performance difference between RLO-Lifted on ViT-S/16 and ViT-B/16, suggesting that the optimal lifting parameter η may depend on model capacity. The explicit velocity state in RLO-Lifted introduces temporal smoothing, which may impede necessary rapid adaptation on smaller models like ViT-S/16, while larger models benefit from this smoothing. p. 9 The scale-dependent behavior
of RLO-Lifted, which performs well on ResNet-50 and ViT-B/16 but poorly on ViT-S/16, suggests that the optimal
lifting parameter η may depend on model capacity in ways that warrant further investigation. We conjecture that
smaller models require faster adaptation to rapid landscape changes, which the explicit velocity state in RLO-Lifted
may impede. The scale-dependent behavior
of RLO-Lifted, which performs well on ResNet-50 and ViT-B/16 but poorly on ViT-S/16, suggests that the optimal
lifting parameter η may depend on model capacity in ways that warrant further investigation. We conjecture that
smaller models require faster adaptation to rapid landscape changes, which the explicit velocity state in RLO-Lifted
may impede. strong inductive biases of convolution constrain the optimization landscape to be relatively benign regardless of the
descent direction. Vision Transformers. 33
Figure 3: Validation accuracy curves for ImageNet classification using ResNet-50 (left), ViT-B/16 (center), and ViT-S/16
(right). RLO-Λ consistently achieves the highest final accuracy across all architectures (73.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Practical Guidance
The paper concludes that the NAIM-Lyapunov framework provides a generative grammar for optimization algorithms rather than merely a post-hoc taxonomy. The Lyapunov-based construction methodology transforms optimizer design from heuristic experimentation into strict Lyapunov controller design. p. 9 The
framework thus provides a generative grammar for optimization algorithms rather than merely a post-hoc taxonomy. The Lyapunov-based construction methodology transforms optimizer design from heuristic experimentation into strict
Lyapunov controller design. Our theoretical analysis in Section 3 predicts that smoother direction field generators Φ
should yield more stable optimization by reducing the drift term δk in the Lyapunov bound. To test this prediction, we
modify RLO-Lifted to compare its default smooth hyperbolic tangent Φ = tanh(γ·) against the discontinuous sign
function Φ = sign(·) used in Lion and the base RLO variant. Rather, it serves as a convenient mechanism for automatic learning rate adaptation that can be
substituted by manual tuning without fundamental loss of performance. Third, the comparison between RLO-Lifted and Nolifted (η = 1) reveals minimal difference in accuracy. The consistent strong performance of RLO-Λ across all three architectures supports the theoretical framework developed
in Sections 3 and 4. The adaptive preconditioning in RLO-Λ, which constructs the target manifold Λ using secondmoment information, appears to provide robust benefits across different model families. Systematic ablation validates that the NAIM geometric structure provides optimization benefits independent of auxiliary
mechanisms, while large-scale experiments demonstrate that RLO-Λ achieves the best average performance over
ImageNet classification benchmarks. 8 Acknowledgment
This research is based on work supported in part by AFOSR grant FA9550-19-1-0169, AFRL grant FA8651-21-F-1027,
and Office of Naval Research Grant N00014-13-1-0151. Any opinions, findings and conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsoring agency. References
[1] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. We introduce a unified framework that reinterprets optimization as a closed loop
∗Corresponding author. arXiv:2601. arXiv:2601. 22284v1 [cs.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
RIEMANNIAN LYAPUNOV OPTIMIZER: A UNIFIED FRAMEWORK FOR OPTIMIZATION Yixuan Wang, Omkar Sudhir Patil, Warren E. Dixon Department of Mechanical and Aerospace Engineering University of Florida
--- Page 2 ---
ABSTRACT We introduce Riemannian Lyapunov Optimizers (RLOs), a family of optimization algorithms that unifies classic optimizers within one geometric framework Unlike heuristic improvements to existing optimizers, RLOs are systematically derived from a novel control-theoretic framework that reinterprets optimization as an extended state discrete-time controlled dynamical system on a Riemannian parameter manifold. Central to this framework is the identification of a Normally Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages: rapid alignment of the speed state to a target graph, followed by controlled evolution within it. p. 0 Unlike heuristic improvements to
existing optimizers, RLOs are systematically derived from a novel control-theoretic framework
that reinterprets optimization as an extended state discrete-time controlled dynamical system on
a Riemannian parameter manifold. Central to this framework is the identification of a Normally
Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages:
rapid alignment of the speed state to a target graph, followed by controlled evolution within it. Central to this framework is the identification of a Normally
Attracting Invariant Manifold (NAIM), which organizes training dynamics into two distinct stages:
rapid alignment of the speed state to a target graph, followed by controlled evolution within it. We
formalize this by constructing a strict Lyapunov function that certifies convergence to a target manifold. This perspective yields a constructive “optimizer generator" that not only recovers classic algorithms
but enables the principled design of RLOs. We validate our theory via geometric diagnostics and
demonstrate that grounding optimizer design in control theory yields state-of-the-art performance
in large-scale benchmarks. We validate our theory via geometric diagnostics and
demonstrate that grounding optimizer design in control theory yields state-of-the-art performance
in large-scale benchmarks. Overall, RLOs bridge control theory and modern machine learning
optimization, providing a unified language and a systematic toolkit for designing stable, effective
optimizers.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
1 Introduction The evolution of deep learning optimization has produced a vast ecosystem of algorithms designed to navigate complex loss landscapes [1, 2, 3] This lineage includes Stochastic Gradient Descent (SGD) [4, 5], momentum methods [6, 7], Nesterov’s Accelerated Gradient (NAG) [8], and a diverse array of adaptive methods such as AdaGrad [9], RMSProp [10], Adam [11], AdamW [12], and Adafactor [13]. While these methods are empirically excellent and form the backbone of modern machine learning, they are largely treated as a collection of distinct heuristics. p. 0 Recently, specialized techniques like Lion [14], Shampoo [15],
and Sophia [16] have pushed the boundaries of efficiency in large scale training by utilizing sign based updates [17, 18]
or second order information [19]. While these methods are empirically excellent and form the backbone of modern
machine learning, they are largely treated as a collection of distinct heuristics. While these methods are empirically excellent and form the backbone of modern
machine learning, they are largely treated as a collection of distinct heuristics. Theoretical analyses of these optimizers
remain fragmented, often focusing on narrow properties or specific algorithmic artifacts [20, 21] while failing to provide
a global, principled explanation for their shared success. While these methods are empirically excellent and form the backbone of modern
machine learning, they are largely treated as a collection of distinct heuristics. Theoretical analyses of these optimizers
remain fragmented, often focusing on narrow properties or specific algorithmic artifacts [20, 21] while failing to provide
a global, principled explanation for their shared success.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 4 ---
controlled dynamical system on a Riemannian manifold. Our motivation is grounded in the observation of two timescale dynamics where the system state must track a specific relationship between its velocity and the gradient field We formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric skeleton relating the update speed to the target direction field. p. 1 Our motivation is grounded in the observation of two timescale
dynamics where the system state must track a specific relationship between its velocity and the gradient field. We
formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric
skeleton relating the update speed to the target direction field. We
formalize this using the concept of a Normally Attracting Invariant Manifold (NAIM), which serves as the geometric
skeleton relating the update speed to the target direction field. Drawing inspiration from the idea of backstepping in
nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and
the distance to the NAIM. Drawing inspiration from the idea of backstepping in
nonlinear control theory [26, 27, 28], we construct a strict Lyapunov function that encodes both the objective value and
the distance to the NAIM. This approach allows us to derive a controller that actively forces the system to track the
manifold, transforming optimizer design from heuristic experimentation into a principled synthesis of Lyapunov based
feedback laws.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Accelerated convergence for deep learning optimization by implementing Riemannian Lyapunov Optimizers (RLOs) which unify classic optimizers within a geometric framework to systematically design stable and effective optimizers.
-
Enables the principled design of new optimizers by using a constructive
optimizer generator
derived from control theory, allowing researchers to synthesize feedback laws that explicitly regulate auxiliary optimizer states. -
Provides provable convergence guarantees for optimization algorithms by constructing a strict Lyapunov function that certifies convergence to a target manifold, ensuring robustness even under stochastic gradient disturbances.
-
Improves parameter adaptation in models like Adam and RMSProp by reinterpreting adaptive preconditioning matrices as Riemannian metrics, allowing the system to learn local geometry rather than relying on heuristic scaling.
-
Enhances optimization stability on vision transformers by using RLO-Lifted with an optimal lifting parameter eta close to 1.0 for smaller architectures, achieving accuracy competitive with or exceeding the RLO-Lambda variant.
Abstract
We introduce the Riemannian Lyapunov Framework (RLF), a unified framework that represents first order optimization as a closed loop controlled dynamical system on a Riemannian parameter manifold. Within this framework, optimizers are specified by a quadruple (M, g, Φ, η) consisting of a Riemannian metric, an internal state transition map, a direction field generator, and a lifting parameter. Classical and modern methods including SGD, Adam, AdamW, Lion and Shampoo are recovered as specific instantiations of this quadruple. The central structural object of the framework is a time indexed family of target submanifolds in the extended state space of parameters, velocities, and internal memory, which we call the Normally Attracting Invariant Manifold (NAIM) family. These submanifolds organize training dynamics into two timescales: a fast contraction of the velocity state onto the current target submanifold, followed by a slow descent along it. Using Riemannian backstepping, we construct a strict Lyapunov function on the extended state space and prove Uniform Ultimate Boundedness (UUB) of the optimization trajectory under a descent alignment assumption and bounded stochastic disturbances. Under the additional Polyak-Lojasiewicz (PL) condition, the Lyapunov function converges linearly to a bounded neighborhood of the optimum, defined as the drift of the target direction field between consecutive iterates after parallel transport. To validate the framework's generativity, we derive three concrete instantiations of the quadruple, each exercising a distinct design axis, and show on geometric diagnostics, image classification benchmarks, and language modeling that their empirical behavior matches the theoretical predictions. The framework casts optimizer design as controller synthesis, complementing heuristic perspectives by providing a constructive template with provable guarantees.
Sources
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Adam: A Method for Stochastic Optimization
- Decoupled Weight Decay Regularization
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
- On the Convergence of Adam and Beyond
- Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis