Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework
Junwen Qiu, Bohao Ma, Andre Milzarek, Junyu Zhang
math.OC, cs.LG, math.DS, stat.ML
Submitted: 2026-08-04
Comments: 39 pages
License: http://creativecommons.org/licenses/by/4.0/
The gist: Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation.
Terminology
Abstract
Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation. This condition gives a direct way to rule out convergence to strict saddles, but it also oversimplifies the actual noise structure, and does not match many stochastic optimization regimes. In overparameterized or interpolation models, the noise may vanish near stationarity. In finite-sum problems, the stochastic gradient noise may lie in a low-dimensional, data-dependent subspace. In these (common) scenarios, UE is naturally not satisfied. In this paper, we prove an abstract almost sure avoidance theorem for stochastic recursions without UE. The theorem replaces UE-type requirements by verifiable pathwise conditions. In applications, these conditions follow, e.g., from local smoothness and finite-moment assumptions under standard i.i.d. sampling, or from the finite-sum structure under without-replacement sampling. Since the stochastically sampled maps generally do not share a fixed point, the celebrated center-stable manifold argument used in deterministic analyses is not directly applicable. Instead, we use a path-dependent change of variables together with a pathwise Lyapunov--Perron-based proof strategy. As applications, we obtain strict saddle avoidance for stochastic mirror descent (including SGD) and for random reshuffling. For nonsmooth composite objectives, we prove avoidance results for a proximal-type stochastic gradient method. Combining these insights with suitable iterate convergence guarantees, this allows establishing convergence to local minimizers of the original objective function.
Sources
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- On the Trajectories of SGD Without Replacement
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Optimality, identifiability, and sensitivity
- A fresh look into variational analysis of $\mathcal C^2$-partly smooth functions
- On the local convergence of the semismooth Newton method for composite optimization
- On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
- Proximal random reshuffling under local Lipschitz continuity
- Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumption
- A non-autonomous center-stable set theorem for saddle avoidance in optimization
- Variational Properties of Decomposable Functions. Part I: Strict Epi-Calculus and Applications
- A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Properties
- Random Reshuffling with Momentum: Complexity Bounds and Last-iterate Convergence
- How Good is SGD with Random Shuffling?
- Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition
- When Are Nonconvex Problems Not Scary?
- A Geometric Analysis of Phase Retrieval
- High Probability Guarantees for Random Reshuffling
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning
- Lower Bounds for Randomized Linear-Oracle Optimization over Strongly Convex Sets
- Second-Order Optimization with Asynchronous Curvature