Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations
math.OC, cs.LG
Submitted: 2025-10-28
Updated: 2025-10-28
Sources
- Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning
- The critical locus of overparameterized neural networks
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks
- Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
- First-order methods almost always avoid saddle points: the case of vanishing step-sizes
- On the saddle point problem for non-convex optimization
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Second-Order Optimization with Asynchronous Curvature
- Online Control via Counterfactual Tracking
- Asynchronous Replanning in Two Population Linear Quadratic Mean Field Games: Information Requirements and Stability