Sharpness of Minima in Deep Matrix Factorization
stat.ML, cs.LG, math.OC
Submitted: 2025-09-30
Updated: 2026-10-28
Comments: 23 pages, 5 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Understanding the loss-landscape geometry near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and…
Terminology
Abstract
Understanding the loss-landscape geometry near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity for characterizing this geometry is the maximum eigenvalue of the Hessian of the loss. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in deep matrix factorization problems, resolving an open question posed by Mulayoff and Michaeli [2020]. This expression reveals a fundamental property of the loss landscape in deep matrix factorization: It is sufficient, but not necessary, for flatness that, across all layers, the product of the spectral norms of the left and right intermediate factors equals a constant determined by the depth and the spectral norm of the target matrix. Most notably, in both depth-2 matrix factorization and deep overparameterized scalar factorization, we show that this condition is both necessary and sufficient for flatness, which implies that flat minima are spectral-norm balanced and not necessarily balanced in depth-2 matrix factorization. To complement our theory, we provide empirical verification, relying on our exact expression, of an escape phenomenon during gradient-based training near a minimizer of a deep matrix factorization problem.
Sources
- Qualitatively characterizing neural network optimization problems
- On the geometry of flat minima
- The large learning rate phase of deep learning: the catapult mechanism
- In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference