Profile-aware assessment of block-diagonal Hessian approximations in neural networks
cs.LG
Submitted: 2026-10-31
Updated: 2026-11-18
Terminology
Sources
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Shampoo: Preconditioned Stochastic Tensor Optimization
- Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
- Towards Quantifying the Hessian Structure of Neural Networks
- Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions
- Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks
- Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
- Practical Gauss-Newton Optimisation for Deep Learning
- BackPACK: Packing more into backprop
- Two-Level K-FAC Preconditioning for Deep Learning
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- A New Perspective on Shampoo's Preconditioner
- New insights and perspectives on the natural gradient method
- The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
- Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
- Adaptive Gradient Methods at the Edge of Stability
- A Defense of the Quadratic Model
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis