Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
cs.LG
Submitted: 2026-04-13
Updated: 2026-11-05
Code: https://github.com/comiam/dag-hesse
Terminology
Sources
- The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
- The Shattered Gradients Problem: If resnets are the answer, then what is the question?
- Local properties of neural networks through the lens of layer-wise Hessians
- Practical Gauss-Newton Optimisation for Deep Learning
- BackPACK: Packing more into backprop
- ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Shampoo: Preconditioned Stochastic Tensor Optimization
- Complexity of Linear Regions in Deep Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
- New insights and perspectives on the natural gradient method
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Spectral Normalization for Generative Adversarial Networks
- On the difficulty of training Recurrent Neural Networks
- Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
- Exponential expressivity in deep neural networks through transient chaos
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis