How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Julius Girardin, Emanuele Troiani, Yizhou Xu, Vittorio Erba, Florent Krzakala, Lenka Zdeborová
cs.LG, cond-mat.dis-nn, cs.AI, stat.ML
Submitted: 2026-06-26
Code: https://github.com/SPOC-group/WidthQuadraticNetworks
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
- Deriving Neural Scaling Laws from the statistics of natural language
- ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit
- Optimal scaling laws in learning hierarchical multi-index models
- Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime
- Stationary Points of Shallow Neural Networks with Quadratic Activation Function
- Scaling Laws for Neural Language Models
- A Solvable Model of Neural Scaling Laws
- High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks
- A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator
- Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formula
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks