When Bias Meets Trainability: Connecting Theories of Initialization
cs.LG, cs.AI, stat.ML
Submitted: 2025-05-17
Updated: 2026-02-28
Journal ref: International Conference on Learning Representations, volume 2026, pages 39872--39922 (2026)
Code: https://github.com/abassi98/igb_and_trainability
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: The statistical properties of deep neural networks (DNNs) at initialization play an important role to comprehend their trainability and the intrinsic architectural biases they possess before data
Terminology
Abstract
The statistical properties of deep neural networks (DNNs) at initialization play an important role to comprehend their trainability and the intrinsic architectural biases they possess before data exposure Well established mean field (MF) theories have uncovered that the distribution of parameters of randomly initialized networks strongly influences the behavior of the gradients, dictating whether they explode or vanish. Recent work has showed that untrained DNNs also manifest an initial guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof that links IGB to previous MF theories for a vast class of DNNs, showing that efficient learning is tightly connected to a network prejudice towards a specific class. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral.
Sources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Where You Place the Norm Matters: From Prejudiced to Neutral Initializations
- Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
- Adam: A Method for Stochastic Optimization
- Deep Information Propagation
- Tensor Programs II: Neural Tangent Kernel for Any Architecture
- Dynamical Mean-Field Theory of Self-Attention Neural Networks
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks