Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective
The study models Stochastic Gradient Descent (SGD) as Langevin dynamics to find an optimal weight initialization variance for deep neural networks. By comparing the initial distribution to the quasi-stationary distribution using Kullback-Leibler divergence, the authors derive a mathematical criterion. This criterion suggests that initializing weights with a specific variance minimizes expected loss and improves training performance compared to standard methods like He initialization.