Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets
Javier Maass, Lénaïc Chizat
stat.ML, cs.LG, math.PR
Submitted: 2026-07-18
Code: https://github.com/xavimaass/dropout_and_ram_2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning.
Terminology
Abstract
Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.
Sources
- ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit
- The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram
- Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime
- Genericity of Polyak-Lojasiewicz Inequalities for Entropic Mean-Field Neural ODEs
- Shake-Shake regularization
- Improving neural networks by preventing co-adaptation of feature detectors
- A convergence result of a continuous model of deep learning via a \L{}ojasiewicz--Simon inequality
- Mean-Field Neural ODEs via Relaxed Optimal Control
- On Surprising Effectiveness of Masking Updates in Adaptive Optimizers
- Stochastic Particle Gradient Descent for Infinite Ensembles
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Galactica: A Large Language Model for Science
- Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches