Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry
cs.LG, cond-mat.dis-nn, cond-mat.stat-mech, cs.AI
Submitted: 2026-09-02
Updated: 2026-09-27
Comments: 49 pages, 9 in the main text, v2 corrected theoretical results for Adam with verified additional experiments in the appendix
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks.
Terminology
Abstract
We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which nested architectural symmetries force subnetworks to merge in discrete blocks rather than by single-edge attachment. These structural transitions register as variance spikes in a macroscopic order parameter echoing physical phase transitions. We further state sufficient conditions under which the trapping argument carries over to Adam and AdamW under heavy-tailed gradient noise and measure them on a trained Transformer.
Sources
- Anomalous diffusion dynamics of learning in deep neural networks
- Dropout Neural Network Training Viewed from a Percolation Perspective
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Scaling Laws for Neural Language Models
- Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
- Phase Diagram of Initial Condensation for Two-layer Neural Networks
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks