Attention is Just Another Name for Coupling? A Fast-Slow ODE Perspective on Hierarchical Pretraining
Zhengyuan Gao
stat.ML, cs.AI, cs.LG
Submitted: 2026-06-15
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: We re-interpret Transformer pretraining as a fast-slow, singularly perturbed flow along depth, with untied weights as its non-autonomous feature.
Terminology
Abstract
We re-interpret Transformer pretraining as a fast-slow, singularly perturbed flow along depth, with untied weights as its non-autonomous feature. The linearised dynamics is a depth-ordered product of layer maps. Along a token-homogeneous reference trajectory, the linearised layer factorises along the eigenbasis of a frozen attention kernel. Past a computable saturation depth, the flow factors through the block coarse-graining -- in other words, running the layers is running the coarse variable, dually. Weight perturbations supported on the decaying bundle move neither the persistent component of the distinguished trajectory nor the frozen kernel to first order, so the framework partitions parameter space into visible and invisible directions, with the cross-block coupling of the slow path sitting entirely on the visible side. How large a gate the slow path can carry is bounded by a stability margin. On the data side: if block emissions follow an exponential family, block-mean pooling captures all the information the slow path can use; but if neighbouring blocks carry no shared structure, no cross-block channel can help the prediction, and the gate amplitude is invisible in the prediction risk. Stability delimits what the architecture may do; the data decides what it will.
Sources
- Layer Normalization
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- A mathematical perspective on Transformers
- Gaussian Error Linear Units (GELUs)
- Scaling Laws for Neural Language Models
- MiniMax Sparse Attention
- Parcae: Scaling Laws For Stable Looped Language Models
- Hierarchical Reasoning Model
- mHC: Manifold-Constrained Hyper-Connections
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches