A First-Order Mean Field Control Analysis of Transformer Layers under Cross-Entropy Training
Cheng Huan, Hongwei Yuan
math.OC, math.DS, stat.ML
Submitted: 2026-06-22
Comments: 44 pages
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Neural Ordinary Differential Equations
- Deep Residual Learning and PDEs on Manifold
- A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training
- mHC: Manifold-Constrained Hyper-Connections
- Improving neural networks by preventing co-adaptation of feature detectors
- Dropout Training as Adaptive Regularization
- Qualitatively characterizing neural network optimization problems
- The Loss Surfaces of Multilayer Networks
- Attention Is All You Need
Related papers
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
- Incremental Learning in Mirror Flows
- Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning
- Lower Bounds for Randomized Linear-Oracle Optimization over Strongly Convex Sets
- Second-Order Optimization with Asynchronous Curvature