Attention's forward pass and Frank-Wolfe
math.OC
Submitted: 2025-08-13
Updated: 2026-11-17
Code: https://github.com/borjanG/2025-transformers-frank-wolfe
Terminology
Sources
- Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
- Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
- Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse
- Perceptrons and localization of attention's mean-field landscape
- Breaking BERT: Evaluating and Optimizing Sparsified Attention
- A Unified Perspective on the Dynamics of Deep Transformers
- Critical attention scaling in long-context transformers
- Quantitative Clustering in Mean-Field Transformer Models
- Transformer Circuits Can Realize Clustering Algorithms
- Synchronization on circles and spheres with nonlinear interactions
- Kinetic theory for Transformers and the lost-in-the-middle phenomenon
- Large-time asymptotics in deep learning
- Dynamic metastability in the self-attention model
- Measure-to-measure interpolation using Transformers
- On the number of modes of Gaussian kernel density estimators
- Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics
- Homogenized Transformers
- Convergence Rate of Frank-Wolfe for Non-Convex Objectives
- The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
- Synchronization of mean-field models on the circle
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise