An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
Cheng Huan, Hongwei Yuan
stat.ML, cs.LG, math.OC
Submitted: 2026-07-20
Comments: 41 pages, 4 figures
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Continuum Attention for Neural Operators
- Neural Ordinary Differential Equations
- Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
- Robust Learning with Jacobian Regularization
- A First-Order Mean Field Control Analysis of Transformer Layers under Cross-Entropy Training
- A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training
- On uniform-in-time diffusion approximation for stochastic gradient descent
- Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Attention Is All You Need
- Efficient Streaming Language Models with Attention Sinks
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches