Theory on Attention Dynamics for Out-of-Distribution In-Context Learning

arXiv:2609.36448 · cs.LG · Submitted 2026-09-29 · Read on arXiv

cs.LG

Submitted: 2026-09-29

Updated: 2026-09-29

Terminology

• In-Context Learning — 1× in this paper · explained in 4 episodes

Sources

• Transformers learn to implement preconditioned gradient descent for in-context learning
• In-Context Learning through the Bayesian Prism
• A Closer Look at In-Context Learning under Distribution Shifts
• What learning algorithm is in-context learning? Investigations with linear models
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
• In-Context Learning with Long-Context Models: An In-Depth Exploration
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
• In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
• Looped Transformers as Programmable Computers
• When can in-context learning generalize out of task distribution?
• How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
• A Theory of Emergent In-Context Learning as Implicit Structure Induction
• Understanding Emergent In-Context Learning from a Kernel Regression Perspective
• Pretrained Transformers Improve Out-of-Distribution Robustness
• Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
• An Information-Theoretic Analysis of In-Context Learning
• A Latent Space Theory for Emergent Abilities in Large Language Models
• In-context Learning for Mixture of Linear Regressions: Existence, Generalization and Training Dynamics

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks