Why Transformers Struggle with Distribution-Independent In-Context Learning
stat.ML, cs.AI, cs.LG
Submitted: 2025-02-05
Updated: 2026-11-09
Code: https://github.com/omyokun/icl-polynomials
Terminology
Sources
- What learning algorithm is in-context learning? Investigations with linear models
- Layer Normalization
- Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions
- A Survey on In-context Learning
- Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
- How Well Can Transformers Emulate In-context Newton's Method?
- Re-examining learning linear functions in context
- SSA: Improving Performance With a Better Scoring Function
- In-context Learning and Induction Heads
- In-Context Learning through the Bayesian Prism
- How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
- An Explanation of In-context Learning as Implicit Bayesian Inference
- What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference