Optimal In-context Adaptivity and Distributional Robustness of Transformers
stat.ML, cs.LG, math.ST, stat.TH
Submitted: 2025-10-27
Updated: 2026-11-06
Comments: 45 pages, 6 figures. Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study in-context learning problems where a Transformer is pretrained on tasks drawn from a mixture distribution π= sum α in A λ α π α, called the pretraining prior…
Terminology
Abstract
We study in-context learning problems where a Transformer is pretrained on tasks drawn from a mixture distribution π= sum α in A λ α π α, called the pretraining prior, in which each mixture component π α is a distribution on tasks of a specific difficulty level indexed by α. Our goal is to understand the performance of the pretrained Transformer when evaluated on a different test distribution μ, consisting of tasks of fixed difficulty β in A, and with potential distribution shift relative to π β, subject to the chi-squared divergence χ 2(μ,π β) being at most κ. In particular, we consider nonparametric regression problems with random smoothness, and multi-index models with both random smoothness and random effective dimension. We prove that a large Transformer pretrained on sufficient data achieves the optimal rate of convergence corresponding to the difficulty level β, uniformly over test distributions μ in the chi-squared divergence ball. Thus, the pretrained Transformer is able to achieve faster rates of convergence on easier tasks and is robust to distribution shift at test time. Finally, we prove that even if an estimator had access to the test distribution μ, the convergence rate of its expected risk over μ could not be faster than that of our pretrained Transformers, thereby providing a more appropriate optimality guarantee than minimax lower bounds.
Sources
- Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners
- Test time training enhances in-context learning of nonlinear functions
- Deep learning with missing data
- In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
- An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference