In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
The paper proves that In-Context Learning (ICL) is a form of Bayesian inference within meta-learning. It decomposes ICL risk into two parts: the Bayes Gap, measuring approximation error to the optimal predictor, and Posterior Variance, representing task uncertainty. This framework allows researchers to analytically understand how pre-trained models generalize and how context length affects performance.