Generalization Analysis of Transformers in Distribution Regression

arXiv:2606.29256 · stat.ML, cs.LG · Submitted 2026-06-28 · Read on arXiv

Peilin Liu, Ding-Xuan Zhou

stat.ML, cs.LG

Submitted: 2026-06-28

Journal ref: Neural Computation 37(2):260-293, 2025

DOI: 10.1162/neco_a_01726

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Transformer — 79× in this paper · explained in 2 episodes
• Generalization — 29× in this paper · explained in 14 episodes
• Transformers — 25× in this paper · explained in 2 episodes
• Transformer Encoder — 23× in this paper · explained in 2 episodes
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Expressivity — 5× in this paper · explained in 2 episodes
• Hilbert Space — 5× in this paper · explained in 2 episodes
• Universality — 5× in this paper · explained in 2 episodes
• Attention Mechanism — 4× in this paper · explained in 3 episodes
• Composition — 4× in this paper · explained in 2 episodes
• Transfer Learning — 4× in this paper · explained in 16 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Decoder — 2× in this paper · explained in 2 episodes
• Embeddings — 2× in this paper · explained in 2 episodes
• Error Decomposition — 2× in this paper · explained in 4 episodes
• Generalization Bounds — 2× in this paper · explained in 2 episodes
• Inductive Bias — 2× in this paper · explained in 3 episodes
• Language Models (LLMs) — 2× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 9 episodes
• Parameter-Efficient Fine-Tuning — 2× in this paper · explained in 3 episodes
• Transformer Architecture — 2× in this paper · explained in 4 episodes
• Wasserstein Distance — 2× in this paper · explained in 6 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Convolutional Neural Networks (CNNs) — 1× in this paper · explained in 3 episodes
• Fixed Point — 1× in this paper · explained in 2 episodes
• Gating Network — 1× in this paper · explained in 2 episodes
• Generalization Capabilities — 1× in this paper · explained in 2 episodes
• In-Context Learning — 1× in this paper · explained in 7 episodes
• Inverse Problem — 1× in this paper · explained in 2 episodes
• Kernel Ridge Regression — 1× in this paper · explained in 2 episodes
• Linear Complexity — 1× in this paper · explained in 2 episodes
• Maximum Mean Discrepancy (MMD) — 1× in this paper · explained in 3 episodes
• Mixture of Experts — 1× in this paper · explained in 2 episodes
• Mixture of Experts (MoE) — 1× in this paper · explained in 6 episodes
• Multimodal Inputs — 1× in this paper · explained in 2 episodes
• Optimal Transport — 1× in this paper · explained in 5 episodes
• Papers — 1× in this paper · explained in 3 episodes
• ReLU Network — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes

Sources

• Learning Factored Representations in a Deep Mixture of Experts
• Mixtral of Experts
• Learning Theory of Distribution Regression with Neural Networks
• Neural Operators for Nonlinear Functionals on RKHS

Related papers

• Behavior of prediction performance metrics with rare events
• Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
• A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
• Scalable Posterior Uncertainty for Flexible Density-Based Clustering
• A Transport Geometry of Benign Overfitting
• Efficient quadratic entropy with distance sketches