Quantitative Gaussian-Process limits of Tensor Programs
Andrea Agazzi, Eloy Mosig García, Dario Trevisan
cs.LG, math.PR, stat.ML
Submitted: 2026-07-07
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study the infinite-width Gaussian-process limit of random neural networks through the lens of tensor programs, and we provide a quantitative convergence theory in Wasserstein distance.
Terminology
Abstract
We study the infinite-width Gaussian-process limit of random neural networks through the lens of tensor programs, and we provide a quantitative convergence theory in Wasserstein distance. Our main result gives explicit finite-width error bounds, of order inverse square-root of the widths between finite-network executions and their Gaussian-process limits. The framework is architecture-agnostic and covers feed-forward models together with weight-sharing schemes relevant for recurrent and transformer-type architectures.
Sources
- Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
- Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalities
- Wide neural networks with general weights: convergence rate and explicit dependence on the hyper-parameters
- Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle
- Gaussian Process Behaviour in Wide Deep Neural Networks
- Infinite attention: NNGP and NTK for deep attention networks
- Homogenized Transformers
- Quantitative convergence of trained single layer neural networks to Gaussian processes
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Tensor Programs II: Neural Tangent Kernel for Any Architecture
- Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
- Tensor Programs III: Neural Matrix Laws
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks