Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

arXiv:2605.22010 · stat.ML, cs.LG · Submitted 2026-05-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks".

Jane: Here's the gist of the paper. Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks considers how finite-width neural networks track their infinite-width mean-field counterparts by establishing non-asymptotic weak propagation of chaos that holds uniformly in time The key takeaway is that whenever the convergence rate of the meanfield, population-loss dynamics is faster than t − 2, we can attain a loss of ϵ with only poly(d/ϵ) neurons, training samples, and GD steps Setting and Preliminaries The work defines the setting for one-hidden layer neural networks parameterized by a distribution ρ ∈ P(S), where fρ(x):= Ew∼ρσ(x; w), and focuses on the excess squared loss LD(ρ) 1/3, is crucial for attaining a uniform-in-time bound This leads to the conclusion that whenever the mean-field dynamics has a convergence rate faster than 1/t squared (possibly after a burn-in time of order O(log(d))), we can attain a loss of ϵ in a network with only poly(d)/ϵ neurons Key Contributions and Implications The authors establish non-asymptotic weak PoC that holds uniformly in time, allowing the transfer of mean-field convergence guarantees at arbitrary convergence thresholds ϵ to the finite-width setting The result holds for slower rates Lt ≲ t − 2 but requires accounting for higher order terms that appear in the non-asymptotic analysis

AAM23: Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz.

Tom: Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics.

Jane: In The Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623.

Tom: PMLR, 2023

ABS13: Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter.

Paper summary: Tom: So, looking at "Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks," the core contribution is establishing a non-asymptotic weak propagation of chaos that holds uniformly in time <ref:2605.22010#pg1>.

Jane: That result allows for the transfer of mean field convergence guarantees at any convergence threshold epsilon to the finite width setting <ref:2605.22010#pg3>.

Lu: The paper shows that this is achievable by exploiting the convergence rate of the mean-field deterministic Wasserstein-gradient flow dynamics, which is a key technical move <ref:2605.22010#pg1>.

Meng: For us, the implication is that we can use these scaling laws to predict how fast our finite networks will learn on real traffic without having to run simulations for infinitely long <ref:2605.22010#pg3>.

Lalam: I’m pretty sure this paper confirms what we see in the literature about how the structure of a network impacts its convergence behavior under gradient descent <ref:2605.22010#pg3>.

Tom: So, to wrap up, it’s about showing that under specific conditions on the mean field dynamics—specifically a convergence rate faster than t to the power of minus two—we can achieve a loss of epsilon using only polynomial in d over epsilon neurons and training samples <ref:2605.22010#pg1>.

Jane: It’s about taking those deep theoretical guarantees from the infinite network level and making them applicable to the actual networks we build, which is what this paper does <ref:2605.22010#pg3>.

Conclusion: Tom: So we're wrapping up on "Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks."

Jane: It basically says that even for these smaller, shallow neural networks, they can track the behavior of their much bigger counterparts over time.

Tom: Right. The main authors are Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. They're talking about how chaos spreads in these finite-width systems.

Lu: What's important is that they prove this chaos propagation holds uniformly in time across the whole process of training.

Meng: So it means if we train a small network, its learning behavior is predictable even when it's still getting started, not just after it’s fully trained.

Jane: That sounds like a big deal for real-world deployment then. It suggests we don't have to wait forever to see if our training process is actually working as expected.

Tom: Yeah, and the paper points out that if the mean field dynamics converge faster than t to the power of minus two, you can get a small error with only some polynomial number of neurons and training samples.

Lu: That's the non-asymptotic part. It’s not just about what happens at infinity; it’s about what happens on our actual finite hardware.

Jane: So the big question then becomes how fast does that mean field actually converge in practice? That seems to be the next big thing we need to figure out for real learning systems.

Massachusetts Institute of Technology · Courant Institute School of Mathematics, Computing and Data Science, New York University

stat.ML, cs.LG

Submitted: 2026-05-21

Updated: 2026-11-11

Comments: 52 pages

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks considers how finite-width neural networks track their infinite-width mean-field counterparts by establishing non-asymptotic weak…

Key concepts

Weak Propagation of Chaos (PoC)
This is a mathematical concept describing how the randomness in an infinite system (the mean-field) propagates to finite systems (the neural network). Weak PoC means that the finite network's behavior closely follows the statistical properties of the large, infinite system, allowing for transfer of convergence guarantees.
Mean-Field Dynamics
This refers to the idealized, infinite-width version of a neural network where every neuron interacts with all others via an average signal. Analyzing this simpler model helps establish baseline convergence rates that are then adapted to the more complex, finite-width setting.
Uniform-in-Time Bound
This means the mathematical guarantee on performance (like loss reduction) remains valid regardless of how long the training process runs. It ensures that as training progresses, the network's error decreases predictably and consistently throughout its entire duration.
$L_t \lesssim t^{-2}$ Convergence Rate
This condition describes how quickly the mean-field model converges to its final state. If this convergence is fast enough (faster than $1/t^2$), the paper proves that the finite network can achieve a target error $\epsilon$ efficiently, requiring only poly($d/\epsilon$) resources.

Terminology

Summary

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks considers how finite-width neural networks track their infinite-width mean-field counterparts by establishing non-asymptotic weak propagation of chaos that holds uniformly in time The key takeaway is that whenever the convergence rate of the meanfield, population-loss dynamics is faster than t − 2, we can attain a loss of ϵ with only poly(d/ϵ) neurons, training samples, and GD steps

Setting and Preliminaries

The work defines the setting for one-hidden layer neural networks parameterized by a distribution ρ ∈ P(S), where fρ(x):= Ew∼ρσ(x; w), and focuses on the excess squared loss LD(ρ) 1/3, is crucial for attaining a uniform-in-time bound This leads to the conclusion that whenever the mean-field dynamics has a convergence rate faster than 1/t squared (possibly after a burn-in time of order O(log(d))), we can attain a loss of ϵ in a network with only poly(d)/ϵ neurons

Key Contributions and Implications

The authors establish non-asymptotic weak PoC that holds uniformly in time, allowing the transfer of mean-field convergence guarantees at arbitrary convergence thresholds ϵ to the finite-width setting The result holds for slower rates Lt ≲ t − 2 but requires accounting for higher order terms that appear in the non-asymptotic analysis

[AAM23] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. p. 0

Where the paper says it

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks Margalit Glasgow1 and Joan Bruna2 1Massachusetts Institute of Technology 2Courant Institute School of Mathematics, Computing and Data Science, New York University May 22, 2026 Abstract We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρˆmt to its infinite-width counterpart fρ MF t , which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρ MF t − fρˆmt ∥ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter.

In The Thirty Sixth Annual Conference on Learning Theory, pages 2552–2623. p. 0
Where the paper says it

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks Margalit Glasgow1 and Joan Bruna2 1Massachusetts Institute of Technology 2Courant Institute School of Mathematics, Computing and Data Science, New York University May 22, 2026 Abstract We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρˆmt to its infinite-width counterpart fρ MF t , which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρ MF t − fρˆmt ∥ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter.

PMLR, 2023

[ABS13] Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized gauss–seidel methods. Mathematical programming, 137(1):91–129, 2013

[AZLS19] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. p. 4

Where the paper says it

For finite times, several works [MMN18, MMM19, AAM22, MZD+23] have used a Grönwall-based approach to attain non-asymptotic PoC guarantees with tight dependencies on the dimension, and have applied these to show feature learning in neural networks. Finally, propagation of chaos has been studied in several other neural network settings, for example ResNets [Chi25] and deep transformers [GLPR25, CLLR26], where in the latter the relevant particle system is the embeddings of the m tokens in the context.

In International conference on machine learning, pages 242–252. p. 0
Where the paper says it

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks Margalit Glasgow1 and Joan Bruna2 1Massachusetts Institute of Technology 2Courant Institute School of Mathematics, Computing and Data Science, New York University May 22, 2026 Abstract We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρˆmt to its infinite-width counterpart fρ MF t , which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρ MF t − fρˆmt ∥ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter.

PMLR, 2019

[BAEVW26] Gerard Ben Arous, Murat Erdogdu, Nuri Mert Vural, and Denny Wu. Learning quadratic neural networks in high dimensions: Sgd dynamics and scaling laws. p. 1

Where the paper says it

3 Proof of Lemma 2. 41 B Supplemental Experimental Details for Misspecified Sobolev single-index model 44 1 Introduction Feature Learning in Shallow NNs: The defining characteristic of neural networks is their ability to automatically learn useful representations out of high-dimensional data, which can then be transferred to downstream tasks.

Advances in Neural Information Processing Systems, 38:146725–146812, 2026

[BAGJ21] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. p. 1

Where the paper says it

1) where σ is a non-linear activation function, and S ⊆ R d . 1) is an idealized model, far from the bleeding edge of modern architectures, its non-asymptotic learning guarantees under gradient descent remain largely open.

J. Mach. Learn. Res., 22:106–1, 2021

[BBPV23] Alberto Bietti, Joan Bruna, and Loucas Pillaud-Vivien. On learning gaussian multi-index models with gradient flow. p. 2

Where the paper says it

The focus of [GWB25] was on obtaining non-asymptotic bounds on ∥∆t∥ for problems with long burn-in times that improved upon the Grönwall estimate, specifically targeting systems where the required timescale to achieve small error is O(poly(d)), e. single-index models (SIMs) with large information exponent.

arXiv preprint arXiv:2310. p. 0
Where the paper says it

4 Proof of Preliminary Lemmas . 22 1 arXiv:2605.

19793, 2023

[Chi22a] Lénaïc Chizat. Mean-field langevin dynamics: Exponential convergence and annealing. p. 2

Where the paper says it

The latter question of small-ϵ convergence is the main motivation of this work. A common strategy to attain PoC at longer timescales is to add noise in the dynamics, leading to the socalled Mean-Field Langevin dynamics [HRSS19, NWS22, Chi22a].

Transactions on Machine Learning Research, 2022

[RS23] Matthew Rosenzweig and Sylvia Serfaty. Global-in-time mean-field convergence for singular riesz-type diffusive flows. p. 3

Where the paper says it

However, there are several rotatable exceptions: in [DT25], weak PoC is shown for the super-critical Kumamoto model which has continuum of invariant measures, and [RS23] attain a uniform-in-time PoC guarantee in a singular-interaction setting where the mean-field object converges at an inversely polynomial rate. We emphasize that in the context of shallow NN feature learning, one typically needs to anneal the diffusive dynamics, leading to exponential runtime [TS24, WMHC24].

The Annals of Applied Probability, 33(2):954–998, 2023 <ref:46

[RVE18] Grant M Rotskoff and Eric Vanden-Eijnden. Neural networks as Interacting Particle Systems: Asymptotic convexity of the Loss Landscape and Universal Scaling of the Approximation Error. p. 3

Where the paper says it

Mean-Field Langevin dynamics: [Chi22a, HRSS19, NWS22] studied the effect of adding a diffusive term in the Wasserstein Gradient Flow dynamics arising from Shallow NNs; in particular, by leveraging the aforementioned log-sobolev contraction tools, [CLRW24] prove uniform-in-time propagation of chaos for mean-field Langevin dynamics under functional convexity, with bounds in Wasserstein and relative entropy. [SNW22] prove a quantitative weak propagation-of-chaos result for mean-field gradient Langevin dynamics, with O(1/m) finite-particle discretization error uniformly over time, explicitly motivated by infinite-width two-layer neural networks; see also [WMHC24, MHWE24, TS24, Nit24].

arXiv preprint arXiv:1805. p. 0
Where the paper says it

4 Proof of Preliminary Lemmas . 22 1 arXiv:2605.

00915, 2018 <ref:31

[Szn91] Alain-Sol Sznitman. Topics in propagation of chaos. p. 0

Where the paper says it

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks Margalit Glasgow1 and Joan Bruna2 1Massachusetts Institute of Technology 2Courant Institute School of Mathematics, Computing and Data Science, New York University May 22, 2026 Abstract We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρˆmt to its infinite-width counterpart fρ MF t , which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρ MF t − fρˆmt ∥ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter.

Lecture notes in mathematics, pages 165–251, 1991 <ref:38

[TS24] Shokichi Takakura and Taiji Suzuki. Mean-field analysis on two-layer neural networks from a kernel perspective. p. 3

Where the paper says it

Mean-Field Langevin dynamics: [Chi22a, HRSS19, NWS22] studied the effect of adding a diffusive term in the Wasserstein Gradient Flow dynamics arising from Shallow NNs; in particular, by leveraging the aforementioned log-sobolev contraction tools, [CLRW24] prove uniform-in-time propagation of chaos for mean-field Langevin dynamics under functional convexity, with bounds in Wasserstein and relative entropy. [SNW22] prove a quantitative weak propagation-of-chaos result for mean-field gradient Langevin dynamics, with O(1/m) finite-particle discretization error uniformly over time, explicitly motivated by infinite-width two-layer neural networks; see also [WMHC24, MHWE24, TS24, Nit24].

arXiv preprint arXiv:2403. p. 0
Where the paper says it

4 Proof of Preliminary Lemmas . 22 1 arXiv:2605.

14917, 2024 <ref:31

[VW23] AW van der Vaart and Jon A Wellner. Empirical processes. p. 1

Where the paper says it

Thanks to the permutation symmetry of neurons in (1. 1), the gradient-flow training dynamics admit an Eulerian description in terms of the empirical measure ρˆ m = 1 m Pm i=1 δwi ∈ P(S).

In Weak Convergence and Empirical Processes: With Applications to Statistics, pages 127–384. p. 0
Where the paper says it

Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradientflow dynamics.

Springer, 2023 <ref:31

[WMHC24] Guillaume Wang, Alireza Mousavi-Hosseini, and Lénaïc Chizat. Mean-field langevin dynamics for signed measures via a bilevel approach. p. 2

Where the paper says it

A common strategy to attain PoC at longer timescales is to add noise in the dynamics, leading to the socalled Mean-Field Langevin dynamics [HRSS19, NWS22, Chi22a]. The diffusion term creates a contraction in the Wasserstein metric, quantified via a uniform logarithmic Sobolev inequality (LSI), leading to uniformin-time propagation of chaos [CLRW24, SWN23, KZC+24, Nit24].

Advances in Neural Information Processing Systems, 37:35165–35224 <ref:31

[ZCZG20] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. p. 0

Where the paper says it

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks Margalit Glasgow1 and Joan Bruna2 1Massachusetts Institute of Technology 2Courant Institute School of Mathematics, Computing and Data Science, New York University May 22, 2026 Abstract We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network fρˆmt to its infinite-width counterpart fρ MF t , which evolves in the mean-field dynamics. While constant-time horizon bounds for ∥fρ MF t − fρˆmt ∥ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Uniform in time propagation of chaos (PoC) guarantees for finite-width neural networks allow for attaining a loss of epsilon using only poly(d)/epsilon resources, including neurons, training samples, and gradient steps when the mean-field convergence rate is faster than t negative 2.

  2. The system can achieve uniform in time bounds on function error by exploiting the convergence rate of the mean-field deterministic Wasserstein gradient flow dynamics instead of relying on constant-time horizon bounds that require strong convexity or logarithmic Sobolev inequalities.

  3. The improved analysis allows for achieving population loss epsilon with only poly(d)/epsilon neurons, training samples, and GD steps whenever the convergence rate of the mean-field dynamics is faster than t negative 2, even in noiseless settings without assumptions on the geometry of the landscape near the optimum.

  4. The method can be applied to other forms of discretization, including finite number of samples and time discretization, extending its applicability beyond just finite neurons or standard gradient descent steps.

  5. For specific loss decay rates faster than t negative 2 but slower than t negative 6, the system can attain a loss of epsilon with only poly(d)/epsilon resources at a time related to the mean-field excess MSE loss at time t star m approximately m over log d.

Abstract

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network f ρ t m to its infinite-width counterpart f ρ t MF, which evolves in the mean-field dynamics. While constant-time horizon bounds for |f ρ t MF - f ρ t m| may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by L t the mean-field excess MSE loss at time t and m the number of neurons, under standard regularity assumptions and the condition integral 0 infinity L t 1/2 dt =O(d), we obtain the uniform in time bound |f ρ t MF- f ρ t m| squared poly(d) m- (1,c/6) whenever L t t-c. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than t-2, we can attain a loss of ε with only poly(d/ε) neurons, training samples, and GD steps.

Sources

Related papers