Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

arXiv:2607.02003 · stat.ML, cs.LG · Submitted 2026-07-02 · Read on arXiv

Matej Benko, Pierre Bousquet, Iwona Chlebicka, Błażej Miasojedow

stat.ML, cs.LG

Submitted: 2026-07-02

License: http://creativecommons.org/licenses/by/4.0/

The gist: Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics.

Terminology

Abstract

Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of lambda-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost C cubed regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of 1/alpha relative to the regularization parameter, and prove that finite-width networks of size N achieve the continuum optimum at an O(1/N) rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.

Sources

Related papers