The Spectral Neuron
Alex Shtoff
stat.ML, cs.LG
Submitted: 2026-08-16
Updated: 2026-08-18
Project page: https://www.kaggle.com/c/criteo-display-ad-challenge
License: http://creativecommons.org/licenses/by/4.0/
The gist: As machine learned models increase in complexity and expressive power, features of simpler models, such as interpretability and control over the shape of the modeled function are lost.
Terminology
Abstract
As machine learned models increase in complexity and expressive power, features of simpler models, such as interpretability and control over the shape of the modeled function are lost. On the one edge of the spectrum we have simple linear models are transparent and possess good interpretability and explainability properties, but have a limited expressive power. On the other edge we have neural networks, that have expressive power that improves with scaling, but are mostly opaque. In this work we develop the spectral neuron concept: a scalar model given by f= lambda k (A 0+ sum i=1 n x i A i), with learned real symmetric matrices A 0,,A n. The input enters the model through an affine matrix function, but the prediction is obtained by reading one of its eigenvalues. Thus, the model is nonlinear, but the source of nonlinearity is still mathematically explicit. This gives us a useful middle ground: the model can become more expressive as the matrix dimension grows, while retaining a degree of structural interpretability through the learned matrices. For example, extremal eigenvalues yield convex or concave functions, semidefinite constraints on the coefficient matrices impose monotonicity, and the associated eigenspaces characterize local feature sensitivity. We study the robustness, structural interpretability, and shape-control properties of this model family, and then test whether it can be learned and scaled in practice. We develop a systematic study of this model family, bringing together spectral results from several mathematical literatures to characterize its expressivity, robustness, interpretability, and shape-control properties.
Sources
- Squareplus: A Softplus-Like Algebraic Rectifier
- HyperNetworks
- Adam: A Method for Stochastic Optimization
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches