Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
cs.LG, cs.AI
Submitted: 2023-11-29
Updated: 2025-06-12
Comments: 24 pages, 17 figures, Poster at ICML 2025
Journal ref: Proceedings of the 42nd International Conference on Machine Learning, PMLR Vol. 267, pp. 44175-44198 (ICML 2025)
Code: https://github.com/NREL/triangle_net
License: http://creativecommons.org/licenses/by/4.0/
The gist: In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth.
Terminology
Abstract
In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth. However, this is highly unlikely to happen when the initial parameters are sampled randomly, which therefore often leads to the use of networks that are unnecessarily large. To address this problem, we introduce a novel parameterization of the network that restricts its weights so that a depth d network produces exactly 2 d linear regions at initialization and maintains those regions throughout training under the parameterization. This approach allows us to learn approximations of convex, one dimensional functions that are several orders of magnitude more accurate than their randomly initialized counterparts. We further demonstrate a preliminary extension of our construction to multidimensional and non-convex functions, allowing the technique to replace traditional dense layers in various architectures.
Sources
- When Deep Learning Meets Polyhedral Theory: A Survey
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Why Deep Neural Networks for Function Approximation?
- KAN: Kolmogorov-Arnold Networks
- Unsupervised Representation Learning via Neural Activation Coding
- Bounding and Counting Linear Regions of Deep Neural Networks
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis