Parallel Layer Normalization for Universal Approximation
The research investigates how grouping Layer Normalization (LN) units affects a neural network's ability to approximate functions. It introduces Parallel Layer Normalization (PLN-Nets), where fixed-dimensional LN units are applied in parallel between affine layers. The findings show that while shallow LN-Nets are limited, PLN-Nets achieve universal approximation for univariate functions when the number of parallel groups grows.