Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks
The paper introduces a novel weight initialization method for deep ReLU networks by solving an optimization problem on the Stiefel manifold. This method ensures weights are semi-orthogonal, preserving scale and calibrating pre-activation statistics from the start. Theoretically, this prevents dying ReLUs and stabilizes training in very deep architectures, showing superior performance across various datasets compared to standard initializations.