On the Depth Transfer of Neural Optimizers via L-Smoothness
The paper investigates how deep learning models transfer performance across different depths by analyzing L-smoothness in residual networks. The authors show that inverse-depth scaling provides a geometric guarantee for depth-uniform sensitivity and smoothness bounds, suggesting that optimal learning rates remain stable even when scaling model depth significantly. This allows for efficient hyperparameter transfer across architectures.