Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
The study systematically tested 3,938 model variants by injecting 25 different learning rate schedules into 30 diverse neural networks to find optimal training recipes. Results show that CyclicLR is best for mobile and convolutional models, while CosineAnnealingWarmRestarts suits transformer-based architectures. This proves that scheduler choice must depend on the specific model type for best performance.