On the Depth Transfer of Neural Optimizers via L-Smoothness

arXiv:2610.14344 · cs.LG, math.OC · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "On the Depth Transfer of Neural Optimizers via L-Smoothness".

Jane: The gist Scaling deep learning models efficiently requires predictable performance transfer across compute regimes.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let’s talk about who wrote this. The paper is "On the Depth Transfer of Neural Optimizers via L-Smoothness," written by Wong, Xu, Li, and Lu. It’s a team from universities like Chicago, Peking, and UBC.

Jane: They are bringing together mathematical theory with practical deep learning application here. What they are doing is looking at how the Lipschitz and L-smoothness constants of the loss scale as you go deeper into a residual network.

Lu: The authors establish that because these bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks—including branches of block depth greater than or equal to two such as transformer and ResNet blocks—without needing assumptions about initialization or correlations between layer weights.

Meng: So it’s not just for one specific type of network structure; it’s general enough for most deep architectures? That sounds promising because we don't want to build a whole new framework just for GPT-three versus BERT, right <ref:2610.14344#pg2>?

Lalam: Lalam confirms that the mechanism they found is robust across different residual branch depths. The core idea is that the scaling relationship between those smoothness constants dictates a specific learning rate scale without needing extra depth-dependent curvature corrections for updates bounded in the prescribed norm.

The paper's summary: Tom: So, summarizing what they found, it’s about deriving an inverse-depth residual branch scaling, which is defined as alpha equals theta of one over K. They show that this scaling is sufficient to ensure that the optimal learning rate stays aligned across depths.

Jane: The paper shows numerical results confirming this transfer. For example, they look at GPT-two models trained with Muon and MOGA, where all depths from two up to one hundred select nearly the same optimal learning rate when measured by final-epoch training loss <ref:2610.14344#pg2>.

Lu: On CIFAR-one hundred ResNets, they found that for MOGA settings with p equals one point five and p equals two point five, all three evaluated settings share the same best tested learning rate across ResNets with eight, sixteen, and thirty-two residual blocks. AdamW also retains a common low-loss region, although its best tested rate shifts toward smaller values at the largest GPT-two depths <ref:2610.14344#pg2>.

Meng: That’s a lot of data points showing stability across different architectures and optimizers. But what’s the practical implication for someone trying to tune hyperparameters? Does it mean less trial and error?

Lalam: Lalam says that this empirical evidence supports the idea that "Muon and MOGA exhibit a wide stable learning rate area that transfers as depth increases." So, you can pick one setting based on a smaller model and expect it to be good for a much larger one.

The paper's improvements: Tom: The authors suggest several improvements based on their findings. First, they provide worst-case guarantees for depth-uniform sensitivity and smoothness bounds under this scaling prescription. This is a big deal because it gives us theoretical backing for the empirical transfer we saw.

Jane: They show that for MLP residual blocks, Proposition one establishes that the network satisfies Assumption two in a localized sense, meaning the W-regularity conditions hold on the ball where iterates are bounded, and this extends to attention blocks too if you use row or column normalization of Wq, Wk, Wv and Wo <ref:2610.14344#pg2>.

Lu: The key improvement is moving away from architecture-specific heuristic analyses. They provide a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA.

Meng: So they are showing us *why* this works structurally, not just showing us that it happened on a few models? That’s what we need for real deployment pipelines.

Lalam: Lalam notes that they also show that the final results suggest that the optimal learning rates across seven depths still stay within a tight region. This indicates strong stable learning rate transfer. Plus, they confirm their scaling prescriptions work empirically for different architectures, testing GPT-two ResNet, and residual MLPs for their baseline <ref:2610.14344#pg2>.

Conclusion: Tom: To wrap up the paper "On the Depth Transfer of Neural Optimizers via L-Smoothness," the main implication is that we have a geometric account of depth scaling. They proved that inverse-depth residual scaling is a worst-case guarantee for depth uniform sensitivity and smoothness bounds.

Jane: This means we can effectively transfer learning rates from small models to much deeper models, reducing the computation budget you need to spend on hyperparameter tuning during training. It gives us a principled way to scale step sizes without needing extra depth-dependent curvature corrections for updates bounded in the prescribed norm.

Lu: The analysis establishes inverse-depth residual scaling as a worst case guarantee for depth uniform sensitivity and smoothness bounds, which supports learning rate transfer across the evaluated architectures and optimizer geometries. It’s about providing a geometric explanation of that scaling mechanism.

Meng: For production, this means we can rely on these scaling rules to predict how training will behave when we scale up our models, instead of having to re-tune everything from scratch for every new depth.

Lalam: Lalam is confident that the final results show that the optimal learning rates across seven depths still stay within a tight region, which really points to strong stable learning rate transfer. The paper establishes this scaling prescription works empirically for GPT-two ResNet, and residual MLPs <ref:2610.14344#pg2>.

Tom: So, in short, the paper "On the Depth Transfer of Neural Optimizers via L-Smoothness" gives us a way to scale learning rates predictably across model depth using the L-smoothness framework.

Jane: It’s a solid piece of work that connects abstract math about smoothness constants to concrete results about stable training dynamics in real deep learning models.

Maximilian Wong, Chenyu Xu, Jiajin Li, Yiping Lu

University of Chicago · Peking University · Sauder School of Business University of British Columbia

cs.LG, math.OC

Submitted: 2026-10-09

Updated: 2026-10-09

Project page: https://skylion007.github.io/OpenWebTextCorpus

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist Scaling deep learning models efficiently requires predictable performance transfer across compute regimes.

Key concepts

L-smoothness framework: This framework is used to understand how the Lipschitz and L-smoothness constants of network layers change as the network gets deeper (depth K). It provides a mathematical way to describe the curvature penalty during optimization, which is crucial for predicting performance transfer across different model sizes.
Inverse-depth residual scaling: This is a proposed scaling method for learning rates that relates them inversely to the depth of the residual network. The paper proves this scaling provides worst-case guarantees for depth-uniform sensitivity and smoothness bounds, suggesting it's a robust way to handle deep architectures.
Depth-uniform sensitivity: This refers to the property where a network's behavior or its sensitivity to changes remains relatively consistent regardless of how many layers it has. The paper establishes that their proposed scaling method ensures this property holds under worst-case conditions for residual networks.
Learning rate transfer: This is the phenomenon where an optimal learning rate found for a smaller model can be successfully applied to a much deeper model without significant performance loss. The empirical results show that specific optimizers like Muon and MOGA exhibit this stable transfer across models with up to 64-fold depth differences.

Summary

The gist Scaling deep learning models efficiently requires predictable performance transfer across compute regimes.

Theoretical Framework and Core Finding

The authors establish that hyperparameter transfer can be understood through the L-smoothness framework of Xu et al. (2026), which examines how Lipschitz and L-smoothness constants scale with depth K Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks—including branches of block depth ≥ 2 such as transformer and ResNet blocks—without assumptions on initialization or correlations between layer weights Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA

Mathematical Guarantees for Depth Independence

The main theorems provide worst-case guarantees for depth-uniform sensitivity and smoothness bounds under the proposed scaling For MLP residual blocks, Proposition 1 establishes that the network satisfies Assumption 2 in a localized sense of Remark 2, meaning the W-regularity conditions hold on the ball where iterates are bounded Similarly for attention blocks under row or column normalization of Wq, Wk, Wv, Wo and bounded activations

Empirical Validation of Learning Rate Transfer

Numerical results confirm that inverse-depth scaling leads to stable learning rate transfer across architectures, models spanning a 64-fold depth counts choose nearly the same optimal learning rate. Specifically, for GPT2 models trained with Muon and MOGA, all depths select nearly the same learning rate. Furthermore, on CIFAR-100 ResNets with MOGA p ∈ 1.5 and p = 2.5, all three evaluated settings share the same best tested learning rate across ResNets with 8, 16, and 32 residual blocks. This demonstrates that Muon and MOGA exhibit a wide stable learning rate area that transfers as depth increases.

Conclusion on Practical Application

The analysis provides a geometric account of depth scaling and supports learning-rate transfer across the evaluated architectures and optimizer geometries. The experiments show that optimal learning rates stay nearly invariant on the tested state-of-the-art optimizers. This allows models over a 64-fold depth range to effectively transfer learning rates from small models to deeper models, reducing the computation budget of hyperparameter tuning. The analysis establishes inverse-depth residual scaling as a worst case guarantee for depth uniform sensitivity and smoothness bounds. Larger residual multipliers can also support learning rate transfer in certain settings. The final results suggest that the optimal learning rates across 7 depths still stay within a tight region shows that the optimal learning rates across 7 depths still stay within a tight region>. This indicates strong stable learning rate transfer. The results also provide empirical evidence that our scaling prescriptions work for different architectures; in total we have tested GPT-2, ResNet, and residual MLPs for our baseline. The results also provide empirical evidence that our scaling prescriptions work for different architectures; in total we have tested GPT-2, ResNet, and residual MLPs for our baseline>.

The gistScaling deep learning models efficiently requires predictable performance transfer across compute regimes.

Improvements for AI systems

Bold learning rate transfer across depth for large models: The system can reliably transfer optimal learning rates from shallow proxy models to massive models, as shown by Figure 4 where models that vary drastically in depth obtain their best validation loss performance at around the same peak learning rate. This eliminates the need for expensive hyperparameter tuning during training on very large architectures.
Bold robust optimization across residual network depths: The system can maintain a stable learning rate across a 64-fold increase in depth for architectures like GPT-2 and ResNets, as demonstrated by the optimal learning rates across 7 depths still stay within a tight region. This ensures that the training dynamics remain predictable regardless of the specific architectural scaling.
Bold theoretical foundation for optimizer selection: The system can utilize an L-smoothness constant that is independent of depth to guide learning rate selection, motivated by the finding that a common smoothness bound across depths therefore ensures that the curvature term does not require smaller learning rates as the network becomes deeper. This allows for principled scaling of step sizes without needing additional depth-dependent curvature corrections.
Bold generalization beyond specific architectures: The framework provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA, meaning the derived scaling rules are applicable to general residual networks—including branches of block depth ≥ 2 such as transformer and ResNet blocks—without assumptions on initialization or correlations between layer weights.
Bold precise regularization for attention mechanisms: By employing row or column normalization of Wq, Wk, Wv, Wo, the system can ensure that the attention block satisfies Assumption 2, allowing for depth-independent guarantees of Lipschitz continuity and smoothness in attention-based layers.

Abstract

Scaling deep learning models efficiently requires predictable performance transfer across compute regimes. Establishing reliable hyperparameter transfer allows optimal training dynamics and structural choices to be discovered on low-cost proxy models, avoiding catastrophic failures during massive-scale training. Existing depth-transfer analyses are architecture-specific and heuristic: hyperparameter transfer is only justified through toy blocks, and both rely on infinite-depth dynamics, a specific initialization, and feature-learning criteria. Building on L-smoothness framework of Xu et al. (2026), we understand hyperparameter transfer as how do the Lipschitz and L-smoothness constants of the loss scale with the depth K ? We show that inverse-depth residual branch scaling α=Θ(1/K) is sufficient to make both constants depth-independent at fixed width, uniformly over the bounded parameter domain and for arbitrary p to q geometries. Because the bounds depend on each residual branch only through its Lipschitz and smoothness constants, the argument applies to general residual networks (including branches of block depth at least 2 such as transformer and ResNet blocks) without assumptions on initialization or correlations between layer weights. Since the smoothness constant controls the curvature penalty in the descent inequality, our bounds motivate a learning-rate scale without an additional depth-dependent curvature correction for updates bounded in the prescribed norm. This provides a geometric explanation of inverse-depth residual scaling and its application to the operator-norm geometries associated with AdamW, Muon, and MOGA. Learning-rate sweeps on GPT-2 (depths 2--128) and CIFAR-100 ResNets (8--32 residual blocks) confirm that Muon and MOGA exhibit stable learning-rate transfer under these prescriptions, with a common optimal rate across a 64-fold increase in depth.