On Compatible Generalization Bounds of Deep Residual Networks From a Dynamical System Perspective

arXiv:2602.20921 · cs.LG · Submitted 2026-02-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "On Compatible Generalization Bounds of Deep Residual Networks From a Dynamical System Perspective".

Jane: Here's the gist of the paper. Deep neural networks have significantly advanced machine learning, with model depth playing a central role in their successes <ref:2602.20921#pg2> The resulting bounds are of order O(1/√S) with respect to the number of training samples S, and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions These findings provide a unified understanding of generalization across both discrete- and continuous-time ResNets, helping to close the gap in both the order of sample complexity and assumptions between the discrete- and continuous-time settings Dynamical System Modeling Framework The work establishes generalization error bounds for both discrete- and continuous-time residual networks by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit This approach models discrete-time ResNets as approximations of continuous-time neural ordinary differential equations (ODEs) The dynamical system framework refers to a unified modeling and analytical viewpoint that encompasses both discretetime ResNets and their continuous-time limits The analysis proceeds from discrete ResNets and exploits their convergence to continuous-time dynamics Derivation of Generalization Bounds The authors establish generalization error bounds for discrete- and continuous-time ResNets simultaneously, achieving optimal order O(1/√S) in both settings The bounds are derived via a new contraction-type inequality for a broad class of activation functions, leading to a refined bound structure consisting of a depth-independent leading term together with a non-positive, depth-dependent structural correction term induced by the activation functions Theorem 3.5 provides the generalization error for discrete-time ResNets and Theorem 3.9 provides the generalization error for continuous-time ResNets Depth Uniformity and Asymptotic Behavior The resulting bounds admit an explicit depth-uniform upper bound because the generalization error admits a structure-dependent negative term which, when discarded, yields a depth-uniform upper bound This implies that increasing network depth does not lead to an accumulation of generalization error, which is consistent with empirical observations of ResNets in practical applications The analysis further admits an asymptotic interpretation as the network depth tends to infinity, suggesting that the generalization capacity of deep ResNets can remain stable in the deep-layer limit Influence of Activation Functions The authors introduce a class of activation functions A(R) that admit a structured decomposition, such as ψ = ϕ1(· − α) − ϕ2(− · −β) This leads to a refined contraction inequality that captures structural properties beyond standard Lipschitz continuity Empirical experiments show that ResNets equipped with activation functions admitting such a decomposition tend to yield smaller generalization gaps on CIFAR10 and CIFAR100 in the later stage of training compared to the standard leaky ReLU This dynamical system framework offers a unified and theoretically grounded perspective on the interplay between architectural stability, depth, and generalization The work derives new generalization bounds for both discrete- and continuous-time ResNets that attain a complexity order of O(1/√S) The discrete-time bound further incorporates a structure-dependent negative term, which can potentially tighten estimates under suitable conditions Overall, this framework helps to close the order gap between the existing generalization analyses for discrete- and continuous-time ResNets Appendix A.

Tom: Definition of Rademacher complexity Definition A.1.

Jane: The (empirical) Rademacher complexity of the function class F for i.i.d.

Tom: sample Z = z(1), z(2), · · ·, z(S) from B is RZ (F) = 1/S Eε sup f∈F X S s=1 ε(s)f z(s) where the expectation is taken over ε = ε(1),..., ε(S) and uniformly distributed over −1, 1 <ref:2602.20921#pg9> For convenience, we denote ε = ε(1),..., ε(S) and ϵ = δ(s) and δ(sj) are independently and uniformly distributed over −1, 1 <ref:2602.20921#pg9> Also, for the convenience of readers, several key referred results are provided in the supplementary material Acknowledgments This work was partially supported by the National Natural Science Foundation of China (grant 12271273) and the Key Program (21JCZDJC00220) of the Natural Science Foundation of Tianjin, China The references listed are

1: through

47: REFERENCES

1: P.

Paper summary: Tom: We're going into the specifics now on "On Compatible Generalization Bounds of Deep Residual Networks From a Dynamical System Perspective". This paper sets out to establish generalization error bounds for both discrete-time and continuous-time residual networks by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit.

Jane: The core thesis is that this unified modeling approach treats discrete-time ResNets as approximations of continuous-time neural ordinary differential equations, using their convergence to continuous-time dynamics to analyze the asymptotic behavior with respect to network depth.

Lu: They achieve optimal order one over the square root of the number of training samples S in both settings, and they include a structure-dependent negative term that helps yield depth-uniform and asymptotic generalization bounds under milder assumptions.

Meng: So, this isn't just another bound; it's aiming to provide a consistent framework that works across both discrete and continuous time network settings.

Tom: It’s about closing the gap in sample complexity and the assumptions between those two settings, which is what makes this work significant for how we analyze deep learning models.

Jane: The paper also introduces a new class of activation functions with a specific decomposition structure, like psi equals phi1 of dash alpha minus phi2 of minus dash beta, which leads to a refined contraction inequality.

Lu: That decomposition structure allows them to derive a sharper generalization error bound by capturing structural properties beyond just standard Lipschitz continuity, and experiments show ResNets with these functions perform better on CIFAR10 and CIFAR100 later in training compared to standard leaky ReLU.

Meng: From an engineering view, if we can find activation functions that have this specific structure, it might give us a practical way to tune the network architecture for better generalization performance without needing super deep architectures.

Tom: It's about finding these structural properties through the activation function design itself, which is a clever way to control the Rademacher complexity of the hypothesis class.

Jane: And they show that this negative structural term in Theorem three point five reflects a structural effect induced by those activation functions, depending on constants like Lipphi1, Lipphi2, alpha, and beta.

Lu: That suggests that learning the parameters for these activation functions might enable the network to discover configurations that improve generalization performance beyond what we get with standard setups.

Meng: If we can discover these good activation configurations early on in training, it could mean less waste in computation time for us at scale.

Conclusion: Tom: So, wrapping up this discussion on "On Compatible Generalization Bounds of Deep Residual Networks From a Dynamical System Perspective", the authors have really provided a unified and theoretically grounded perspective on how architectural stability, network depth, and generalization interact.

Jane: They've derived new generalization bounds for both discrete-time and continuous-time ResNets that hit the optimal complexity order of one over the square root of S in both settings.

Lu: The key contribution is that the discrete-time bound includes a structure-dependent negative term, which can potentially tighten those estimates if we use suitable conditions.

Meng: For us shipping models, this means we have a more solid theoretical foundation to rely on when predicting how much data S we need for a certain level of generalization performance.

Tom: It helps close that order gap between the existing analyses for discrete- and continuous-time ResNets by showing they are compatible under these new bounds.

Jane: Essentially, this dynamical system framework gives us a single theoretical lens through which to view the interplay between architecture, depth, and generalization performance in deep residual networks.

Nankai University

cs.LG

Submitted: 2026-02-24

Updated: 2026-11-10

Code: https://github.com/Huangjsh15/Dynamical-ResNet

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Deep neural networks have significantly advanced machine learning, with model depth playing a central role in their successes <ref:2602.20921#pg2> The resulting bounds are of order O(1/√S) with…

Key concepts

Dynamical System Modeling Framework
This approach models discrete ResNets as approximations of continuous-time neural ordinary differential equations (ODEs). It provides a unified analytical viewpoint that covers both discrete and continuous settings, allowing the analysis to proceed by exploiting the convergence properties of ResNets in the deep-layer limit.
Rademacher Complexity
This is a measure used to bound generalization error by quantifying how well a function class can fit random noise. The paper uses it alongside flow maps of dynamical systems to derive generalization bounds for both discrete and continuous-time ResNets, helping to control the complexity of the hypothesis class.
Structure-Dependent Negative Term
The derived bounds include a negative structural term dependent on activation functions. This term reflects specific structural effects induced by the activation functions, suggesting that learning appropriate activation parameters can improve generalization performance by controlling this structural correction.

Terminology

Summary

Deep neural networks have significantly advanced machine learning, with model depth playing a central role in their successes <ref:2602. p. 020921#pg2> The resulting bounds are of order O(1/√S) with respect to the number of training samples S, and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions These findings provide a unified understanding of generalization across both discrete- and continuous-time ResNets, helping to close the gap in both the order of sample complexity and assumptions between the discrete- and continuous-time settings

Dynamical System Modeling Framework

The work establishes generalization error bounds for both discrete- and continuous-time residual networks by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit This approach models discrete-time ResNets as approximations of continuous-time neural ordinary differential equations (ODEs) The dynamical system framework refers to a unified modeling and analytical viewpoint that encompasses both discretetime ResNets and their continuous-time limits The analysis proceeds from discrete ResNets and exploits their convergence to continuous-time dynamics

Derivation of Generalization Bounds

The authors establish generalization error bounds for discrete- and continuous-time ResNets simultaneously, achieving optimal order O(1/√S) in both settings The bounds are derived via a new contraction-type inequality for a broad class of activation functions, leading to a refined bound structure consisting of a depth-independent leading term together with a non-positive, depth-dependent structural correction term induced by the activation functions Theorem 3.5 provides the generalization error for discrete-time ResNets and Theorem 3. p. 99 provides the generalization error for continuous-time ResNets

Depth Uniformity and Asymptotic Behavior

The resulting bounds admit an explicit depth-uniform upper bound because the generalization error admits a structure-dependent negative term which, when discarded, yields a depth-uniform upper bound This implies that increasing network depth does not lead to an accumulation of generalization error, which is consistent with empirical observations of ResNets in practical applications The analysis further admits an asymptotic interpretation as the network depth tends to infinity, suggesting that the generalization capacity of deep ResNets can remain stable in the deep-layer limit

Influence of Activation Functions

The authors introduce a class of activation functions A(R) that admit a structured decomposition, such as ψ = ϕ1(· − α) − ϕ2(− · −β) This leads to a refined contraction inequality that captures structural properties beyond standard Lipschitz continuity Empirical experiments show that ResNets equipped with activation functions admitting such a decomposition tend to yield smaller generalization gaps on CIFAR10 and CIFAR100 in the later stage of training compared to the standard leaky ReLU

Conclusion

This dynamical system framework offers a unified and theoretically grounded perspective on the interplay between architectural stability, depth, and generalization The work derives new generalization bounds for both discrete- and continuous-time ResNets that attain a complexity order of O(1/√S) The discrete-time bound further incorporates a structure-dependent negative term, which can potentially tighten estimates under suitable conditions Overall, this framework helps to close the order gap between the existing generalization analyses for discrete- and continuous-time ResNets

Appendix A. Definition of Rademacher complexity

Definition A.1. The (empirical) Rademacher complexity of the function class F for i. p. 6i.d. sample Z = z(1), z(2), · · ·, z(S) from B is RZ (F) = 1/S Eε sup f∈F X S s=1 ε(s)f z(s) where the expectation is taken over ε = ε(1),. p. 0.., ε(S) and uniformly distributed over −1, 1 <ref:2602. p. 020921#pg9> For convenience, we denote ε = ε(1),. p. 3.., ε(S) and ϵ = δ(s) and δ(sj) are independently and uniformly distributed over −1, 1 <ref:2602. p. 520921#pg9> Also, for the convenience of readers, several key referred results are provided in the supplementary material

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (grant 12271273) and the Key Program (21JCZDJC00220) of the Natural Science Foundation of Tianjin, China The references listed are [1] through [47]

REFERENCES

[1] P. L. Bartlett, D. p. 0 J. Foster, and M. J. Telgarsky, Spectrally-normalized margin bounds for neural networks, Advances in Neural Information Processing Systems, 30 (2017). p. 0

[2] P. L. Bartlett and S. p. 0 Mendelson, Rademacher and gaussian complexities: Risk bounds and structural results, Journal of Machine Learning Research, 3 (2002), pp. p. 2 463–482.

[3] P. L. Bartlett and R. p. 0 C. Williamson, The vc dimension and pseudodimension of two-layer neural networks with discrete inputs, Neural Computation, 8 (1996), pp. p. 1 625–628.

[4] A Beck, First-order methods in optimization, SIAM, 2017. p. 0

[5] M. Benning, E. Celledoni, M. Jerrhardt, B. Owren, and C.-B Schonlieb ¨, Deep learning as optimal control problems: Models and numerical methods, Journal of Computational Dynamics, 6 (2019), pp. p. 1 171–198.

[6] L. Bleistein and A Guilloux, On the generalization capacities of neural controlled differential equations, in Workshop on New Frontiers in Learning, Control, and Dynamical Systems at the International Conference on Machine Learning (ICML 2023), 2023. p. 1

[7] O. Bousquet and A Elisseeff, Stability and generalization, Journal of Machine Learning Research, 2 (2002), pp. p. 1 499–526.

[8] R. T Chen, Y Rubanova, J Bettencourt, and D K Duvenaud, Neural ordinary differential equations, Advances in Neural Information Processing Systems, 31 (2018). p. 1

[9] X Chen, Y Zhang, C Reisinger, and L Song, Understanding deep architecture with reasoning layer, in Advances in Neural Information Processing Systems, 2020.

[10] D.-A Clevert, Fast and accurate deep network learning by exponential linear units (elus), arXiv preprint arXiv:1511. p. 007289, (2015).

[11] W E A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, 1 (2017), pp. p. 0 1–11.

[12] W E C Ma, and Q Wang, Rademacher complexity and the generalization error of residual networks, Communications in Mathematical Sciences, 18 (2020), pp. p. 0 1755–1774.

[13] E Emmrich, Discrete versions of Gronwall’s lemma and their application to the numerical analysis of parabolic problems, Techn. p. 1 Univ., 1999.

[14] X Glorot, A Bordes, and Y Bengio, Deep sparse rectifier neural networks, in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, 2011, pp. p. 0 315–323.

[15] P Goldberg and M Jerrum, Bounding the vapnik-chervonenkis dimension of concept classes parameterized by real numbers, Machine Learning, 18 (1995), pp. p. 0 131–148.

[16] I Goodfellow, Y Bengio, and A Courville, Deep learning, MIT press, 2016. p. 0

[17] P Grohs and G Kutyniok, Mathematical aspects of deep learning, Cambridge University Press, 2022. p. 0

[18] S Gunther, L Ruthotto J B Schroder E C Cyr and N R Gauger, Layer-parallel training of deep residual neural networks, SIAM Journal on Mathematics of Data Science, 2 (2020), pp. p. 0 1–23.

[19] E Haber and L Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2017), p. p. 1 014004.

[20] J Hanson and M Raginsky, Rademacher complexity of neural odes via chen-fliess series, in 6th Annual Learning for Dynamics & Control Conference, PMLR, 2024, pp. p. 1 758–769.

[21] F He, T Liu, and D Tao, Why resnet works? p. 0 residuals generalize, IEEE Transactions on Neural Networks and Learning Systems, 31 (2020), pp. p. 0 5349–5362.

[22] K He, X Zhang, S Ren, and J Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. p. 9 1026–1034.

[23] K He, X Zhang, S Ren, and J Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. p. 0 770–778.

[24] J Huang, Y Gao, and C Wu, On dynamical system modeling of learned primal-dual with a linear operator K: stability and convergence properties, Inverse Problems, 40 (2024), p. p. 1 075006.

[25] J Huang, H Su, X. p. 0-C Tai, and C Wu, Mathematical modeling and convergence analysis of deep neural networks with dense layer connectivities, Submitted, (2024). p. 0

[31] X Li, J Lu, Z Wang, J Haupt, and T Zhao On tighter generalization bound for deep neural networks: Cnns resnets and beyond, arXiv preprint arXiv:1806. p. 005159, (2018).

[32] Z Li, J Han, E Weinan, and Q Li Approximation and optimization theory for linear continuous-time recurrent neural networks, Journal of Machine Learning Research, 23 (2022), pp. p. 1 1–85.

[33] Y Lu, A Zhong, Q Li, and B Dong Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations, in Proceedings of Machine Learning Research, vol. p. 1 80, PMLR, 2018, pp. 3282–3291.

[34] P Marion Generalization bounds for neural ordinary differential equations and deep residual networks, Advances in Neural Information Processing Systems, 36 (2023), pp. p. 1 48918–48938.

[35] A Maurer, A vector-contraction inequality for rademacher complexities, in Algorithmic Learning Theory: 27th International Conference, Springer, 2016, pp. p. 6 3–17.

[36] B Neyshabur, S Bhojanapalli, and N Srebro, A pac-bayesian approach to spectrally-normalized margin bounds for neural networks, in International Conference on Learning Representations, 2018. p. 0

[37] B Neyshabur, R Tomioka, and N Srebro Norm-based capacity control in neural networks, in Conference on Learning Theory, PMLR, 2015, pp. p. 1 1376–1401.

[38] V Pandey Overcoming overfitting and large weight update problem in linear rectifiers: Thresholded exponential rectified linear units, arXiv preprint arXiv:2006. p. 002797, (2020).

[39] P Parpas and C Muir Predict globally, correct locally: Parallel-in-time optimization of neural networks, Automatica, 171 (2025), https://doi. p. 1org/10.1016/j.automatica.2024.111976.

[40] L Ruthotto and E Haber, Deep neural networks motivated by partial differential equations, Journal of Mathematical Imaging and Vision, 62 (2020), pp. p. 1 352–364.

[41] E Schnoor, A Behboodi, and H Rauhut, Generalization error bounds for iterative recovery algorithms unfolded as neural networks, Information and Inference: A Journal of the IMA, 12 (2023), pp. p. 1 2267–2299.

[42] S Shalev-Shwartz and S Ben-David, Understanding machine learning: From theory to algorithms, Cambridge University Press, 2014. p. 1

[43] A Sherstinsky, Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network, Physica D: Nonlinear Phenomena, 404 (2020), p. p. 1 132306.

[44] A Shultzman, E Azar, M R Rodrigues, and Y C Eldar Generalization and estimation error bounds for model-based neural networks, in International Conference on Learning Representations, 2023. p. 0

[45] J Sokolic, R Giryes, G Sapiro, and M Rodrigues Generalization error of invariant classifiers, in Artificial Intelligence and Statistics, PMLR, 2017, pp. p. 0 1094–1103.

[46] M Thorpe and Y van Gennip Deep limits of residual neural networks, Research in the Mathematical Sciences, 10 (2023), p. p. 2 6.

[47] R Vershynin, High-dimensional probability: An introduction with applications in data science, vol. p. 0 47, Cambridge University Press, 2018. p. 0

--- Page 1 ---

ON THE GENERALIZATION BEHAVIOR OF DEEP RESIDUAL NETWORKS FROM A DYNAMICAL SYSTEM PERSPECTIVE JINSHU HUANG†, MINGFEI SUN†, AND CHUNLIN WU‡ Abstract. Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes In this work, we establish generalization error bounds for both discrete- and continuous-time residual networks (ResNets) by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit The resulting bounds are of order O(1/√S) with respect to the number of training samples S, and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions

Pruning redundant reasoning tokens

The authors establish generalization error bounds for both discrete- and continuous-time ResNets by combining Rademacher complexity, flow maps of dynamical systems, and the convergence behavior of ResNets in the deep-layer limit The resulting bounds are of order O(1/√S) with respect to the number of training samples S, and include a structure-dependent negative term, yielding depth-uniform and asymptotic generalization bounds under milder assumptions

How the training signal is built

The authors approximate the true risk using a training dataset S:= (d(s), g(s)) consisting of independently and identically distributed (i.i.d.) samples drawn from D The goal is then to find a hypothesis xL(·; Θtotal L) by solving the following sampled optimal control problem This problem involves minimizing the empirical risk RˆS(xL(·; Θtotal L)) subject to the constraint that xL(d(s); Θtotal L) is the end state of Eq. p. 5(2.1) with input d(s)

Where the gains come from

The authors derive a refined contraction inequality by using the decomposition structure of activation functions, which plays a key role in controlling the Rademacher complexity of the hypothesis class, leading to sharper generalization error bounds The negative structural term in Theorem 3.5 reflects a structural effect induced by the activation function This term depends on several activation-related constants, including Lipϕ1, Lipϕ2, α, and β This suggests that learning activation parameters may enable the network to discover activation configurations that yield improved generalization performance The analysis also admits a natural asymptotic interpretation with respect to depth, suggesting that discretizations of ODE-based architectures yield consistent discrete networks and lead to generalization bounds for continuous-time ResNets The bound also attains the optimal O(1/√S) rate established in prior works on discrete-time ResNets [31, 21, 12] The empirical validation confirms that the generalization gap decreases as the number of training samples increases, closely matching the theoretical rate of O(1/√S) The results further confirm that the generalization error decreases as the number of training samples increases

The authors derive a refined contraction inequality by using the decomposition structure of activation functions, which plays a key role in controlling the Rademacher complexity of the hypothesis class, leading to sharper generalization error bounds The negative structural term in Theorem 3.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified Generalization Bounds for ResNets: The system can establish generalization error bounds of order O(1/sqrt(S)) simultaneously for both discrete-time ResNets and their continuous-time counterparts, providing a unified understanding across both settings.

  2. Depth-Uniform Generalization Control: The model can derive a depth-uniform upper bound on generalization error by exploiting a new contraction-type inequality that results in a depth-independent leading term together with a non-positive, depth-dependent structural correction term.

  3. Improved Activation Function Design: By learning activation parameters such as alpha and beta, the system can discover activation configurations that yield improved generalization performance, as empirical results show smaller generalization gaps compared to standard leaky ReLU on CIFAR10 and CIFAR100.

  4. Asymptotic Stability Analysis: The dynamical system framework allows for an asymptotic interpretation of deep networks by establishing the convergence of discrete-time learning problems to their continuous-time counterparts in the deep-layer limit, suggesting that the generalization capacity of deep ResNets can remain stable in the deep-layer limit.

  5. Enhanced Sample Complexity Guarantees: The system provides sharper sample complexity guarantees than prior works, attaining an O(1/sqrt(S)) rate for both discrete- and continuous-time settings, which is superior to the O(1/sqrt(4S)) bound obtained in some discrete-time settings.

Sources

Related papers