Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation

arXiv:2610.23842 · math.ST, stat.ME, stat.TH · Submitted 2026-10-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Mathematics. The day in mathematics: new theorems, new proofs, and what they open.

Ada: We're looking at the paper: "Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation".

Paper summary: Ada: So, looking at the whole paper, "Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation", the main thing is establishing this precise compatibility between learning about the overall regression function and shrinking those redundant layers toward an identity representation.

Emmy: They argue that this augmentation serves to resolve those deep decompositions by selecting a more canonical, identity-biased layer representation, which they show is robust even when predictive advantages disappear, according to page one of THIS PAPER > Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation <ref:2610.23842#pg1,Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation>.

Kolya: It seems the most important effect isn't just better prediction; it’s that the method resolves redundant deep decompositions by picking a more canonical, identity-biased layer representation, which is what they found in the sample size sweep.

Ada: Yeah, so to summarize, they proved that this technique is effective as a representation stabilizing device even when the predictive advantage itself starts to fade. It’s about getting a more structured view of those deep layers instead of just chasing the best prediction possible for a given noise level.

Emmy: It sets up a clear theoretical framework showing how function-level learning and layerwise shrinkage work together perfectly, which is something we needed to address when we used DGPs as hierarchical representations, according to page two of THIS PAPER > Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation <ref:2610.23842#pg1,Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation>.

Kolya: It’s a way to manage the identifiability problem head-on by adding this penalty term, which is a neat structural fix rather than just tweaking the training objective.

Ada: Exactly. The authors show that this augmentation changes the posterior odds between symmetry-related decompositions by an exponential penalty ratio, which is how they get that shrinkage toward identity mapping in Theorem four point one of THIS PAPER > Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation <ref:2610.23842#pg1,posterior odds between symmetry-related decompositions by an exponential penalty ratio>.

Conclusion: Ada: So, we’re wrapping up this look at "Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation." This paper essentially shows how to make sense of those complex deep layer structures in deep Gaussian processes by fixing a fundamental problem with symmetry between adjacent layers.

Emmy: Exactly. It takes these flexible priors and gives us a way to pull those redundant layers back toward the identity mapping, which is what we call canonical representation.

Kolya: From my side, the main thing is that they prove this works even when you look at function-level contraction under proper prior conditions, which suggests it’s not just some lucky trick for specific data.

Ada: Right. So they’ve established a precise link between learning the overall regression function and shrinking those extra layers down to a simpler form, which is pretty clean in theory.

Emmy: It means that instead of having this huge, tangled mess of deep layer decompositions, you can have one that has a more stable structure underneath, even if the predictive accuracy doesn't jump dramatically.

Kolya: And they showed this augmentation actually resolves the identifiability issues caused by those adjacent layer symmetries by changing the posterior odds using an exponential penalty ratio.

Ada: So the core message is that we can use this method to select a more canonical, identity-biased layer representation, which is what we wanted structurally.

Emmy: It’s about finding a stable structure for these deep neural network layers within the Gaussian process framework without losing the overall function-level learning.

Kolya: And they point out that this effect is pretty robust because it remains effective even when the predictive advantage disappears, which is important for real applications.

Ada: It reinforces the idea that this isn't just about getting a better prediction; it’s about resolving redundant deep decompositions by selecting a more canonical, identity-biased layer representation.

Emmy: And now we have to look at what kinds of function families these deep Gaussian processes can actually handle, because that opens up a whole new area for estimation.

University of Hong Kong · Rice University · Lawrence Berkeley National Laboratory

math.ST, stat.ME, stat.TH

Submitted: 2026-10-16

Updated: 2026-11-17

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 91/100

The gist: Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions <ref:2610.23842#pg2>.

Key concepts

Structural Non-identifiability
This occurs because applying an invertible transformation and its inverse to two adjacent layers leaves the overall regression map unchanged. This means the model cannot uniquely determine which specific intermediate layer configuration is correct, even if the final output function is estimable.
Augmented $ ho$-fractional Posterior
This is a modified posterior used during inference that reweights different layer decompositions. It introduces a non-saturating penalty based on how much each layer deviates from an identity mapping. This penalty breaks the symmetry between equivalent layers, favoring simpler, more canonical representations.
Function-level Contraction
This refers to the property where the posterior distribution contracts towards a specific function in terms of R´enyi divergence. The augmentation ensures this contraction is preserved under proper conditions while simultaneously shrinking redundant layers toward the identity mapping.

Terminology

Summary

Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions <ref:2610. p. 0

Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

23842#pg2>. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio <ref:2610. p. 0
Where the paper says it

We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio.

23842#pg2> and favors identity-biased parsimonious decompositions within an equivalence class of layers. p. 0
Where the paper says it

Under proper choices of the penalty and a prior mass condition, it preserves function-level fractional posterior contraction, while shrinking the redundant layers towards the identity mapping. Thus, the likelihood does not identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected representative is not established.

Structural Non-identifiability

Deep Gaussian processes (DGPs) are defined through compositions of latent GP layers, so the likelihood depends on the layer tuple only through the composed regression map. The adjacent-layer symmetry example shows that an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. p. 0

Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

This phenomenon generalizes identifiability issues to functional spaces and means posterior contraction of C(θ) does not imply contraction of θ. p. 1
Where the paper says it

This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. The statistical consequence is that posterior contraction of C(θ) does not imply posterior contraction of θ.

Symmetry-breaking by Augmentations

The authors introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. This augmentation changes posterior odds between symmetry-related decompositions through the difference in their total identity penalty. p. 0

Where the paper says it

We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio.

Theorem 4.1 shows such augmentation resolves the identifiability issue caused by adjacent layer symmetries. p. 2
Where the paper says it

We introduce augmented posteriors that encourage each non-final DGP layer towards identity mapping. 1 shows such augmentation resolves the identifiability issue caused by the adjacent layer symmetries.

Function-level Contraction and Layerwise Parsimony

The augmented DGP can preserve function-level contraction in R´enyi divergence under proper prior mass conditions. Theorem 5.1 shows that if ηj,n are chosen to be relatively small for the active block, redundant layers are pulled towards the identity mapping. p. 0

Where the paper says it

The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio. Under proper choices of the penalty and a prior mass condition, it preserves function-level fractional posterior contraction, while shrinking the redundant layers towards the identity mapping.

Theorem 5.2 proves that for certain scaling conditions, the posterior mass on redundant layers can be arbitrarily small. p. 3
Where the paper says it

Our use of the identity is related but statistically different: the classical DGP architecture remains, but a posterior factor tilts mass among likelihood-equivalent decompositions. The novelty of the present work is motivated by a non-concentration phenomenon, addressed by an exact posterior-odds symmetry-breaking formula, and a set of rigorous conditions under which identity shrinkage of redundant layers coexists with function-level contraction.

Functional Spaces for DGP

The work investigates which function families DGPs can handle, suggesting they might lift adaptation to a new level beyond mere smoothness. Corollary 5. p. 8

Where the paper says it

The example does not apply to an unmodified stationary Mat´ern first-layer process on all of R d , whose RKHS contains no nonzero global constants. Corollary 3.

p. 8
Where the paper says it

The example does not apply to an unmodified stationary Mat´ern first-layer process on all of R d , whose RKHS contains no nonzero global constants. Corollary 3.

4 shows that for intrinsic lower dimensional structures, the DGP can be optimally estimated at the rate of n−1/3. p. 1
Where the paper says it

, 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are mostly function-level: they describe the learning of the composed regression map but not the learning of the individual layers, calling into the question: how much can we rely on the estimated intermediate layers when composition structures are of concern?

p. 1
Where the paper says it

, 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are mostly function-level: they describe the learning of the composed regression map but not the learning of the individual layers, calling into the question: how much can we rely on the estimated intermediate layers when composition structures are of concern?

Empirical Findings

The comparison between regular DGP and augmented DGP shows that the augmented DGP improves prediction for low and moderate noise levels. The main message is that canonicalization is primarily a structural objective, but in favorable compositional regimes it can be obtained with little or no loss of predictive accuracy. The augmentation resolves redundant deep decompositions by selecting a more canonical, identity-biased layer representation. p. 0

Where the paper says it

Thus, the likelihood does not identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures ∗Email: yczhu@hku.

Conclusion

The theory establishes a precise compatibility between function-level learning and identity-biased shrinkage of redundant layers. The results support the main interpretation of the proposed augmentation, which is to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation. p. 0

Where the paper says it

Thus, the likelihood does not identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures ∗Email: yczhu@hku.

p. 0
Where the paper says it

Thus, the likelihood does not identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures ∗Email: yczhu@hku.

The most robust effect is not merely to improve prediction, but to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation. p. 0
Where the paper says it

Thus, the likelihood does not identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures ∗Email: yczhu@hku.

p. 0
Where the paper says it

Thus, the likelihood does not identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures ∗Email: yczhu@hku.

The sample-size sweep reinforces the distinction between function-level learning and layerwise canonicalization. p. 1
Where the paper says it

, 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are mostly function-level: they describe the learning of the composed regression map but not the learning of the individual layers, calling into the question: how much can we rely on the estimated intermediate layers when composition structures are of concern?

The augmentation remains effective as a representation-stabilizing device even when the predictive advantage disappears. p. 1
Where the paper says it

, 2018; Schmidt and O’Hagan, 2003). In such cases, the likelihood does not select a unique layer representation, and the posterior may fail to concentrate on any fixed representative when the prior remains sufficiently symmetric.

References

Bhattacharya, A., D. Pati, and Y. Yang (2019) is cited for Bayesian fractional posteriors. p. 7

Where the paper says it

Specifically, for ρ ∈ (0, 1], the ρ-fractional posterior of θ = (f1, f2, · · · , fJ ) is defined as Πρ(dθ|Dn) ∝ [p(Dn|C(θ))]ρΠ(dθ). When ρ = 1, the ρ-fractional posterior reduces to the standard Bayesian posterior.

p. 7
Where the paper says it

Specifically, for ρ ∈ (0, 1], the ρ-fractional posterior of θ = (f1, f2, · · · , fJ ) is defined as Πρ(dθ|Dn) ∝ [p(Dn|C(θ))]ρΠ(dθ). When ρ = 1, the ρ-fractional posterior reduces to the standard Bayesian posterior.

Castillo, I. p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

and T. Randrianarisoa (2025) are cited for Deep horseshoe gaussian processes. p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

Damianou, A. p. 1
Where the paper says it

1 Introduction Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al.

p. 1
Where the paper says it

1 Introduction Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al.

and N. D. Lawrence (2013) are cited for Deep Gaussian processes. p. 1
Where the paper says it

1 Introduction Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al.

p. 1
Where the paper says it

1 Introduction Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al.

Dunlop, M. M., M. Girolami, A. M. Stuart, and A. L. Teckentrup (2018) are cited for How deep are deep Gaussian processes? p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

Finocchio, G. p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

and J. Schmidt-Hieber (2023) are cited for Posterior contraction for deep gaussian process priors. p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

Gu, Y. and G. Xu (2020) are cited for Partial identifiability of restricted latent class models. p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

He, K., X. Zhang, S. Ren, and J. Sun (2016) are cited for Deep residual learning for image recognition. p. 3
Where the paper says it

(2018) study how depth changes DGP behavior. Residual networks instead parameterize a layer as identity plus a perturbation (He et al.

p. 3
Where the paper says it

(2018) study how depth changes DGP behavior. Residual networks instead parameterize a layer as identity plus a perturbation (He et al.

Linero, A. R. (2025) is cited for Defensive model expansion for robust bayesian inference. p. 4
Where the paper says it

∼ N(0, σ2 ), (3) with f ∗ : X → R being the data-generating function. By putting a DGP prior (2) on the regression function f ∗ in (3), we obtain a valid Bayesian model.

p. 4
Where the paper says it

∼ N(0, σ2 ), (3) with f ∗ : X → R being the data-generating function. By putting a DGP prior (2) on the regression function f ∗ in (3), we obtain a valid Bayesian model.

Luo, H., A. Horiguchi, and L. Ma (2026) are cited for Efficient decision trees for tensor regressions. p. 0
Where the paper says it

23842v1 [math. ST] 16 Oct 2026

p. 0
Where the paper says it

23842v1 [math. ST] 16 Oct 2026

Luo, H., G. Nattino, and M. T. Pratola (2022) are cited for Sparse additive gaussian process regression. p. 1
Where the paper says it

1 Introduction Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al.

Papastamoulis, P. p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

and I. Ntzoufras (2022) are cited for On the identifiability of bayesian factor analytic models. p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

Rezende, D. p. 3
Where the paper says it

, 2003; Zammit-Mangion et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and Mohamed, 2015).

p. 3
Where the paper says it

, 2003; Zammit-Mangion et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and Mohamed, 2015).

and S. Mohamed (2015) are cited for Variational inference with normalizing flows. p. 3
Where the paper says it

, 2003; Zammit-Mangion et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and Mohamed, 2015).

p. 3
Where the paper says it

, 2003; Zammit-Mangion et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and Mohamed, 2015).

Salimbeni, H. p. 1
Where the paper says it

They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al. , 2022; Salimbeni and Deisenroth, 2017; Zhu et al.

p. 1
Where the paper says it

They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al. , 2022; Salimbeni and Deisenroth, 2017; Zhu et al.

and M. Deisenroth (2017) are cited for Doubly stochastic variational inference for deep gaussian processes. p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

Sampson, P. p. 3
Where the paper says it

(2020); here we quantify non-concentration and study an explicit representation preference. Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al.

p. 3
Where the paper says it

(2020); here we quantify non-concentration and study an explicit representation preference. Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al.

D. and P. Guttorp (1992) are cited for Nonparametric estimation of nonstationary spatial covariance structure. p. 3
Where the paper says it

(2020); here we quantify non-concentration and study an explicit representation preference. Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al.

p. 3
Where the paper says it

(2020); here we quantify non-concentration and study an explicit representation preference. Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al.

Schmidt, A. p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

M. and A. O’Hagan (2003) are cited for Bayesian inference for non-stationary spatial covariance structure via spatial deformations. p. 3
Where the paper says it

(2020); here we quantify non-concentration and study an explicit representation preference. Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al.

p. 3
Where the paper says it

(2020); here we quantify non-concentration and study an explicit representation preference. Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al.

Schmidt-Hieber, J. p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

p. 1
Where the paper says it

, 2022; Salimbeni and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023).

(2020) is cited for Nonparametric regression using deep neural networks with relu activation function. p. 1
Where the paper says it

, 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are mostly function-level: they describe the learning of the composed regression map but not the learning of the individual layers, calling into the question: how much can we rely on the estimated intermediate layers when composition structures are of concern?

p. 1
Where the paper says it

, 2024), and theoretically, because suitably constructed DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are mostly function-level: they describe the learning of the composed regression map but not the learning of the individual layers, calling into the question: how much can we rely on the estimated intermediate layers when composition structures are of concern?

Szabo, B. and Y. Zhu (2026) are cited for Vecchia gaussian processes. p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

Ustyuzhaninov, I. p. 3
Where the paper says it

Lower bounds for shallow GP priors establish advantages of depth for compositional classes (Giordano et al. Compositional non-identifiability and uncertainty over DGP layers are explicitly studied by Ustyuzhaninov et al.

p. 3
Where the paper says it

Lower bounds for shallow GP priors establish advantages of depth for compositional classes (Giordano et al. Compositional non-identifiability and uncertainty over DGP layers are explicitly studied by Ustyuzhaninov et al.

, I. Kazlauskaite, M. Kaiser, E. Bodin, N. Campbell, and C. H. Ek (2020) are cited for Compositional uncertainty in deep Gaussian processes. p. 3
Where the paper says it

Compositional non-identifiability and uncertainty over DGP layers are explicitly studied by Ustyuzhaninov et al. (2020); here we quantify non-concentration and study an explicit representation preference.

p. 3
Where the paper says it

Compositional non-identifiability and uncertainty over DGP layers are explicitly studied by Ustyuzhaninov et al. (2020); here we quantify non-concentration and study an explicit representation preference.

van der Vaart, A. W., and J. H. van Zanten (2008) are cited for Reproducing kernel hilbert spaces of gaussian priors. p. 5
Where the paper says it

With a little abuse of notations, we also use Π(dfj ) to denote the marginal prior law on the jth layer, which is itself the product of the coordinate-wise GP law of fj,k. For each layerwise process GPj , let Hj denote its reproducing kernel Hilbert space (RKHS), and ∥·∥Hj be its corresponding RKHS norm.

p. 5
Where the paper says it

With a little abuse of notations, we also use Π(dfj ) to denote the marginal prior law on the jth layer, which is itself the product of the coordinate-wise GP law of fj,k. For each layerwise process GPj , let Hj denote its reproducing kernel Hilbert space (RKHS), and ∥·∥Hj be its corresponding RKHS norm.

Zammit-Mangion, A. p. 3
Where the paper says it

Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. , 2003; Zammit-Mangion et al.

p. 3
Where the paper says it

Second, warping and deformation models use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. , 2003; Zammit-Mangion et al.

, T. L. J. Ng, Q. Vu, and M. Filippone (2022) are cited for Deep compositional spatial models. p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

p. 1
Where the paper says it

This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras, 2022) to functional spaces, and cannot be simply attributed to multi-modal posterior.

Zhu, Y., M. Peruzzi, C. Li, and D. B. Dunson (2024) are cited for Radial neighbours for provably accurate scalable approximations of gaussian processes. p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

p. 0
Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

--- Page 1 ---

Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. p. 0

Where the paper says it

Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation Yichen Zhu∗ Department of Statistics, the University of Hong Kong Hengrui Luo† Department of Statistics, Rice University and Computational Research Division, Lawrence Berkeley National Laboratory Abstract Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged.

Under explicit symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed regression function is estimable. p. 0
Where the paper says it

The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. Under explicit symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed regression function is estimable.

We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. p. 0
Where the paper says it

Under explicit symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed regression function is estimable. We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty.

Pruning redundant reasoning tokens

The authors introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio. p. 0

Where the paper says it

We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio.

Under proper choices of the penalty and a prior mass condition, it preserves function-level fractional posterior contraction, while shrinking the redundant layers towards the identity mapping. p. 0
Where the paper says it

The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio. Under proper choices of the penalty and a prior mass condition, it preserves function-level fractional posterior contraction, while shrinking the redundant layers towards the identity mapping.

p. 0
Where the paper says it

The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio. Under proper choices of the penalty and a prior mass condition, it preserves function-level fractional posterior contraction, while shrinking the redundant layers towards the identity mapping.

How the training signal is built

The augmentation changes the target only through the additive term PJ−1j=1 ln Rj (fj), so standard DGP variational or MCMC schemes can be reused with a simple penalty correction. The algorithm 2 summarizes the corresponding sparse-variational objectives for the unaugmented and augmented targets. p. 5

Where the paper says it

With a little abuse of notations, we also use Π(dfj ) to denote the marginal prior law on the jth layer, which is itself the product of the coordinate-wise GP law of fj,k. For each layerwise process GPj , let Hj denote its reproducing kernel Hilbert space (RKHS), and ∥·∥Hj be its corresponding RKHS norm.

Where the gains come from

Theorem 4.1 gives the exact change in posterior mass between symmetry-related sets through the product of the prior Radon-Nikod´ym factor and the exponential difference in identitydeviation penalties. p. 3

Where the paper says it

Our use of the identity is related but statistically different: the classical DGP architecture remains, but a posterior factor tilts mass among likelihood-equivalent decompositions. The novelty of the present work is motivated by a non-concentration phenomenon, addressed by an exact posterior-odds symmetry-breaking formula, and a set of rigorous conditions under which identity shrinkage of redundant layers coexists with function-level contraction.

The augmentation changes posterior odds between symmetry-related decompositions through the difference in their total identity penalty. p. 0
Where the paper says it

We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio.

Under proper choices of the penalty and a prior mass condition, it preserves function-level fractional posterior contraction, while shrinking the redundant layers towards the identity mapping.

The authors investigate which function families DGPs can handle, suggesting they might lift adaptation to a new level beyond mere smoothness. Corollary 5.4 shows that for intrinsic lower dimensional structures, the DGP can be optimally estimated at the rate of n−1/3.

The theory establishes a precise compatibility between function-level learning and identity-biased shrinkage of redundant layers. The results support the main interpretation of the proposed augmentation, which is to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation. The most robust effect is not merely to improve prediction, but to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation.

Bhattacharya, A., D. Pati, and Y. Yang (2019) is cited for Bayesian fractional posteriors. Castillo, I. and T. Randrianarisoa (2025) are cited for Deep horseshoe gaussian processes. Damianou, A. and N. D. Lawrence (2013) are cited for Deep Gaussian processes. Dunlop, M. M., M. Girolami, A. M. Stuart, and A. L. Teckentrup (2018) are cited for How deep are deep Gaussian processes? Finocchio, G. and J. Schmidt-Hieber (2023) are cited for Posterior contraction for deep gaussian process priors. Gu, Y. and G. Xu (2020) are cited for Partial identifiability of restricted latent class models. He, K., X. Zhang, S. Ren, and J. Sun (2016) are cited for Deep residual learning for image recognition. Linero, A. R. (2025) is cited for Defensive model expansion for robust bayesian inference. Luo, H., A. Horiguchi, and L. Ma (2026) are cited for Efficient decision trees for tensor regressions. Luo, H., G. Nattino, and M. T. Pratola (2023) are cited for Sparse additive gaussian process regression. p. 1

Where the paper says it

1 Introduction Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary and compositional structures (Damianou and Lawrence, 2013; Luo et al.

Papastamoulis, P. and I. Ntzoufras (2022) are cited for On the identifiability of bayesian factor analytic models. Rezende, D. and S. Mohamed (2015) are cited for Variational inference with normalizing flows. Salimbeni, H. and M. Deisenroth (2017) are cited for Doubly stochastic variational inference for deep gaussian processes. Sampson, P. D. and P. Guttorp (1992) are cited for Nonparametric estimation of nonstationary spatial covariance structure. Schmidt, A. M. and A. O’Hagan (2003) are cited for Bayesian inference for non-stationary spatial covariance structure via spatial deformations. Schmidt-Hieber, J. (2020) is cited for Nonparametric regression using deep neural networks with relu activation function. Szabo, B. and Y. Zhu (2026) are cited for Vecchia gaussian processes. Ustyuzhaninov, I., I. Kazlauskaite, M. Kaiser, E. Bodin, N. Campbell, and C. H. Ek (2020) are cited for Compositional uncertainty in deep Gaussian processes. van der Vaart, A. W., and J. H. van Zanten (2008) are cited for Reproducing kernel hilbert spaces of gaussian priors. Zammit-Mangion, A., T. L. J. Ng, Q. Vu, and M. Filippone (2022) are cited for Deep compositional spatial models. Zhu, Y., M. Peruzzi, C. Li, and D. B. Dunson (2024) are cited for Radial neighbours for provably accurate scalable approximations of gaussian processes.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation: The augmented DGP can resolve identifiability issues caused by adjacent layer symmetries by reweighting layer decompositions with a non-saturating identity-deviation penalty, favoring identity-biased parsimonious representations within an equivalence class of layers.

  2. Function Level Contraction Preservation: By choosing penalty parameters eta j n appropriately, the augmented DGP preserves function level contraction in R´enyi divergence while shrinking redundant layers towards the identity mapping, thus selecting a canonical representative without changing the DGP architecture.

  3. Layerwise Parsimony Achievement: The system can be tuned such that redundant layers are forced toward near-identity mappings while active blocks maintain their prior mass, achieving layerwise parsimony by setting the tuning window n epsilon squared n less than or equal to eta j n less than or equal to n epsilon squared n over epsilon˜n for layers j less than J0.

  4. Handling Local and Intrinsic Structures: The augmented DGP framework can be adapted to estimate functions with local structures, such as non-differentiable cusps, by using a proxy function and carefully chosen regularization parameters that allow the model to contract to the true function at rates faster than those achievable by smoother models.

  5. Discontinuous Function Estimation: For discontinuous step functions, the augmented DGP can achieve a posterior contraction rate of n minus alpha squared plus three alpha times d for some choice of J and penalty weights, allowing estimation rates potentially approaching parametric rates under favorable conditions.

Abstract

Deep Gaussian processes (DGPs) are flexible priors for estimating compositional functions, but function-level accuracy does not determine their intermediate-layer representations. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. Under explicit symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed regression function is estimable. We introduce a layerwise exponential tilt of the DGP prior, using a penalty proportional to each hidden layer's distance from the identity map, with stronger weights on earlier layers. The methodology links an exact posterior-odds rule along likelihood-preserving symmetries to quantitative control of redundant layers. Under proxy and local prior-mass conditions, we derive a penalty window that preserves function-level fractional posterior contraction while localizing redundant layers at certain radius. The resulting design rule calibrates representation regularization to the function-estimation rate. It favors decompositions with earlier layers close to identity; uniqueness and automatic depth selection are not established. The general theory and methodology are supported with detailed examples.

Sources

Related papers