Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Mathematics. The day in mathematics: new theorems, new proofs, and what they open.
Ada: We're looking at the paper: "Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation".
Paper summary: Ada: So, looking at the whole paper, "Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation", the main thing is establishing this precise compatibility between learning about the overall regression function and shrinking those redundant layers toward an identity representation.
Emmy: They argue that this augmentation serves to resolve those deep decompositions by selecting a more canonical, identity-biased layer representation, which they show is robust even when predictive advantages disappear, according to page one of THIS PAPER > Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation <ref:2610.23842#pg1,Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation>.
Kolya: It seems the most important effect isn't just better prediction; it’s that the method resolves redundant deep decompositions by picking a more canonical, identity-biased layer representation, which is what they found in the sample size sweep.
Ada: Yeah, so to summarize, they proved that this technique is effective as a representation stabilizing device even when the predictive advantage itself starts to fade. It’s about getting a more structured view of those deep layers instead of just chasing the best prediction possible for a given noise level.
Emmy: It sets up a clear theoretical framework showing how function-level learning and layerwise shrinkage work together perfectly, which is something we needed to address when we used DGPs as hierarchical representations, according to page two of THIS PAPER > Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation <ref:2610.23842#pg1,Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation>.
Kolya: It’s a way to manage the identifiability problem head-on by adding this penalty term, which is a neat structural fix rather than just tweaking the training objective.
Ada: Exactly. The authors show that this augmentation changes the posterior odds between symmetry-related decompositions by an exponential penalty ratio, which is how they get that shrinkage toward identity mapping in Theorem four point one of THIS PAPER > Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation <ref:2610.23842#pg1,posterior odds between symmetry-related decompositions by an exponential penalty ratio>.
Conclusion: Ada: So, we’re wrapping up this look at "Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation." This paper essentially shows how to make sense of those complex deep layer structures in deep Gaussian processes by fixing a fundamental problem with symmetry between adjacent layers.
Emmy: Exactly. It takes these flexible priors and gives us a way to pull those redundant layers back toward the identity mapping, which is what we call canonical representation.
Kolya: From my side, the main thing is that they prove this works even when you look at function-level contraction under proper prior conditions, which suggests it’s not just some lucky trick for specific data.
Ada: Right. So they’ve established a precise link between learning the overall regression function and shrinking those extra layers down to a simpler form, which is pretty clean in theory.
Emmy: It means that instead of having this huge, tangled mess of deep layer decompositions, you can have one that has a more stable structure underneath, even if the predictive accuracy doesn't jump dramatically.
Kolya: And they showed this augmentation actually resolves the identifiability issues caused by those adjacent layer symmetries by changing the posterior odds using an exponential penalty ratio.
Ada: So the core message is that we can use this method to select a more canonical, identity-biased layer representation, which is what we wanted structurally.
Emmy: It’s about finding a stable structure for these deep neural network layers within the Gaussian process framework without losing the overall function-level learning.
Kolya: And they point out that this effect is pretty robust because it remains effective even when the predictive advantage disappears, which is important for real applications.
Ada: It reinforces the idea that this isn't just about getting a better prediction; it’s about resolving redundant deep decompositions by selecting a more canonical, identity-biased layer representation.
Emmy: And now we have to look at what kinds of function families these deep Gaussian processes can actually handle, because that opens up a whole new area for estimation.
University of Hong Kong · Rice University · Lawrence Berkeley National Laboratory
math.ST, stat.ME, stat.TH
Submitted: 2026-10-16
Updated: 2026-11-17
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 91/100
The gist: Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions <ref:2610.23842#pg2>.
Key concepts
- Structural Non-identifiability
- This occurs because applying an invertible transformation and its inverse to two adjacent layers leaves the overall regression map unchanged. This means the model cannot uniquely determine which specific intermediate layer configuration is correct, even if the final output function is estimable.
- Augmented $ ho$-fractional Posterior
- This is a modified posterior used during inference that reweights different layer decompositions. It introduces a non-saturating penalty based on how much each layer deviates from an identity mapping. This penalty breaks the symmetry between equivalent layers, favoring simpler, more canonical representations.
- Function-level Contraction
- This refers to the property where the posterior distribution contracts towards a specific function in terms of R´enyi divergence. The augmentation ensures this contraction is preserved under proper conditions while simultaneously shrinking redundant layers toward the identity mapping.
Terminology
Summary
Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions <ref:2610. p. 0 Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio. Under proper choices of the penalty and a prior mass
condition, it preserves function-level fractional posterior contraction, while shrinking
the redundant layers towards the identity mapping. Thus, the likelihood does not
identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected
representative is not established.Where the paper says it
Where the paper says it
Where the paper says it
Structural Non-identifiability
Deep Gaussian processes (DGPs) are defined through compositions of latent GP layers, so the likelihood depends on the layer tuple only through the composed regression map. The adjacent-layer symmetry example shows that an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. p. 0 Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. The statistical consequence is that posterior contraction of C(θ) does not imply posterior contraction of θ.Where the paper says it
Where the paper says it
Symmetry-breaking by Augmentations
The authors introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. This augmentation changes posterior odds between symmetry-related decompositions through the difference in their total identity penalty. p. 0 We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio. We introduce augmented posteriors that
encourage each non-final DGP layer towards identity mapping. 1 shows such
augmentation resolves the identifiability issue caused by the adjacent layer symmetries.Where the paper says it
Where the paper says it
Function-level Contraction and Layerwise Parsimony
The augmented DGP can preserve function-level contraction in R´enyi divergence under proper prior mass conditions. Theorem 5.1 shows that if ηj,n are chosen to be relatively small for the active block, redundant layers are pulled towards the identity mapping. p. 0 The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio. Under proper choices of the penalty and a prior mass
condition, it preserves function-level fractional posterior contraction, while shrinking
the redundant layers towards the identity mapping. Our use of the identity is related but statistically
different: the classical DGP architecture remains, but a posterior factor tilts mass among
likelihood-equivalent decompositions. The novelty of the present work is motivated by a
non-concentration phenomenon, addressed by an exact posterior-odds symmetry-breaking
formula, and a set of rigorous conditions under which identity shrinkage of redundant layers
coexists with function-level contraction.Where the paper says it
Where the paper says it
Functional Spaces for DGP
The work investigates which function families DGPs can handle, suggesting they might lift adaptation to a new level beyond mere smoothness. Corollary 5. p. 8 The example does not apply to an
unmodified stationary Mat´ern first-layer process on all of R
d
, whose RKHS contains no
nonzero global constants. Corollary 3. The example does not apply to an
unmodified stationary Mat´ern first-layer process on all of R
d
, whose RKHS contains no
nonzero global constants. Corollary 3. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are
mostly function-level: they describe the learning of the composed regression map but not
the learning of the individual layers, calling into the question: how much can we rely on the
estimated intermediate layers when composition structures are of concern? , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are
mostly function-level: they describe the learning of the composed regression map but not
the learning of the individual layers, calling into the question: how much can we rely on the
estimated intermediate layers when composition structures are of concern?Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Empirical Findings
The comparison between regular DGP and augmented DGP shows that the augmented DGP improves prediction for low and moderate noise levels. The main message is that canonicalization is primarily a structural objective, but in favorable compositional regimes it can be obtained with little or no loss of predictive accuracy. The augmentation resolves redundant deep decompositions by selecting a more canonical, identity-biased layer representation. p. 0 Thus, the likelihood does not
identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected
representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures
∗Email: yczhu@hku.Where the paper says it
Conclusion
The theory establishes a precise compatibility between function-level learning and identity-biased shrinkage of redundant layers. The results support the main interpretation of the proposed augmentation, which is to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation. p. 0 Thus, the likelihood does not
identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected
representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures
∗Email: yczhu@hku. Thus, the likelihood does not
identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected
representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures
∗Email: yczhu@hku. Thus, the likelihood does not
identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected
representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures
∗Email: yczhu@hku. Thus, the likelihood does not
identify a unique “true” set of layers; the augmentation favors identity-biased parsimonious decompositions within an equivalence class of layers; uniqueness of a selected
representative is not established. Keywords: Gaussian processes, deep structures, identifiability, layer structures
∗Email: yczhu@hku. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are
mostly function-level: they describe the learning of the composed regression map but not
the learning of the individual layers, calling into the question: how much can we rely on the
estimated intermediate layers when composition structures are of concern? , 2018; Schmidt and
O’Hagan, 2003). In such cases, the likelihood does not select a unique layer representation,
and the posterior may fail to concentrate on any fixed representative when the prior remains
sufficiently symmetric.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
Bhattacharya, A., D. Pati, and Y. Yang (2019) is cited for Bayesian fractional posteriors. p. 7 Specifically, for ρ ∈ (0, 1], the ρ-fractional posterior of
θ = (f1, f2, · · · , fJ ) is defined as
Πρ(dθ|Dn) ∝ [p(Dn|C(θ))]ρΠ(dθ). When ρ = 1, the ρ-fractional posterior reduces to the standard Bayesian posterior. Specifically, for ρ ∈ (0, 1], the ρ-fractional posterior of
θ = (f1, f2, · · · , fJ ) is defined as
Πρ(dθ|Dn) ∝ [p(Dn|C(θ))]ρΠ(dθ). When ρ = 1, the ρ-fractional posterior reduces to the standard Bayesian posterior. , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. 1 Introduction
Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. 1 Introduction
Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. 1 Introduction
Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. 1 Introduction
Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. (2018) study how depth
changes DGP behavior. Residual networks instead parameterize a layer as identity plus a
perturbation (He et al. (2018) study how depth
changes DGP behavior. Residual networks instead parameterize a layer as identity plus a
perturbation (He et al. ∼ N(0, σ2
), (3)
with f
∗
: X → R being the data-generating function. By putting a DGP prior (2) on the
regression function f
∗
in (3), we obtain a valid Bayesian model. ∼ N(0, σ2
), (3)
with f
∗
: X → R being the data-generating function. By putting a DGP prior (2) on the
regression function f
∗
in (3), we obtain a valid Bayesian model. 23842v1 [math. ST] 16 Oct 2026 23842v1 [math. ST] 16 Oct 2026 1 Introduction
Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. , 2003; Zammit-Mangion
et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and
Mohamed, 2015). , 2003; Zammit-Mangion
et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and
Mohamed, 2015). , 2003; Zammit-Mangion
et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and
Mohamed, 2015). , 2003; Zammit-Mangion
et al. , 2022); compositions of invertible maps also underlie normalizing flows (Rezende and
Mohamed, 2015). They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al. , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. (2020); here we quantify non-concentration
and study an explicit representation preference. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. (2020); here we quantify non-concentration
and study an explicit representation preference. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. (2020); here we quantify non-concentration
and study an explicit representation preference. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. (2020); here we quantify non-concentration
and study an explicit representation preference. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). (2020); here we quantify non-concentration
and study an explicit representation preference. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. (2020); here we quantify non-concentration
and study an explicit representation preference. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2022; Salimbeni
and Deisenroth, 2017; Zhu et al. , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are
mostly function-level: they describe the learning of the composed regression map but not
the learning of the individual layers, calling into the question: how much can we rely on the
estimated intermediate layers when composition structures are of concern? , 2024), and theoretically, because suitably constructed
DGP priors can attain minimax contraction rates (up to a poly-logarithm factor in sample
size) for nonparametric regression of compositional smooth functions (Castillo and Randrianarisoa, 2025; Finocchio and Schmidt-Hieber, 2023). Existing guarantees, however, are
mostly function-level: they describe the learning of the composed regression map but not
the learning of the individual layers, calling into the question: how much can we rely on the
estimated intermediate layers when composition structures are of concern? Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Lower bounds for shallow GP priors establish advantages of depth for compositional classes
(Giordano et al. Compositional non-identifiability and uncertainty over DGP layers
are explicitly studied by Ustyuzhaninov et al. Lower bounds for shallow GP priors establish advantages of depth for compositional classes
(Giordano et al. Compositional non-identifiability and uncertainty over DGP layers
are explicitly studied by Ustyuzhaninov et al. Compositional non-identifiability and uncertainty over DGP layers
are explicitly studied by Ustyuzhaninov et al. (2020); here we quantify non-concentration
and study an explicit representation preference. Compositional non-identifiability and uncertainty over DGP layers
are explicitly studied by Ustyuzhaninov et al. (2020); here we quantify non-concentration
and study an explicit representation preference. With a little abuse of notations, we also use
Π(dfj ) to denote the marginal prior law on the jth layer, which is itself the product of the
coordinate-wise GP law of fj,k. For each layerwise process GPj
, let Hj denote its reproducing
kernel Hilbert space (RKHS), and ∥·∥Hj be its corresponding RKHS norm. With a little abuse of notations, we also use
Π(dfj ) to denote the marginal prior law on the jth layer, which is itself the product of the
coordinate-wise GP law of fj,k. For each layerwise process GPj
, let Hj denote its reproducing
kernel Hilbert space (RKHS), and ∥·∥Hj be its corresponding RKHS norm. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. , 2003; Zammit-Mangion
et al. Second, warping and deformation models
use latent transformations to represent nonstationarity or non-Gaussian structure (Sampson and Guttorp, 1992; Schmidt and O’Hagan, 2003; Snelson et al. , 2003; Zammit-Mangion
et al. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. This is the adjacent-layer symmetry example that motivates and drives our methodology. This phenomena generalizes the
identifiability issue in latent factor models (Gu and Xu, 2020; Papastamoulis and Ntzoufras,
2022) to functional spaces, and cannot be simply attributed to multi-modal posterior. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
Deep Gaussian processes (DGPs) are flexible priors for compositional regression functions, but inference for intermediate layers faces questions. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. p. 0 Canonical Layer Inference for Deep Gaussian
Processes via Symmetry-Breaking
Augmentation
Yichen Zhu∗
Department of Statistics, the University of Hong Kong
Hengrui Luo†
Department of Statistics, Rice University
and
Computational Research Division, Lawrence Berkeley National Laboratory
Abstract
Deep Gaussian processes (DGPs) are flexible priors for compositional regression
functions, but inference for intermediate layers faces questions. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. The basic obstruction
is an adjacent-layer symmetry: applying an invertible transformation and its inverse
to two adjacent layers leaves the composed regression map unchanged. Under explicit
symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed
regression function is estimable. Under explicit
symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed
regression function is estimable. We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty.Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The authors introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions by an exponential penalty ratio. p. 0 We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio. The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio. Under proper choices of the penalty and a prior mass
condition, it preserves function-level fractional posterior contraction, while shrinking
the redundant layers towards the identity mapping. The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio. Under proper choices of the penalty and a prior mass
condition, it preserves function-level fractional posterior contraction, while shrinking
the redundant layers towards the identity mapping.Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The augmentation changes the target only through the additive term PJ−1j=1 ln Rj (fj), so standard DGP variational or MCMC schemes can be reused with a simple penalty correction. The algorithm 2 summarizes the corresponding sparse-variational objectives for the unaugmented and augmented targets. p. 5 With a little abuse of notations, we also use
Π(dfj ) to denote the marginal prior law on the jth layer, which is itself the product of the
coordinate-wise GP law of fj,k. For each layerwise process GPj
, let Hj denote its reproducing
kernel Hilbert space (RKHS), and ∥·∥Hj be its corresponding RKHS norm.Where the paper says it
Where the gains come from
Theorem 4.1 gives the exact change in posterior mass between symmetry-related sets through the product of the prior Radon-Nikod´ym factor and the exponential difference in identitydeviation penalties. p. 3 Our use of the identity is related but statistically
different: the classical DGP architecture remains, but a posterior factor tilts mass among
likelihood-equivalent decompositions. The novelty of the present work is motivated by a
non-concentration phenomenon, addressed by an exact posterior-odds symmetry-breaking
formula, and a set of rigorous conditions under which identity shrinkage of redundant layers
coexists with function-level contraction. We then introduce an augmented ρ-fractional posterior that reweights layer decompositions by a non-saturating identity-deviation penalty. The augmentation changes posterior odds between symmetry-related decompositions
by an exponential penalty ratio.Where the paper says it
Where the paper says it
The authors investigate which function families DGPs can handle, suggesting they might lift adaptation to a new level beyond mere smoothness. Corollary 5.4 shows that for intrinsic lower dimensional structures, the DGP can be optimally estimated at the rate of n−1/3.
The theory establishes a precise compatibility between function-level learning and identity-biased shrinkage of redundant layers. The results support the main interpretation of the proposed augmentation, which is to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation. The most robust effect is not merely to improve prediction, but to resolve redundant deep decompositions by selecting a more canonical, identity-biased layer representation.
Bhattacharya, A., D. Pati, and Y. Yang (2019) is cited for Bayesian fractional posteriors. Castillo, I. and T. Randrianarisoa (2025) are cited for Deep horseshoe gaussian processes. Damianou, A. and N. D. Lawrence (2013) are cited for Deep Gaussian processes. Dunlop, M. M., M. Girolami, A. M. Stuart, and A. L. Teckentrup (2018) are cited for How deep are deep Gaussian processes? Finocchio, G. and J. Schmidt-Hieber (2023) are cited for Posterior contraction for deep gaussian process priors. Gu, Y. and G. Xu (2020) are cited for Partial identifiability of restricted latent class models. He, K., X. Zhang, S. Ren, and J. Sun (2016) are cited for Deep residual learning for image recognition. Linero, A. R. (2025) is cited for Defensive model expansion for robust bayesian inference. Luo, H., A. Horiguchi, and L. Ma (2026) are cited for Efficient decision trees for tensor regressions. Luo, H., G. Nattino, and M. T. Pratola (2023) are cited for Sparse additive gaussian process regression. p. 1 1 Introduction
Deep Gaussian processes (DGPs) model a regression function as a composition of Gaussian process (GP) layers. They are attractive both empirically, for capturing nonstationary
and compositional structures (Damianou and Lawrence, 2013; Luo et al.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Canonical Layer Inference for Deep Gaussian Processes via Symmetry-Breaking Augmentation: The augmented DGP can resolve identifiability issues caused by adjacent layer symmetries by reweighting layer decompositions with a non-saturating identity-deviation penalty, favoring identity-biased parsimonious representations within an equivalence class of layers.
-
Function Level Contraction Preservation: By choosing penalty parameters eta j n appropriately, the augmented DGP preserves function level contraction in R´enyi divergence while shrinking redundant layers towards the identity mapping, thus selecting a canonical representative without changing the DGP architecture.
-
Layerwise Parsimony Achievement: The system can be tuned such that redundant layers are forced toward near-identity mappings while active blocks maintain their prior mass, achieving layerwise parsimony by setting the tuning window n epsilon squared n less than or equal to eta j n less than or equal to n epsilon squared n over epsilon˜n for layers j less than J0.
-
Handling Local and Intrinsic Structures: The augmented DGP framework can be adapted to estimate functions with local structures, such as non-differentiable cusps, by using a proxy function and carefully chosen regularization parameters that allow the model to contract to the true function at rates faster than those achievable by smoother models.
-
Discontinuous Function Estimation: For discontinuous step functions, the augmented DGP can achieve a posterior contraction rate of n minus alpha squared plus three alpha times d for some choice of J and penalty weights, allowing estimation rates potentially approaching parametric rates under favorable conditions.
Abstract
Deep Gaussian processes (DGPs) are flexible priors for estimating compositional functions, but function-level accuracy does not determine their intermediate-layer representations. The basic obstruction is an adjacent-layer symmetry: applying an invertible transformation and its inverse to two adjacent layers leaves the composed regression map unchanged. Under explicit symmetry and prior-comparability conditions, we show that the posterior may not concentrate on any prescribed intermediate-layer representation, even when the composed regression function is estimable. We introduce a layerwise exponential tilt of the DGP prior, using a penalty proportional to each hidden layer's distance from the identity map, with stronger weights on earlier layers. The methodology links an exact posterior-odds rule along likelihood-preserving symmetries to quantitative control of redundant layers. Under proxy and local prior-mass conditions, we derive a penalty window that preserves function-level fractional posterior contraction while localizing redundant layers at certain radius. The resulting design rule calibrates representation regularization to the function-estimation rate. It favors decompositions with earlier layers close to identity; uniqueness and automatic depth selection are not established. The general theory and methodology are supported with detailed examples.
Sources
Related papers
- Sigmoid-FTRL: Design-Based Adaptive Neyman Allocation for AIPW Estimators
- Nonparametric Regression on Fixed-Cardinality Subsets:Minimax Risk and Random-Design Effects
- Conformal Prediction for Dyadic Regression Under Complex Missingness
- Optimal Linear Functional Estimation for Proportionally High-Dimensional Linear Models
- Semiparametric Efficient Inference under Non-Informative Complex Survey Designs
- Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel