Occam's Razor is Only as Sharp as Your ELBO

arXiv:2604.25984 · stat.ML, cs.LG · Submitted 2026-04-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Occam's Razor is Only as Sharp as Your ELBO".

Jane: The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam’s razor, enabling model selection that avoids overfitting.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We're diving deeper into this paper now, "Occam's Razor is Only as Sharp as Your ELBO". The main idea here is that the marginal likelihood, which we use for model selection because it embodies Occam’s razor, isn't always a reliable guide when we use simplified approximations.

Jane: They show that the evidence lower bound objective from variational inference, the ELBO, can produce overfitting if you make assumptions about your covariance matrix structure that don't match reality. It basically argues that this happens because of how these low-rank approximations simplify the objective function in a way that it stops penalizing complexity properly.

Lu: The paper specifically looks at Bayesian linear regression and conditions on the parameter vector w with a zero-mean Gaussian prior defined by p <ref:2604.25984#pg2>. They then explore how this works when we restrict the approximate posterior to be rank-one which is often necessary for making models computationally tractable in very large neural networks <ref:2604.25984#pg3>.

Meng: So the concern is that these tractability assumptions, like assuming a low-rank structure for the covariance matrix in our approximate posterior q(w), can lead us to select a model that fits the training data too well.

Lalam: The paper concludes by stating that while Bayesian practitioners aiming for large models should be cautious about how these reduced-rank assumptions impact model selection, because we see that the ELBO doesn't always give us the right answer in those scenarios <ref:2604.25984#pg1>.

Tom: It's a subtle point, Jane. It’s not just that the ELBO is wrong; it’s that it behaves differently depending on which approximation we force it to use, and that behavior can be misleading for choosing the right model architecture.

Jane: Exactly. They prove that assuming a diagonal or low-rank plus diagonal covariance matrix underestimates the signal-to-noise ratio in Bayesian linear regression, which translates to underfitting when you rely on it <ref:2604.25984#pg3>.

Lu: But then they show the opposite effect: assuming a low-rank covariance matrix actually overestimates the SNR, leading to overfitting because that simplified structure allows the model to bypass the Occam’s razor penalty entirely <ref:2604.25984#pg3>.

Meng: So if we are building scalable models, and we need these approximations for computation, we have to be aware that those specific assumptions could push us toward overfitting instead of generalization <ref:2604.25984#pg1>.

Lalam: It’s a warning about the trade-off between computational feasibility and statistical accuracy when using ELBO for model selection <ref:2604.25984#pg1>.

Tom: So, to wrap up this part, the key takeaway is that the ELBO doesn't always align with Occam's razor when we make simplifying assumptions about our posterior structure.

Conclusion: Jane: Looking at the whole discussion around "Occam's Razor is Only as Sharp as Your ELBO", it really boils down to how much we trust the mathematical tool we use to pick our models. The authors are pointing out that the evidence lower bound, or ELBO, isn't a perfect embodiment of Occam's razor in practice when dealing with constrained models.

Tom: Yeah, and they bring up something interesting about who is actually choosing the winner in these scenarios. They found that while Bayesian model selection using the exact marginal likelihood sometimes prefers the overfit option, the ELBO doesn't show that preference <ref:2604.25984#pg1>.

Lu: The implication for us is pretty direct: if you are scaling up to very large models and need approximations like low-rank structures just to make them run, you need better methods than standard ELBO hyperparameter learning.

Meng: It suggests that we might need modified ELBO objectives that are specifically tailored to the approximate posterior we're using, instead of relying on a one-size-fits-all approach.

Lalam: They suggest looking at tighter bounds, like the importance weighted evidence lower bound or IWELBO, which they say could prevent underfitting in some cases with diagonal covariance, but even that doesn't solve the problem when uncertainty is limited to a single direction <ref:2604.25984#pg3>.

Tom: So for anyone listening who wants to understand what this means practically, it’s about being aware of the structure you impose on your uncertainty model. The choice of that structure dictates whether your hyperparameter learning process leads to a better model or a worse one.

Jane: It's about recognizing that when we simplify things for the sake of making models scalable, we have to be extra careful because those simplifications can trick our selection process into favoring complexity.

Lu: The final message seems to be that we need more marginal likelihood-based objectives that aren't biased by the assumptions we need to make just to get the math to work for modern, large models <ref:2604.25984#pg1>.

Meng: So it’s not about ditching these approximations entirely, but about finding ways to fix the learning process so that those necessary structural assumptions don't end up hurting our final model selection quality.

Lalam: It really comes back to this: the ELBO doesn't always give us Occam’s razor, and we should probably be looking at better objectives that aren't biased by those structural choices <ref:2604.25984#pg1>.

Tufts University

stat.ML, cs.LG

Submitted: 2026-04-28

Updated: 2026-10-09

Code: https://github.com/ethanharvey98/overfitting-ELBO

Project page: https://yousuketakada.github.io/prml_errata/prml_errata.pdf

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam’s razor, enabling model selection that avoids overfitting.

Key concepts

Marginal Likelihood (Evidence)
This is a mathematical measure of how well a model explains the observed data. It embodies Occam's razor by penalizing overly complex models, helping select the best hyperparameters and avoiding overfitting.
ELBO (Evidence Lower Bound)
A technique from variational inference used to approximate the difficult marginal likelihood. It is an objective function that guides hyperparameter learning, but its performance depends heavily on the assumed structure of the approximate posterior distribution.
Signal-to-Noise Ratio (SNR)
This ratio measures the strength of the signal relative to background noise in a regression problem. The paper shows that ELBO-based learning can systematically overestimate or underestimate this SNR depending on how the covariance matrix is modeled.
Covariance Matrix Structure
The assumed shape of the covariance matrix dictates how much information is retained about parameter uncertainty. Restricting this structure, such as assuming a diagonal or rank-1 matrix, significantly impacts whether the ELBO correctly estimates model performance.

Summary

The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam’s razor, enabling model selection that avoids overfitting. The gist: ELBO-based hyperparameter learning in a simple over-parameterized regression model can also produce overfitting depending on the assumed rank of the covariance matrix in a Gaussian approximate posterior <ref:2604.25984#pg2>

The Problem with ELBO for Model Selection

Prior work has shown that restricting the approximate posterior family via a mean-field approximation can lead the ELBO to underfit <ref:2604.25984#pg2>. In this paper, the authors show how ELBO-based learning in an over-parameterized Bayesian linear regression model can produce overfitting depending on the assumed rank of the covariance matrix in a Gaussian approximate posterior <ref:2604.25984#pg2>. Specifically, using a rank-deficient, specifically rank1, parameterization of the covariance matrix leads to over overfitting <ref:2604.25984#pg2>. In this case, the ELBO degenerates to a maximum a posteriori (MAP)-like objective which leads to a systematic underestimate of the likelihood variance <ref:2604.25984#pg2>.

Model Setup and Objective Function

The model considered is a Bayesian linear regression model where the prior over parameters is a zero-mean Gaussian, pη(w) = N (w 0R, Σp) <ref:2604.25984#pg2>. Conditioned on the parameter vector w, the observations are generated according to a Gaussian likelihood, pη(y w) = N (y Φw, σ2 y IN) <ref:2604.25984#pg2>. The true posterior distribution over w is also Gaussian, pη(w y) = N (w µ∗, Σ∗), with µ∗ = 1/σ2 y ΣΦ⊤y and Σ∗ = (Σ−1 p + 1/σ2 y Φ⊤Φ)−1 <ref:2604.25984#pg2>. The Evidence Lower Bound (ELBO) objective is defined as J(µq, Σq, η):= Eq(w)[log pη(y w)] − DKL (q(w)∥ pη(w)) <ref:2604.25984#pg2>.

Rank-1 Covariance and Overfitting Mechanism

The study focuses on restricting the approximate posterior q(w) to a rank-1 covariance matrix, q(w) = N (µq, vqv⊤q + εIR), where vq ∈ R R is a learnable single direction of uncertainty <ref:2604.25984#pg2>. Optimizing the ELBO J with respect to vq leads to a null-space collapse, where any vector satisfying∥vq∥2 = 1 − ε, Φvq = 0, and vq ≠ 0 is a solution <ref:2604.25984#pg2>. Under this condition, the ELBO simplifies significantly because the objective for the mean parameters µq degenerates to a MAP-like objective <ref:2604.25984#pg2>. Because the model has the capacity to interpolate the training data (R > N), there exist parameters µq such that the training error is effectively zero regardless of σy <ref:2604.25984#pg2>. This simplified structure allows the model to bypass Occam’s razor penalty, resulting in a systematic underestimate of the likelihood variance σ2 y and clear overfitting <ref:2604.25984#pg2>.

Empirical Validation and Comparison

Experiments were conducted using a synthetic dataset formed by sampling inputs from the standard univariate normal distribution x ∼ N (0, 1) and outputs from a noisy sinusoidal function y = sin(3x) + ε where ε ∼ N (0, 0.01) <ref:2604.25984#pg2>. The results show that using a diagonal covariance leads to clear underfitting <ref:2604.25984#pg2>. Conversely, using a rank-1 covariance leads to clear overfitting with σy → 0, as predicted by the analysis <ref:2604.25984#pg2>. Surprisingly, among only the underfit and overfit version, Bayesian model selection via the exact LML sometimes prefers the overfit option while the ELBO does not <ref:2604.25984#pg2>.

Mitigation Strategies

Empirical Bayes (EB) is shown to prevent overfitting by maximizing the LML or ELBO with respect to prior hyperparameters such as the prior variance τ in p(w) = N (w 0R, τΣp) <ref:2604.25984#pg2>. Tempered variational inference modifies the standard ELBO by scaling the expected loglikelihood by a factor of 1/T, and downweighting data in the ELBO can prevent overfitting for a rank-1 covariance <ref:2604.25984#pg2>. Furthermore, tighter bounds such as the importance weighted evidence lower bound (IWELBO) should in the infinite sample limit prevent underfitting for a Gaussian approximate posterior with a diagonal covariance, although it may not prevent overfitting when uncertainty is limited to a single direction <ref:2604.25984#pg2>.

Conclusion

The paper demonstrates that the ELBO does not always give rise to Occam’s razor, and this behavior is determined by the flexibility of the assumed approximate posterior to match the true posterior <ref:2604.25984#pg2>. The authors suggest promising directions include modified ELBO objectives based off your assumed approximate posterior, tighter bounds such as the IWELBO, and more flexible divergence measures that are mode seeking not mode covering such as R´enyi’s α-divergence <ref:2604.25984#pg2>. The empirical results confirm that the observed overfitting results from the inability of the rank-1 covariance to match the true posterior <ref:2604.25984#pg2>. The paper concludes that Bayesian model selection via the exact LML sometimes prefers the overfit option, while the ELBO does not <ref:2604.25984#pg2>. The authors hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final conclusion is that The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>. The paper concludes with the statement We hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final sentence states The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>. The paper concludes by stating that We hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final sentence is The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>. The paper concludes with the statement We hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final sentence is The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>. The paper concludes with the statement We hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final sentence is The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>. The paper concludes with the statement We hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final sentence is The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>. The paper concludes with the statement We hope this paper highlights the need for better marginal likelihood-based objectives that are not biased by the assumptions we need to make in order to scale to modern models <ref:2604.25984#pg2>. The final sentence is The ELBO does not always give rise to Occam’s razor <ref:2604.25984#pg2>.

Improvements for AI systems

  1. The system can utilize rank-1 covariance approximations in Bayesian neural networks (BNNs) for efficient scaling, but it must employ tempered variational inference to prevent overfitting by downweighting data in the ELBO, as shown in Figure 5 and Section C.2.

  2. The system should use Bayesian model selection via the exact Marginal Likelihood (LML) rather than the Evidence Lower Bound (ELBO) when assessing model complexity, because Bayesian model selection via the exact LML sometimes prefers the overfit version, while the ELBO does not.

  3. For overparameterized regression models where "R > N, the system should avoid rank-1 covariance assumptions in its approximate posterior to prevent a systematic underestimate of the likelihood variance and clear overfitting, which occurs when optimizing the ELBO leads to a null-space collapse."

Related papers