Occam's Razor is Only as Sharp as Your ELBO
This study investigates how using Evidence Lower Bound (ELBO) for hyperparameter learning in Bayesian linear regression can lead to overfitting or underfitting based on assumptions about the covariance matrix structure. It proves that assuming a rank-1 covariance matrix causes the ELBO to underestimate signal-to-noise ratio, leading to overfitting, contrasting with diagonal assumptions that cause underfitting. The findings suggest caution when using ELBO for model selection.