Isotropic Gaussian Processes Improve Vanilla Bayesian Optimization in High Dimensions
stat.ML, cs.LG
Submitted: 2026-10-05
Updated: 2026-10-05
Code: https://github.com/XZT008/Standard-GP-is-all-you-need-for-HDBO
License: http://creativecommons.org/licenses/by/4.0/
The gist: High-dimensional Bayesian optimization (BO) often fits Gaussian process (GP) surrogates from far fewer observations than input dimensions.
Terminology
Abstract
High-dimensional Bayesian optimization (BO) often fits Gaussian process (GP) surrogates from far fewer observations than input dimensions. Modern Vanilla BO can perform well in this regime with dimension-aware priors, initialization, and acquisition optimization, but it typically retains automatic relevance determination (ARD), fitting one lengthscale per input coordinate. We study this modeling choice and propose Iso-BO, a controlled modification that replaces the ARD GP with an isotropic GP using one shared lengthscale while keeping the surrounding BO pipeline matched. For radial kernels, we show that the marginal log likelihood (MLL) depends on the inverse-squared ARD lengthscales only through weighted pairwise distances among the observed inputs. The current design can therefore leave some ARD directions exactly invisible or only weakly constrained by the MLL. Iso-BO removes coordinatewise reweighting and fits a single shared scale instead. Lengthscale-fitting and predictive-density diagnostics show that this finite-data effect appears in practice, including when the data-generating process is anisotropic. Across GP-prior, synthetic, and real-world benchmarks, Iso-BO often improves over matched modern Vanilla BO and remains competitive with the included high-dimensional BO baselines under the tested budgets. Stress tests also show the expected boundary wherein sufficiently strong, learnable anisotropy can favor the more flexible ARD model.
Sources
- We Still Don't Understand High-Dimensional Bayesian Optimization
- Leveraging Axis-Aligned Subspaces for High-Dimensional Bayesian Optimization with Group Testing
- High Dimensional Bayesian Optimization using Lasso Variable Selection
- High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy
- Exploring Exploration in Bayesian Optimization
- Rethinking Trust Region Bayesian Optimization in High Dimensions
- Gymnasium: A Standard Interface for Reinforcement Learning Environments
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier