Large-scale empirical tuning and comparison of default optimizers for variational inference
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Large-scale empirical tuning and comparison of default optimizers for variational inference".
Jane: Here's the gist of the paper. Large-scale empirical tuning and comparison of default optimizers for variational inference investigates new adaptive stochastic optimization algorithms in black-box variational inference to establish a state of the art in tuning-free optimization based inference The Core Problem Black-box variational inference (BBVI) generally requires extensive problem-specific tuning for its underlying stochastic gradient-based optimization algorithms, which undermines its promise as a truly “black box” inference algorithm The goal of this work is to characterize the current landscape of these methods in the context of BBVI and determine which can be used to produce a truly automated and tuning-free methodology Empirical Evaluation Setup The researchers performed a large-scale empirical evaluation involving 56 stochastic gradient-based optimization algorithms applied to 1092 Bayesian inference optimization problems, encompassing over 550,000 individual optimization runs and roughly 15 core-years of compute The benchmark problems span a range of difficulty including posterior target dimension 1–104, condition number 1–108, and various variational families The test suite is built from posteriordb v0.6.0 augmented to include noisy evaluation of log posterior densities and gradients due to data subsampling Optimization Algorithm Taxonomy The 56 algorithms are categorized into five families: SGD-Like methods, Stochastic Line Search algorithms, Increasing Batch Size methods, Meta-tuners, and a few others The SGD-Like methods include AdaGrad
32: , Adam
59: , AMSGrad
86: , and Lion
16: Meta-tuners include HyperGradient
3: and Robust and Automated Black-Box VI (RABVI)
108: The test suite includes four common objective functions for each of the 146 posterior distributions, such as maximum a posteriori (MAP) optimization and Kullback-Leibler divergence minimization with a diagonal Gaussian variational family (DiagGaussianVI)
46, 81: Ensemble Performance and Findings The top performer is not a single method, but an ensemble of just 5 default methods that rarely fails and consistently provides near-optimal performance among the set of 56 algorithms This ensemble consists of Adam with step size 10−3 and 10−4, distance-over-weighted-gradient (DoWG) with step size 1, Lion with step size 10−5, and our novel variant of SAA-LBFGS with initial step size 10−8 The Ensemble method performs reliably well compared to the 56 methods across problem and noise types, despite itself only involving running 5 easily-implemented methods There is no clear single best default algorithm across all problems, but running a selection of 5 algorithms suffices to reliably get close to the best-possible observed performance The suggested default step sizes from this paper are not meant to be a panacea but should find value in three scenarios: as good default step sizes when expert tuning is expensive or not possible, as step size values to use when comparing to genuinely tuning-free methods in follow-up research, and as good starting points for expert tuning on individual problems One key takeaway is that an ensemble of 5 methods generally outperforms the whole original set of 56 optimization algorithms The Ensemble method is more likely to find fruitful use as a solid baseline panel of 5 methods against which one should compare other proposed tuning-free optimizers Appendix Details The paper provides detailed pseudocode for neural network architectures in the VAE problems, showing that the parameter dimension can be as high as 8d squared + 844d Appendix A details how Stan posteriors were modified to enable subsampled estimates of sums in the target log probability density Appendix B describes the neural network structure for both the encoder and decoder, resulting in a total of 8d squared + 844d tunable parameters for VAE inference problems Appendix C details the pseudocode for Sample Average Approximation (SAA) methods, showing how to implement variants like SAA and SAACG The streaming RABVI algorithm is also detailed, which uses a compressed batch summaries to monitor convergence diagnostics The results of the comparison of tuned methods are displayed in Figs.
Tom: 6 to 11 The results of this exercise are shown for a representative subset of variational family types and types of stochasticity in Fig.
Jane: 12 Final Recommendation One key takeaway from this paper is that an ensemble of 5 methods consisting of Adam with step size 10−3, Adam with step size 10−4, DoWG with step size 1, Lion with step size 10−5, and SAALBFGS with step size 10−8 generally outperforms the whole original set of 56 optimization algorithms In future work, this ensemble could certainly be implemented as its own method given the availability of sufficient hardware parallelism However, we view the value of this method more as a reduction in the work required for comparisons in future work The results of the comparison of tuned methods are displayed in Figs.
Title and authors: Tom: So we're looking at this paper titled "Large-scale empirical tuning and comparison of default optimizers for variational inference". It sounds like they're trying to figure out which optimization methods you should just use by default when you're doing black box variational inference.
Jane: That’s right, Tom. The authors are Campbell, Huggins, Kim, and Margossian. They’ve put together a really big study here because right now we know that these stochastic gradient based algorithms usually need a ton of problem specific tuning to work well in black box variational inference >
Lu: It’s interesting because the whole point of black box variational inference is supposed to be general, automated Bayesian inference without you having to worry about all those modeling constraints like coordinate ascent or expectation propagation >
Meng: Yeah, but they found that the optimization part is where it falls short. If you don't tune it right for a specific problem, the whole thing becomes unreliable and not robust >
Lalam: This paper investigates a collection of fifty-six different stochastic gradient based optimization algorithms applied to one thousand ninety-two Bayesian inference optimization problems, involving over half a million individual runs and about fifteen core years of compute > <ref:2606.07841#pg1,stochastic gradient based optimization algorithms applied to 1092 Bayesian inference optimization problems>
The paper's summary: Tom: They did this massive empirical evaluation across a really wide range of problems. They tested these fifty-six algorithms on things like posterior target dimensions ranging from one to one hundred and eight, and condition numbers from one to one hundred and eight >
Jane: That’s a lot of different scenarios they ran through. The core problem they’re tackling is that black box variational inference needs extensive tuning for its underlying optimization methods, which kills its promise of being a truly automated inference method >
Lu: They set up this benchmark using the posteriordb version zero point six point zero augmented to include noisy evaluations of log posterior densities and gradients because they subsampled the data >
Meng: So they’re testing them on actual problems, but with some noise thrown in, which is realistic for when you’re actually deploying models >
Lalam: The goal was to see if any of these new adaptive stochastic optimization algorithms could reduce or remove the need for tuning entirely and establish what the state of the art looks like >
The paper's improvements: Tom: Now, here’s where it gets interesting. They didn't find one single perfect default algorithm; instead, they identified an ensemble of just five methods that rarely fails and consistently performs near optimal among all fifty-six options >
Jane: That’s a big finding because it means you don't have to test every single algorithm yourself to get good results; you can just use this small group >
Lu: The ensemble consists of Adam with step size one zero minus three and one zero minus four, distance over weighted gradient with step size one, Lion with step size one zero minus five, and their novel variant of SAA lbfgs starting at a step size of one zero minus eight >
Meng: That’s actually quite small. They found that this ensemble performs reliably well across different problem types and noise levels, even though it only involves running five easily implemented methods >
Lalam: The authors suggest that this ensemble can be used as a solid baseline panel of five methods for follow up work on tuning free optimization in variational inference >
Conclusion: Tom: So, to wrap up the main idea, they found that running an ensemble of these five default methods is generally better than using the whole set of fifty-six algorithms when you want reliable performance >
Jane: The paper suggests those recommended default step sizes aren't meant to be a one size fits all solution; they’re more useful as good starting points for expert tuning when it’s expensive or impossible, or for comparing against methods that claim to be truly tuning free later on >
Lu: They also suggest these step sizes can find value as starting points even for an expert tuning effort on individual problems >
Meng: From a deployment standpoint, the main point is that this ensemble can serve as a solid baseline panel of five methods against which you should compare other proposed tuning free optimizers in future research >
Lalam: The paper "Large-scale empirical tuning and comparison of default optimizers for variational inference" shows that an ensemble of five methods generally outperforms the whole original set of fifty-six optimization algorithms >
Department of Statistics, University of British Columbia · Boston University · Department of Computer and Information Science, Penn State
stat.CO, cs.LG, stat.ML
Submitted: 2026-06-05
Updated: 2026-11-04
Code: https://github.com/trevorcampbell/subsampledposteriordb
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 81/100
The gist: Large-scale empirical tuning and comparison of default optimizers for variational inference investigates new adaptive stochastic optimization algorithms in black-box variational inference to…
Key concepts
- Black-Box Variational Inference (BBVI)
- This is a method for approximating complex probability distributions by finding simpler, tractable distributions. The core challenge is that the underlying optimization algorithms used to find these approximations often require extensive manual tuning specific to each problem.
- Ensemble Performance
- Instead of relying on one perfect algorithm, the study found that combining five different default methods—like Adam and Lion—creates a robust optimizer. This ensemble performs reliably across various problem types and noise levels, suggesting a set of defaults is better than any single method alone.
- Stochastic Gradient-Based Optimization
- These are iterative mathematical techniques used to find the best parameters for a model by taking small steps in the direction that reduces an error function. In this context, they are the engines that search for the optimal variational distribution.
Terminology
Summary
Large-scale empirical tuning and comparison of default optimizers for variational inference investigates new adaptive stochastic optimization algorithms in black-box variational inference to establish a state of the art in tuning-free optimization based inference
The Core Problem
Black-box variational inference (BBVI) generally requires extensive problem-specific tuning for its underlying stochastic gradient-based optimization algorithms, which undermines its promise as a truly “black box” inference algorithm The goal of this work is to characterize the current landscape of these methods in the context of BBVI and determine which can be used to produce a truly automated and tuning-free methodology
Empirical Evaluation Setup
The researchers performed a large-scale empirical evaluation involving 56 stochastic gradient-based optimization algorithms applied to 1092 Bayesian inference optimization problems, encompassing over 550,000 individual optimization runs and roughly 15 core-years of compute The benchmark problems span a range of difficulty including posterior target dimension 1–104, condition number 1–108, and various variational families The test suite is built from posteriordb v0.6.0 augmented to include noisy evaluation of log posterior densities and gradients due to data subsampling
Optimization Algorithm Taxonomy
The 56 algorithms are categorized into five families: SGD-Like methods, Stochastic Line Search algorithms, Increasing Batch Size methods, Meta-tuners, and a few others The SGD-Like methods include AdaGrad [32], Adam [59], AMSGrad [86], and Lion [16] Meta-tuners include HyperGradient [3] and Robust and Automated Black-Box VI (RABVI) [108] The test suite includes four common objective functions for each of the 146 posterior distributions, such as maximum a posteriori (MAP) optimization and Kullback-Leibler divergence minimization with a diagonal Gaussian variational family (DiagGaussianVI) [46, 81]
Ensemble Performance and Findings
The top performer is not a single method, but an ensemble of just 5 default methods that rarely fails and consistently provides near-optimal performance among the set of 56 algorithms This ensemble consists of Adam with step size 10−3 and 10−4, distance-over-weighted-gradient (DoWG) with step size 1, Lion with step size 10−5, and our novel variant of SAA-LBFGS with initial step size 10−8 The Ensemble method performs reliably well compared to the 56 methods across problem and noise types, despite itself only involving running 5 easily-implemented methods
Conclusion on Tuning-Free Methods
There is no clear single best default algorithm across all problems, but running a selection of 5 algorithms suffices to reliably get close to the best-possible observed performance The suggested default step sizes from this paper are not meant to be a panacea but should find value in three scenarios: as good default step sizes when expert tuning is expensive or not possible, as step size values to use when comparing to genuinely tuning-free methods in follow-up research, and as good starting points for expert tuning on individual problems One key takeaway is that an ensemble of 5 methods generally outperforms the whole original set of 56 optimization algorithms The Ensemble method is more likely to find fruitful use as a solid baseline panel of 5 methods against which one should compare other proposed tuning-free optimizers
Appendix Details
The paper provides detailed pseudocode for neural network architectures in the VAE problems, showing that the parameter dimension can be as high as 8d squared + 844d Appendix A details how Stan posteriors were modified to enable subsampled estimates of sums in the target log probability density Appendix B describes the neural network structure for both the encoder and decoder, resulting in a total of 8d squared + 844d tunable parameters for VAE inference problems Appendix C details the pseudocode for Sample Average Approximation (SAA) methods, showing how to implement variants like SAA and SAACG The streaming RABVI algorithm is also detailed, which uses a compressed batch summaries to monitor convergence diagnostics The results of the comparison of tuned methods are displayed in Figs. 6 to 11 The results of this exercise are shown for a representative subset of variational family types and types of stochasticity in Fig. p. 8 12
Final Recommendation
One key takeaway from this paper is that an ensemble of 5 methods consisting of Adam with step size 10−3, Adam with step size 10−4, DoWG with step size 1, Lion with step size 10−5, and SAALBFGS with step size 10−8 generally outperforms the whole original set of 56 optimization algorithms In future work, this ensemble could certainly be implemented as its own method given the availability of sufficient hardware parallelism However, we view the value of this method more as a reduction in the work required for comparisons in future work The results of the comparison of tuned methods are displayed in Figs.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive optimization for Bayesian inference: Implement an ensemble of five default methods (Adam with step size 10-3, Adam with step size 10-4, DoWG with step size 1, Lion with step size 10-5, and SAALBFGS with step size 10-8) as a single robust optimizer. This ensemble is shown to
perform reliably well compared to the 56 methods across problem and noise types.
-
Automated tuning for black-box inference: Utilize the recommended default step sizes from Table 2 column A, which are derived by
minimizing the average rank in Fig. 3
based on different performance criteria (final ELBO, integral of ELBO, final squared gradient norm, and integral of squared gradient norm). This provides asolid baseline for applications where expert tuning is not possible.
-
Scalable inference with streaming algorithms: Employ the Streaming RABVI algorithm to adaptively reduce the step size by monitoring convergence diagnostics like potential scale reduction factor Rb, effective sample size (ESS), and Monte Carlo standard error (MCSE). This allows for efficient optimization in high-dimensional problems by replacing
storage of the full iterate trace with tracking a hierarchy of batch sufficient statistics.
-
Robust sample approximation: Integrate the Sample Average Approximation (SAA) method with its
batch-size increase rule designed to avoid overoptimizing on small batches
to handle stochastic gradient estimation efficiently. This technique allows for optimization using a single, fixed estimate of the true objective function while dynamically increasing the batch size based on descent progress. -
Enhanced performance on difficult problems: Leverage Sample Average Approximation (SAA) methods, specifically SAANA and SAACG, as they
perform surprisingly well on MAP, DiagGaussianVI, and FullGaussianVI problems.
This indicates their utility in tackling high-dimensional or complex variational families effectively.
Sources
- Layer Normalization
- Revisiting the Polyak step size
- Variational Inference using Implicit Distributions
- Parameter-free Stochastic Optimization of Variationally Coherent Functions
- Automated Variational Inference in Probabilistic Programming
- Compressive Bayesian non-negative matrix factorization for mutational signatures analysis
Related papers
- Sticky Manifold Piecewise Deterministic Monte Carlo for Bayesian Model Averaging
- A perturbation theory for Feynman-Kac models and their particle interpretations
- Fast Functional Clustering via Closed-Form Covariance Updates
- A fast non-reversible sampler for Bayesian mixture models
- Inventory and Assortment Optimization under General Choice: A Unified Approximation Framework
- Least squares variational inference