Least squares variational inference

arXiv:2502.18475 · stat.CO · Submitted 2025-02-05 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Least squares variational inference".

Jane: Here's the gist of the paper. Least squares variational inference introduces LSVI as a gradient-free scheme for fixed-point recursion in exponential families, which is equivalent to biased stochastic natural gradient descent and yields efficient complexity for Gaussian distributions.

Tom: Core Methodology LSVI is introduced as a Monte Carlo-based scheme that performs ordinary least squares regression on tempered log-target evaluations under the variational approximation.

Jane: The exact LSVI iteration is defined by the fixed-point equation where the mapping boils down to performing successive least squares (OLS) regression.

Tom: This exact scheme is equivalent to a natural gradient descent iteration on the uKL divergence in the natural space of parameters and a mirror descent step in the moment space.

Title and authors: Tom: So we're looking at this paper called "Least squares variational inference", and it seems to be about finding a way to do variational inference when you don't have the gradients of the log target density.

Jane: Right, so the title itself suggests something about using least squares as a trick in this process. It sounds like they are tackling a problem where traditional methods that need those gradients just aren't available, which is a big deal for practical AI.

Lu: Yeah, I was looking at the core idea in Proposition two point three and it sets up this exact fixed-point equation where the update to the variational parameters is defined by some regression step <ref:2502.18475#pg3>.

Meng: Regression steps sound good theoretically, but from where I'm sitting, you need a concrete algorithm that runs fast and doesn't blow up on my production traffic, so what exactly is this least squares regression doing here?

Tom: Well, the paper says each iteration boils down to performing ordinary least squares regression on tempered log-target evaluations under the variational approximation. It connects this directly to biased stochastic natural gradient descent, which is interesting because it gives us a known path for optimization.

Jane: So if we think about it for a listener driving, it's like instead of trying to calculate the perfect steering angle by measuring every tiny shift in road conditions one by one, you use a regression model to predict the next best move based on what you've already seen.

Lu: That's kind of right, and they show this exact scheme is equivalent to a natural gradient descent iteration on the uKL divergence in the natural space of parameters and a mirror descent step in the moment space. It’s mapping things between different mathematical spaces to keep things stable.

Meng: Mapping spaces sounds complicated when you're trying to deploy something at scale, but if it avoids inverting that massive Fisher information matrix, that's definitely something I can get behind.

Tom: Exactly, and they tackle the big issue: when you're dealing with Gaussian variational families, LSVI exploits a reparameterization of the linear regression. This means they can eliminate the need for inverting the Fisher information matrix altogether.

Jane: Eliminating that inversion is huge because as you go up in dimension, say with high-dimensional data, that matrix gets quadratically larger, which makes traditional methods computationally impossible for many people to run on a regular computer.

Title and authors: Lu: That's the computational efficiency claim they make: O(d three) complexity in the full-covariance case and O(d) in the mean-field case <ref:2502.18475#pg1,O(d^3)$ complexity in the full>. That's a massive saving compared to what you'd typically expect from exact methods there.

Meng: O(d three) might still be slow if we have really high dimensions, but comparing that to having to invert a matrix whose size is proportional to the square of the data dimension, that makes sense for feasibility <ref:2502.18475#pg1>. What about the convergence part?

Tom: They do establish explicit rates for LSVI under standard smoothness and convexity assumptions on the uKL objective. For sufficiently small and linearly decreasing step sizes, they get an O(one/k) convergence rate with respect to the numbers of samples and iterations <ref:2502.18475#pg1,with respect to the numbers of samples and iterations>.

Jane: So, for a listener walking down the street, it means if you take small enough steps regularly, you know exactly how many steps it takes for your movement to settle into a steady pace. It gives us a concrete measure of how fast we're actually learning something useful.

Lu: They also condition their analysis on the event that estimated Fisher information matrices are well-conditioned, which is a necessary caveat because if the FIM isn't good, the whole regression model breaks down.

Meng: That conditioning requirement sounds like another thing I have to monitor in my production system; you can't just deploy it and hope the underlying assumptions about those matrices hold true for every single data batch.

Tom: The paper shows this method works across various tasks, including logistic regression, discrete variable selection, and Bayesian synthetic likelihood. In the case of Bayesian synthetic likelihood, full-covariance LSVI matches the posterior obtained via MCMC at a fraction of the CPU cost.

Jane: That's really telling for people who might be using complex simulations for modeling things; matching MCMC results but with much less computational overhead means we can run more complex analyses much faster.

Lu: They also show how this scheme is used to perform Bayesian variable selection on discrete exponential families, like Bernoulli products, by treating the problem as a regression of log target densities under tempered variational approximations. That shows its versatility beyond just Gaussian settings.

Title and authors: Meng: Discrete distributions are tricky because you can't always use those nice reparameterization tricks they rely on for Gaussians; so being able to apply this to discrete variable selection is valuable for real-world models that aren't perfectly continuous.

Tom: And the paper points out that this LSVI scheme can be used to perform gradient-free optimization without requiring explicit gradients of the log target density, which is useful when gradients are unavailable or noisy.

Jane: So, if you’re working on a model where the target function is really bumpy or non-differentiable, like in likelihood-free inference, this gives you a path forward using these regression steps instead of trying to calculate those difficult derivatives.

Lu: They also introduce some specific improvements in their stochastic variant that updates the OLS estimate using multiple draws from the current approximation. This is important because it makes it more robust when we're dealing with noisy estimates from the variational family itself.

Meng: Multiple draws sound like adding noise to a system, which might help stabilize things on my end, but I need to know if that extra noise translates into better performance or just more instability at scale.

Tom: The paper suggests adaptive step size control based on residual variance can be implemented to ensure convergence of LSVI-MF and LSVI-FC by controlling the linear regression residuals as described in Algorithm four <ref:2502.18475#pg3>. This should give us more robust performance even when exact smoothness and strong convexity parameters are unknown in practice.

Jane: So, instead of picking a fixed learning rate that might be too big or too small, the system actively adjusts its step size based on how much error is left in the regression model itself. That seems like smart engineering for real-world deployment.

Lu: It really addresses the practical difficulty of knowing those exact parameters upfront, which is a major hurdle when you're just trying to get a model running reliably.

Meng: If we can make the step size adapt based on the residuals, that reduces our need for overly conservative settings, which usually means faster overall training time and less wasted compute resources.

Tom: So, to wrap up the summary of this paper "Least squares variational inference", it introduces a gradient-free Monte Carlo scheme that boils down to OLS regression and shows equivalence to biased stochastic natural gradient descent.

Title and authors: Jane: It really points toward making variational inference practical even when we can't get those perfect gradients, and it does so by using clever algebraic tricks for Gaussian families to avoid expensive matrix inversions.

Lu: The main contribution is showing that this scheme is equivalent to natural gradient descent in the parameter space and mirror descent in the moment space, which gives us a clearer picture of its underlying optimization mechanics.

Meng: For deployment, the key thing I see is that it trades a massive matrix inversion problem for a regression problem, which is generally much easier to solve efficiently on standard hardware.

Tom: And they provide concrete complexity bounds like O(d three) and O(d), which tells us exactly how scalable this method is when we're dealing with high-dimensional data <ref:2502.18475#pg1>.

Jane: For the listener, what this means is that you can tackle much more complex probabilistic modeling problems without needing supercomputers just to get a reasonable approximation of the target distribution.

Lu: And they also show it works for discrete distributions too, which opens up possibilities for modeling things that aren't perfectly continuous.

Meng: I'm still watching how they handle the scaling of those residual variance estimates in their adaptive step size control algorithm; that’s where the real engineering work is going to be.

Tom: And they provide all the Python code via a JAX package, including algorithms for mean-field and full-covariance cases, which means we can actually start testing this immediately without having to reinvent the wheel.

Jane: It seems like a solid foundation for researchers and engineers who are working on complex generative models right now because it offers a viable alternative when the standard tools hit a wall.

Lu: This paper gives us a concrete, implementable way to tackle fixed-point recursion in exponential families without relying on explicit gradients, which is an important piece of the representation learning puzzle.

Meng: It's an interesting algorithm, and if it can actually be deployed with those computational bounds they state, then I think it has real potential for running inference on large models.

Tom: So that's our rundown of "Least squares variational inference", a method that uses regression to approximate the fixed-point equation in exponential families and offers significant computational savings for Gaussian distributions.

The paper's summary: Tom: So, this paper is about Least Squares Variational Inference, or LSVI, and essentially it’s a new way to do inference in exponential families that doesn't need gradients.

Jane: That’s right. The main takeaway from the summary is that they frame it as a gradient-free scheme using ordinary least squares regression on some tempered log-target evaluations under the variational approximation.

Lu: What this means practically is that instead of having to calculate those difficult derivatives for the log target density, you just perform successive least squares regressions.

Meng: So, if I'm running a large model and I don't have access to those gradients, I can switch my optimization strategy from backpropagation to this regression method.

Tom: Exactly. And they connect this LSVI iteration directly to biased stochastic natural gradient descent, which gives us a known path for how the parameters should update.

Jane: They also show that this regression process is mathematically equivalent to a natural gradient descent iteration on the uKL divergence in the parameter space and mirror descent in the moment space.

Lu: That equivalence is key because it shows it’s not just some random trick; it follows established optimization theory, but mapped between different spaces.

Meng: Mapping things between spaces sounds like overhead when you're trying to deploy something that needs to be fast on a production server with limited memory.

Tom: But they solve a bigger problem then. For Gaussian distributions, they manage the complexity so we don't have to invert the Fisher information matrix at all.

Jane: That inversion is huge because in high dimensions, that matrix gets enormous and computationally prohibitive for many systems.

Lu: They achieve this by using a specific reparameterization of the linear regression, which brings down the computational cost significantly.

Meng: What are those actual numbers? I need to know if O(d three) is fast enough for my setup or if I’m looking at something that will bottleneck when d gets big.

Tom: For full-covariance cases, they get a big O of d cubed, but for mean-field cases, it drops down to O of d. That difference is significant depending on the complexity of your model structure.

Jane: And beyond just speed, they show this method can be used for Bayesian variable selection even when you're dealing with discrete exponential families like Bernoulli products.

Lu: They treat those discrete problems as a regression of log target densities under a tempered variational approximation, which works where continuous reparameterization tricks fail.

Meng: So it’s not just for smooth Gaussian models; they claim it can handle things that aren't perfectly continuous, which is useful for real-world data that often has discrete features.

Tom: And the other big thing is its use when you don't have gradients at all or they are noisy. It leverages the equivalence to natural gradient descent or mirror descent for gradient-free optimization.

Jane: That means we can use this method even when calculating those derivatives is impossible, which opens up inference on likelihood-free models.

Lu: They also show they can use Monte Carlo estimates for expectations under the variational family to handle complex models, and they claim this gets you posterior approximations at a fraction of the CPU cost compared to exact MCMC methods.

Meng: A fraction of MCMC cost is a huge win for scaling up simulations; if we can beat that by using regression steps instead, that’s something I’ll take seriously for deployment.

Tom: The paper also mentions they have concrete convergence rates, showing an O of one over k dependence on the step size schedule when those assumptions hold true.

Jane: So they aren't just proposing a new thing; they've established explicit convergence rates based on standard smoothness and convexity assumptions on the uKL objective.

Lu: But there are caveats, too; their analysis relies on conditioning on the event that estimated Fisher information matrices are well-conditioned, which means we have to monitor those matrices carefully.

Meng: Monitoring those FIMs sounds like another piece of infrastructure I’ll need to build in—a way to check if the regression models are still behaving well before trusting the results.

Tom: It seems like a solid foundation for using this when you need robust, gradient-free optimization for complex probabilistic models.

Jane: And that leads us nicely into how they actually test this stuff, and what they show across different tasks.

The paper's improvements: Tom: So, we just talked about how LSVI works, and now we're looking at what the authors are suggesting to make it better for real-world use.

Jane: The paper points out a few ways they can improve things, mainly focusing on making that step size control more robust.

Lu: They suggest implementing an adaptive step size control based on residual variance to manage the convergence of both LSVI mean-field and full-covariance versions.

Meng: Adaptive step sizes sound good theoretically, but how do we know what the residual variance is going to be in production? We need a way to track that reliably at scale.

Tom: They call this Algorithm four where you control the linear regression residuals directly to ensure stability even when you don't have exact smoothness parameters.

Jane: It’s about being less rigid with your step size schedule, allowing the system to adjust as it sees how much error is left in the regression model itself.

Lu: That should give us more robust performance when we can't actually know those exact smoothness and strong convexity parameters upfront in practice.

Meng: If we can make it adapt based on the residuals, that means less wasted compute time because you aren't running overly conservative settings all the time.

Tom: They also discuss how to eliminate the need for inverting that Fisher information matrix for Gaussian families by reparameterizing the linear regression.

Jane: That’s a major win because it keeps things computationally feasible in high-dimensional settings without needing that huge matrix inversion you have to worry about.

Lu: They show this is achieved through a specific reparameterization, which results in big O of d cubed for full-covariance and big O of d for mean-field cases.

Tom: That complexity drop is what makes it viable compared to traditional methods that require inverting the Fisher information matrix directly.

Jane: And they show this scheme works for discrete problems too, specifically Bayesian variable selection on things like Bernoulli products, using the same regression idea.

Meng: So if I'm dealing with a discrete model where reparameterization tricks don't work, I can use this LSVI structure instead to get inference done.

Lu: Exactly. It treats those discrete problems as a regression of log target densities under a tempered variational approximation, which is the mechanism they rely on there.

Tom: And they also highlight that the whole LSVI setup can be used for gradient-free optimization, leveraging the equivalence to natural gradient descent or mirror descent.

Jane: That's important because it means we can use this technique even when calculating those derivatives of the log target density is impossible or too noisy.

Meng: If we can do that without gradients, that’s a big thing for models where the likelihood function is really bumpy or just doesn't have clean derivatives.

Lu: And finally, they show how to use Monte Carlo estimates for expectations under the variational family to tackle complex models like Bayesian synthetic likelihood.

Jane: So you can get posterior approximations at a fraction of the CPU cost compared to exact MCMC methods when dealing with those complicated likelihood models.

Tom: It seems like these improvements push LSVI from just a theoretical curiosity toward being a more practical tool for inference on tricky, high-dimensional data.

Conclusion: Tom: So we’re wrapping up on Least Squares Variational Inference, or LSVI, which is this gradient-free method using regression to approximate fixed-point recursion in exponential families and shows good efficiency for Gaussian distributions.

Jane: It basically boils down to doing successive least squares regressions on the log target evaluations under the variational approximation, which they show is equivalent to biased stochastic natural gradient descent.

Lu: The main implication is that you can do inference without needing explicit gradients, which opens things up for likelihood-free inference problems.

Meng: For production, I see it as a way to handle models where the target function is really bumpy or we just don't have those derivatives available in the first place.

Tom: Right. And they show that for Gaussian families, they avoid inverting the Fisher information matrix by using a specific reparameterization of the linear regression, which keeps things fast.

Jane: That’s because that inversion gets way too expensive when you're working with high-dimensional data, and this method gives us complexity bounds like big O of d cubed for full-covariance.

Lu: And they even show it works for discrete problems too, like Bayesian variable selection on Bernoulli products, by treating them as regressions of log target densities.

Meng: That’s good to hear that it handles those discrete features without needing complicated reparameterization tricks that might not work anyway.

Tom: The convergence analysis is solid, showing an O of one over k rate with linearly decreasing step sizes when the assumptions hold true for the uKL objective.

Jane: So, the practical change for us is that we get a concrete path to using this method even when we can't guarantee perfect smoothness or strong convexity parameters in our real models.

Lu: It’s a solid foundation, but they also flag that their convergence analysis depends on estimating well-conditioned Fisher information matrices.

Meng: So, the caveat is you still have to check if those FIM estimates are actually good before you trust the results of the LSVI run.

Tom: Exactly. And they provide all the Python code via a JAX package so we can actually try out these algorithms right away without having to build everything from scratch.

Jane: It’s pretty neat how they’ve put together a whole pipeline, including scripts for logistic regression and Bayesian synthetic likelihood, which matches MCMC results at less CPU cost in some cases.

Lu: The future work they mention is extending LSVI to mixtures of exponential families or even more general variational families, which could really broaden the scope.

Meng: That sounds ambitious, but if it helps us run inference on more complex model structures efficiently, I’m interested in seeing that direction explored.

Tom: So we’ve seen how this Least Squares Variational Inference method uses regression to approximate fixed-point recursion and offers efficiency gains for Gaussian models while being versatile for discrete settings.

Jane: It really gives us a concrete, implementable tool to handle inference when standard gradient methods fall short or when you can't get clean gradients.

Lu: This paper is a good step toward making variational inference more robust across different family types and more computationally practical.

Meng: For me, it’s about getting something that runs reliably on real traffic without needing massive computational resources for every single batch of data.

Lalam: I am confident this direction is sound. The equivalence to natural gradient descent confirms the underlying optimization logic is correct, and I am sure this technique will be used by many researchers looking for a reliable gradient-free path in complex AI modeling.

ENSAE · CREST · IP Paris · GIPSA-Lab, CNRS

stat.CO

Submitted: 2025-02-05

Updated: 2026-10-29

Code: https://github.com/ylefay/LSVI

Importance score: 92/100

The gist: Least squares variational inference introduces LSVI as a gradient-free scheme for fixed-point recursion in exponential families, which is equivalent to biased stochastic natural gradient descent and

Key concepts

LSVI: LSVI is a Monte Carlo-based scheme that uses ordinary least squares regression on tempered log-target evaluations within a variational approximation. It is equivalent to performing successive least squares regressions, which simplifies the fixed-point iteration process for exponential families.
Natural Gradient Descent Equivalence: The exact LSVI iteration is mathematically equivalent to performing natural gradient descent on the uKL divergence in the natural space of parameters and a mirror descent step in the moment space. This connection allows LSVI to function as a gradient-free optimization technique.
Computational Efficiency for Gaussians: For Gaussian variational families, LSVI reparameterizes linear regression to avoid inverting the Fisher information matrix. This results in O(d^3) complexity for the full-covariance case and O(d) complexity for the mean-field case, making high-dimensional inference feasible.
Bayesian Variable Selection: LSVI can be applied to discrete exponential families, like Bernoulli products, by treating the problem as a regression of log target densities under tempered variational approximations. This allows inference even when standard reparameterization tricks are not applicable.

Terminology

Fisher Information — 14× in this paper · explained in 4 episodes
Core — 12× in this paper · explained in 2 episodes
Editor — 12× in this paper · explained in 2 episodes
Least Squares Variational Inference — 11× in this paper
Papers — 10× in this paper · explained in 3 episodes
Convergence Guarantees — 7× in this paper · explained in 3 episodes
Bayesian Inference — 5× in this paper · explained in 7 episodes
Benchmark — 5× in this paper · explained in 2 episodes
Hyperparameters — 5× in this paper · explained in 2 episodes
Robustness — 4× in this paper · explained in 6 episodes
Adam — 3× in this paper · explained in 3 episodes
Computational Efficiency — 3× in this paper · explained in 2 episodes
Fixed Point — 2× in this paper · explained in 2 episodes
Generative Model — 2× in this paper · explained in 2 episodes
Generative Models — 2× in this paper · explained in 4 episodes
Institutional Review Board — 2× in this paper
Regularization — 2× in this paper · explained in 3 episodes
Sequential Monte Carlo — 2× in this paper
Variational Bayes — 2× in this paper · explained in 2 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Ensemble Learning — 1× in this paper · explained in 2 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Generator — 1× in this paper · explained in 2 episodes
Markov Chain Monte Carlo — 1× in this paper
Monotonicity — 1× in this paper · explained in 2 episodes
Objective Function — 1× in this paper · explained in 3 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Representation Learning — 1× in this paper · explained in 3 episodes
Sequential Monte Carlo (SMC) — 1× in this paper · explained in 2 episodes
Steering — 1× in this paper · explained in 2 episodes
Stochastic Gradient Descent (SGD) — 1× in this paper · explained in 5 episodes
World Model — 1× in this paper · explained in 8 episodes
computational cost — 1× in this paper · explained in 2 episodes

Summary

Least squares variational inference introduces LSVI as a gradient-free scheme for fixed-point recursion in exponential families, which is equivalent to biased stochastic natural gradient descent and yields efficient complexity for Gaussian distributions.

Core Methodology

LSVI is introduced as a Monte Carlo-based scheme that performs ordinary least squares regression on tempered log-target evaluations under the variational approximation. The exact LSVI iteration is defined by the fixed-point equation where the mapping boils down to performing successive least squares (OLS) regression. This exact scheme is equivalent to a natural gradient descent iteration on the uKL divergence in the natural space of parameters and a mirror descent step in the moment space.

Computational Efficiency for Gaussians

When dealing with Gaussian variational families, LSVI avoids inverting the Fisher information matrix by exploiting a reparametrisation of the linear regression. This tailored scheme results in a computational complexity of O(d 3) in the full-covariance case and O(d) in the mean-field case. The explicit mapping for Gaussian distributions involves recursive definitions that manage this complexity.

Convergence Guarantees

The convergence analysis establishes explicit rates for LSVI under standard smoothness and convexity assumptions on the uKL objective. The convergence rate depends on the step size schedule, yielding O(1/k) for sufficiently small and linearly decreasing step sizes. The analysis controls the bias introduced by Monte Carlo error by conditioning on the event that estimated FIMs are well-conditioned.

Practical Applications and Performance

The method is demonstrated across various tasks, including logistic regression, discrete variable selection, and Bayesian synthetic likelihood (BSL). Numerical experiments show that LSVI performs competitively with state-of-the-art methods even when gradients are unavailable. For BSL, full-covariance LSVI matches the posterior obtained via MCMC at a fraction of the CPU cost.

Limitations and Future Work

The current approach is limited to exponential families, and extending LSVI to mixtures of exponential families or more general variational families remains a promising direction for future work. Furthermore, a comprehensive theoretical analysis of LSVI’s convergence when the target distribution π is replaced by an unbiased estimator is left for future work. The study also leaves the analysis of constants involved in convergence rates for future work.

Reproducibility and Resources

The authors provide a Python package supporting GPU parallelisation via JAX to replicate the experiments, which includes all discussed algorithms and scripts for reproducing results. All experimental settings, including hyperparameters and schedules, are detailed in Section C. The paper also reports the exact runtime and memory usage for each experiment on specific hardware.

Experimental Setup

Experiments were conducted using various datasets like Pima, Sonar, Census Income, and MNIST, comparing LSVI against NGD and ADVI. The setup details the number of samples N and iterations T used for each comparison. Error bars are reported using one standard-deviation confidence intervals over 100 independent realisations. The hardware specifications and performance statistics are provided in Appendix C, specifically Table 1.

Code and Data Availability

The paper provides a Python (JAX) package that includes all discussed Algorithms (LSVI Algorithm 1, MF-LSVI Algorithm 2, FC-LSVI Algorithm 3, Variance control for the stepsizes Algorithm 4, NGD with details provided in Section C) as well as scripts to reproduce all the listed experiments. The package is explicitly divided into two parts, variational contains the generic implementations while experiments contains three sub-folders for the three distinct variational problems (logistic regression, variable selection and Bayesian synthetic likelihood). All used Python packages are open-source and have permissive licenses, with details provided in Section 1 and Section 5. The datasets used, licenses and download links are provided in Appendix C. The authors confirm that they have not conducted any experiments involving human subjects. No LLMs were used in the manuscript.

Ethics and Licensing

The research conforms to the NeurIPS Code of Ethics, and all used Python packages are open-source with permissive licenses explicitly mentioned. The paper provides information on how to reproduce results, including the exact command and environment needed to run to reproduce the results. The authors have not introduced any new assets, but they provide a well-documented Python (JAX) package for LSVI. All used assets are properly credited and licensed. The authors have not released any new assets, but they provide a well-documented Python (JAX) package for LSVI. The authors confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics. No risk was seen in the application of variational inference procedures. The paper poses no such risks. There is no mention of LLMs in the manuscript, and no LLM was used. No experiment involving human subjects were conducted. The paper does not discuss both potential positive societal impacts and negative societal impacts of the work performed. The authors have not put any safeguards in place for responsible release of data or models that have a high risk for misuse. The creators or original owners of assets are properly credited and are the license and terms of use explicitly mentioned and properly respected. All used Python packages are open-source, have permissive licenses, and are explicitly mentioned both in the manuscript and the code (pyproject.toml with complete dependency specifications). The authors have not introduced any new assets. No experiment involving human subjects were conducted. The paper does not involve crowdsourcing nor research with human subjects.

Theory Assumptions and Proofs

All assumptions are provided in the core manuscript, see Theorem 2.1, Theorem 3.1, Theorem 3.2 and Theorem 3.3. The proofs are deferred to the supplementary materials and are divided in several comprehensive steps, including lemmas in order to make the proofreading procedure easier.

Experimental Result Reproducibility

Experimental Setting/Details

The paper provides all the necessary details to reproduce the experiments, including the hyperparameters (the number of samples N, the number of iterations T, the initialisation distributions and the schedules) which are given in Section C. The full hardware and software specifications are provided in Appendix C, specifically in Table 1.

Experimental Statistical Significance

All experiments were conducted using multiple trials as indicated in the figure labels and the Appendix C. For the logistic regression examples, one standard-deviation confidence intervals are provided over 100 independent realisations. No statistical assumption is made for uncertainty measurement. The factors of variability that the error bars are capturing should be clearly stated. The method for calculating the error bars should be explained.

Experimental Compute Resources

The full hardware and software specifications are provided in Appendix C, specifically in Table 1. All performance statistics are computed using independent realisations for improved robustness.

Code of Ethics

The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics. The authors have carefully read through the NeurIPS Code of Ethics, and we see no violation of any guideline.

Broader Impacts

The paper focuses on VI methods, emphasizing theoretical analysis and algorithmic implementability. Given its abstract and theoretical scope, it does not present identifiable positive or negative societal impacts. The work is foundational in nature and does not directly pertain to real-world applications or deployments.

Safeguards

We see no risk in the application of variational inference procedures. The paper poses no such risks. No experiment involving human subjects were conducted.

Licenses for Existing Assets

All used Python packages are open-source, have permissive licenses, and are explicitly mentioned both in the manuscript and the code (pyproject.toml with complete dependency specifications). The authors have not introduced any new assets.

New Assets

The provided Python (JAX) package for LSVI is well documented and includes a README file with instructions for installation and usage. The license is also included in the package (Apache License 2.0). The authors have not introduced any new assets.

Crowdsourcing and Research with Human Subjects

No experiment involving human subjects were conducted. No experiment involving crowdsourcing nor research with human subjects were conducted.

Declaration of LLM Usage

There is no mention of LLMs in the manuscript, and no LLM was used. The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Improvements for AI systems

Adaptive step size control based on residual variance can be implemented to ensure convergence of LSVI-MF and LSVI-FC by controlling the linear regression residuals as described in Algorithm 4. This allows for robust performance even when exact smoothness and strong convexity parameters are unknown in practice.
Elimination of Fisher information matrix inversion is achieved for Gaussian families by reparameterizing the linear regression, yielding O(d 3) complexity for full-covariance and O(d) for mean-field cases. This makes inference computationally feasible in high-dimensional settings where exact FIM inversion is prohibitive.
The LSVI scheme can be used to perform Bayesian variable selection on discrete exponential families, such as Bernoulli products, by treating the problem as a regression of log target densities under tempered variational approximations. This enables inference where reparameterization tricks are impossible for discrete distributions.
The system can leverage the equivalence between LSVI and natural gradient descent or mirror descent to implement gradient-free optimization without requiring explicit gradients of the log target density, which is useful when gradients are unavailable or noisy.
For Bayesian synthetic likelihood inference, LSVI can be applied to complex models by utilizing Monte Carlo estimates for expectations under the variational family, achieving posterior approximations at a fraction of the CPU cost compared to exact MCMC methods.

Sources

Importance sampling-based gradient method for dimension reduction in Poisson log-normal model