Scalable and Calibrated Sampling for Bayesian Generalized Linear Mixed Model via Stochastic Gradient Markov Chain Monte Carlo

arXiv:2403.03007 · stat.CO, stat.ME, stat.ML · Submitted 2024-03-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Scalable and Calibrated Sampling for Bayesian Generalized Linear Mixed Model via Stochastic Gradient Markov Chain Monte Carlo".

Jane: Here's the gist of the paper. Generalized linear mixed models (GLMMs) present a unique computational bottleneck: conditional methods are tractable but not scalable, and marginal methods are scalable but intractable.

Tom: This paper introduces a stochastic gradient Markov chain Monte Carlo (SGMCMC) algorithm tailored to GLMMs that enables accurate posterior inference in the large-sample regime.

Jane: The Core Methodology The approach develops a stochastic gradient MCMC (SGMCMC) algorithm specifically for GLMMs by using Fisher’s identity to construct a Monte Carlo estimator of the gradient of the marginal log-likelihood.

Tom: This allows SGLD to be feasible when direct gradient computation is impossible, and it uses an asymptotic correction to adjust for variability introduced by both data subsampling and gradient approximation.

Paper summary: Tom: So we're looking at this paper, "Scalable and Calibrated Sampling for Bayesian Generalized Linear Mixed Model via Stochastic Gradient Markov Chain Monte Carlo." The main problem they're tackling is that generalized linear mixed models, or GLMMs, are great for correlated data in medicine and social science because they let you account for within-group correlation with subject-specific random effects. But the challenge is that calculating the marginal likelihood—which is needed for Bayesian inference—is just intractable, which makes standard Markov chain Monte Carlo methods computationally too expensive and slow when you have big datasets.

Jane: Right, so the core claim of this paper is they developed a new algorithm called stochastic gradient Markov chain Monte Carlo, or SGMCMC, that's specifically tailored for GLMMs to solve this scalability issue. What they’re claiming is that this method allows for accurate posterior inference even in the large-sample regime where conventional methods just choke on the computational cost.

Lu: The authors are introducing a stochastic gradient MCMC algorithm using Fisher’s identity to construct an estimator of the gradient of the marginal log-likelihood, which lets them make SGLD feasible when you can't compute that direct gradient easily. They use an asymptotic correction to adjust for variability from both subsampling and this gradient approximation, which is pretty clever because it tries to fix the errors from both sides at once.

Meng: It sounds like they are trying to take a scalable framework, SGLD, and make it work reliably on GLMMs by handling the noise in a very specific way. But I'm worried about how that correction works when you move from simple models to complex ones with many random effects or non-Gaussian structures. If the correction itself introduces new errors that grow with the sample size, we might just be trading one kind of computational problem for another.

Lalam: The paper focuses on estimating the marginal log-likelihood gradient by decomposing it into an expectation using samples from a conditional distribution of a random effect called gamma i. They define an estimator, g hat i, and then they establish that this population estimator is unbiased with a covariance related to the sample size minus one times the covariance of that Monte Carlo estimator, which is denoted as psi i.

Tom: So they’re building this up from estimating those gradients first by using R samples of gamma i drawn from the conditional distribution. And once they have that, they move into a variance correction mechanism based on approximating the discrete-time update in the SGLD algorithm with a forward Euler scheme.

Paper summary: Jane: That forward Euler approximation introduces an approximation noise term, which they isolate into something called gamma i, which involves sigma squared times psi i over two times S plus identity. This leads to an invariant measure pi bar and an estimate of the covariance that you want by solving a Lyapunov equation involving a matrix A and the noise term gamma i.

Lu: The way they solve that Lyapunov equation gives them an estimate of the inverse covariance A minus one by relating it to sigma squared times psi i over two S plus identity, which is how they get that corrected sample set theta k, defined as G applied to the uncorrected and corrected terms plus the mean.

Meng: Okay, so they are using this derived covariance structure to compute new samples theta k after correcting the original ones. But I have to ask, what happens if that correction term doesn't perfectly capture all the variability introduced by those initial gradient approximations? That seems like a big assumption when you're dealing with complex GLMM structures.

Lalam: The empirical validation showed that this corrected SGLD algorithm properly estimates the posterior mean and variance across Gaussian, logistic, and Poisson data. Specifically, in the logistic GLMM simulation with known variance components, the uncorrected SGLD consistently overestimated the posterior variance compared to Gibbs sampler results for a sample size of ten thousand.

Tom: That's a key result because it shows that the corrected version yields estimates that are practically indistinguishable from Gibbs sampler results when you have big data, and importantly, its performance doesn't degrade as you increase the sample size, unlike some other methods we know which get overloaded.

Jane: And then they took this technique into a real-world application using an electronic health records dataset from the Duke Ophthalmic Registry to find patient characteristics related to psychiatric distress. They found that the variance correction meaningfully impacted their inference because ninety-five percent credible intervals shifted in interpretation for key things like insurance status and ethnicity.

Lu: That practical impact is significant because it shows that accounting for variance inflation actually changes how you read the results, which is something you don't always see in theory simulations alone. It highlights why these corrections are necessary when the data has those tricky structures inherent in real-world settings.

Paper summary: Meng: So, what does this mean for deployment? If this method works well across different GLMM types—Gaussian, logistic, Poisson—it suggests a more robust way to handle uncertainty quantification on large datasets without having to switch entirely to much slower methods like exact integration or complex variational inference.

Lalam: One limitation the authors noted is that the accuracy of their proposed method gets significantly worse for exponential link Poisson GLMMs compared to when they test it on logistic GLMMs. They also suggested that changing the link function might be necessary for Poisson models because they don't meet some regularity conditions there.

Tom: So, while it’s working well generally, you can’t just throw this at every single Poisson model without checking those specific conditions first because it breaks down a bit on those exponential links.

Jane: This paper, "Scalable and Calibrated Sampling for Bayesian Generalized Linear Mixed Model via Stochastic Gradient Markov Chain Monte Carlo," is essentially proposing a way to make scalable inference work for GLMMs by using a stochastic gradient MCMC algorithm combined with a specific post-hoc variance correction. It shows that even when the marginal likelihood is intractable, you can get accurate posterior samples if you handle the noise correctly.

Lu: The core idea rests on using Fisher’s identity to estimate the gradient of the marginal log-likelihood and then applying an extended Kalman filter-like linearization technique to correct for approximation noise in the SGLD process. This provides an asymptotic analysis that yields a corrected sample set based on solving a Lyapunov equation.

Meng: From an engineering standpoint, this is attractive because it doesn't rely on expensive direct gradient computations, which is what usually breaks things at scale in these models. The fact that they show performance doesn't degrade with increasing sample size is a big deal for production systems where you want predictable runtime costs.

Lalam: Ultimately, the paper demonstrates that this method offers a viable technique for scalable Bayesian inference in GLMMs by combining SGLD with the derived variance correction, achieving accurate uncertainty quantification across various GLMM settings.

Tom: So to sum up, this paper shows that you can make scalable inference work for GLMMs without getting completely bogged down by the computational cost of the marginal likelihood integral. It’s a promising alternative when sample sizes get large and you need reliable variance estimates.

Conclusion: Tom: So we're wrapping up on this paper about scalable sampling for generalized linear mixed models by stochastic gradient Markov chain monte carlo, or SGMCMC.

Jane: It basically says they found a way to get accurate results for these big models even when the math gets too complicated for standard methods.

Lu: The main thing is using Fisher's identity to estimate the gradient of the marginal log-likelihood and then applying an asymptotic correction to fix the errors from approximation noise.

Meng: So they’ve got this algorithm that doesn't need you to compute those super expensive direct gradients, which is huge for production systems.

Lalam: The researchers showed that this corrected method properly estimates the posterior mean and variance across different data types like Gaussian and logistic models.

Tom: And they used it on a real health records dataset from Duke Ophthalmic Registry, where the correction meaningfully changed how you read the results for things like insurance status.

Jane: It really shows that properly accounting for variance inflation actually changes how you interpret your findings in complex settings.

Lu: The authors highlight that this approach is robust to increases in sample size, unlike some other methods we know which just break under the computational load.

Meng: But they did mention a limitation—this correction gets much less accurate when you're dealing with exponential link Poisson models.

Tom: So it’s a viable tool for large-scale Bayesian inference in GLMMs, but you still gotta watch out for those specific model types.

Jane: It means we don't have to give up on these complex models just because the marginal likelihood is intractable anymore.

Lu: The paper sets the stage for looking into adaptive step-size discretizations and expectation propagation in the context of generalized linear mixed models.

Duke University · Institute for Mathematical Statistics and Actuarial Sciences, University of Bern · University of Miami

stat.CO, stat.ME, stat.ML

Submitted: 2024-03-05

Updated: 2026-11-10

Comments: 33 pages, 5 figures, 2 tables. To be published in Statistics and Computing

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: Generalized linear mixed models (GLMMs) present a unique computational bottleneck: conditional methods are tractable but not scalable, and marginal methods are scalable but intractable.

Key concepts

SGMCMC
This is a specialized MCMC algorithm designed for GLMMs. It combines Stochastic Gradient Langevin Dynamics (SGLD) with techniques to estimate gradients using Monte Carlo methods based on Fisher's identity. This allows the model to explore the posterior distribution efficiently, even when direct gradient calculation is too difficult.
Fisher's Identity
This mathematical tool is used to create an estimator for the gradient of the marginal log-likelihood. Since calculating this gradient directly is often impossible in complex models like GLMMs, Fisher's identity provides a tractable way to approximate it using random samples.
Variance Correction Mechanism
This post-hoc adjustment step corrects errors introduced by approximating the continuous SGLD update with a simpler scheme (forward Euler). It isolates noise from data subsampling and gradient estimation, solving a Lyapunov equation to derive an accurate estimate of the posterior covariance.

Terminology

Summary

Generalized linear mixed models (GLMMs) present a unique computational bottleneck: conditional methods are tractable but not scalable, and marginal methods are scalable but intractable. p. 1

Where the paper says it

Marginal likelihood-based approaches, including variational inference and stochastic gradient MCMC (SGMCMC), are scalable, but require evaluating or differentiating the marginal likelihood, which is intractable in GLMMs. Thus, GLMMs present a unique computational bottleneck: conditional methods are tractable but not scalable, and marginal methods are scalable but intractable.

This paper introduces a stochastic gradient Markov chain Monte Carlo (SGMCMC) algorithm tailored to GLMMs that enables accurate posterior inference in the large-sample regime. p. 0
Where the paper says it

Scalable Bayesian inference for GLMMs is challenging due to an intractable marginal likelihood and a high computational cost incurred by conventional Markov chain Monte Carlo (MCMC) methods. We develop a stochastic gradient MCMC (SGMCMC) algorithm tailored to GLMMs that enables accurate posterior inference in the large-sample regime.

The Core Methodology

The approach develops a stochastic gradient MCMC (SGMCMC) algorithm specifically for GLMMs by using Fisher’s identity to construct a Monte Carlo estimator of the gradient of the marginal log-likelihood. This allows SGLD to be feasible when direct gradient computation is impossible, and it uses an asymptotic correction to adjust for variability introduced by both data subsampling and gradient approximation. p. 0

Where the paper says it

Our approach uses Fisher’s identity to construct a (biased) Monte Carlo estimator of the gradient of the marginal log-likelihood, making SGMCMC feasible when direct gradient computation is impossible. We analyze the additional variability, introduced by both data subsampling and gradient approximation, to derive a post-hoc covariance correction that yields properly calibrated posterior uncertainty.

The algorithm is based on the Stochastic Gradient Langevin Dynamics (SGLD) framework which approximates the full-data gradient with a minibatch average of subject-specific gradient estimators. p. 6
Where the paper says it

algorithm for GLMMs: ∇ ˆfGLMM (Ω; S) = ∇f0 (Ω) + n S X i∈S gˆi (Ω) = ∇f0 (Ω) + nhˆ S (Ω). (7) Our estimator is a function of a “population estimator” hˆ S (Ω) = 1 S P i∈S gˆi(Ω), which is a minibatch average of the subject-specific gradient estimators, gˆi (Ω).

Estimating the Marginal Likelihood Gradient

The paper defines an estimator for the marginal log-likelihood gradient, denoted as gi(omega), by decomposing it into an expectation with respect to the conditional distribution of a random effect γi. A Monte Carlo estimator gˆi(omega) is defined based on R samples of γi drawn from p(γiYi, omega). p. 6

Where the paper says it

The representation of the gradient of the marginal log-likelihood as an expectation in (8) motivates a Monte Carlo estimator for gi(Ω), as in [25]. We define our Monte Carlo estimator based on R samples of γi as, gˆi(Ω) = − 1 R X R r=1 ∇ log p(Yi , γir|Ω), γir iid∼ p(γi |Yi , Ω).

The covariance of this Monte Carlo estimator is defined as Ψi(omega), and an unbiased empirical covariance Ψˆi(omega) is also introduced. p. 7
Where the paper says it

The covariance of the Monte Carlo estimator (9) is defined as Ψi(Ω) = Vγi |Yi,Ω (gˆi(Ω)) = 1 R Vγi |Yi,Ω (∇ log p(Yi , γi |Ω)) , (10) where VX|Y(f(X)) is the conditional covariance of f(X) given Y. The associated empirical covariance is given by Ψˆ i(Ω) = 1 R(R − 1) X R r=1 (∇ log p(Yi , γir|Ω) − gˆi(Ω)) (∇ log p(Yi , γir|Ω) − gˆi(Ω)) ⊤ .

Lemma 3.1 establishes that the population estimator hˆ S (omega) is an unbiased estimator of hn with covariance S−1Ψ(omega).

Variance Correction Mechanism

A post-hoc correction is derived by approximating the discrete-time update in the SGLD algorithm with a forward Euler scheme, which isolates the approximation noise into a term involving Γ(omega) = ϵn2Ψ(omega)/(2S) + Id. This leads to an invariant measure π¯(omega) and an estimate of the desired covariance A−1 by solving the Lyapunov equation AΣ + ΣA⊤ = 2Γ. p. 8

Where the paper says it

Since both Σ and A are symmetric, (15) can be written as ΣA + AΣ = 2Γ. Solving this equation for A, where Σ,Γ are estimated from SGLD samples, yields an estimate of the desired covariance.

The corrected samples Θk are then computed as Θk = G (omegak − omega∗) + omega∗, where G is defined by the Cholesky decomposition of the uncorrected and corrected covariance. p. 9
Where the paper says it

We then define the Cholesky decomposition for the uncorrected and corrected covariance as, Σ = E⊤E and A = F ⊤F. The corrected samples can be computed as follows, Θk = G (Ωk − Ω ∗ ) + Ω ∗ , G = E ⊤F −1 .

Empirical Validation and Performance

Simulations across Gaussian, logistic (binary outcome), and Poisson (count outcome) data demonstrate that the corrected SGLD algorithm properly estimates posterior mean and variance, outperforming existing methods like control variate SGLD. In the logistic GLMM simulation with known variance components, the uncorrected SGLD consistently overestimated the posterior variance, while the corrected version yielded estimates indistinguishable from Gibbs sampler results for n = 10 4. p. 2

Where the paper says it

Our method essentially adapts an extended Kalman filter-like linearization technique frequently used in applied SDE modeling literature [15]. We demonstrate through simulated studies that our method yields an accurate approximation of the posterior variance, while a naive use of control variates method in GLMMs can lead to both inflated and deflated variance estimates.

The performance of the corrected SGLD was not impacted by increases in sample size, unlike Gibbs or NUTS methods which suffered from computational overload. p. 0
Where the paper says it

Scalable Bayesian inference for GLMMs is challenging due to an intractable marginal likelihood and a high computational cost incurred by conventional Markov chain Monte Carlo (MCMC) methods. We develop a stochastic gradient MCMC (SGMCMC) algorithm tailored to GLMMs that enables accurate posterior inference in the large-sample regime.

Real-World Application

The algorithm was applied to a real-world electronic health records dataset from the Duke Ophthalmic Registry to identify patient characteristics associated with psychiatric distress. The covariance correction meaningfully impacted inference, as 95% credible intervals changed interpretation for key covariates such as insurance status and ethnicity. This underscores the importance of properly accounting for variance inflation in complex settings. p. 1

Where the paper says it

and variances in settings with a large number of groups, outperforming existing approaches, including control variate methods. We further demonstrate the method’s practical utility in an analysis of electronic health records data, where accounting for variance inflation materially changes scientific conclusions.

Limitations and Future Directions

A limitation noted is that the accuracy of the proposed method is much worse for exponential link Poisson GLMM than for logistic GLMM. Future research directions include investigating adaptive step-size discretizations to overcome smoothness assumptions and exploring expectation propagation in the context of GLMMS. The authors also suggest that changing the link function might be necessary for Poisson models due to violations of regularity conditions. p. 5

Where the paper says it

(5) The integrand function corresponding to the likelihood p(Yi , γi |Ω) is referred to as the “complete likelihood. ” The integral arising in GLMMs for binary/count data is analytically intractable, due to the non-Gaussian structure of the random effects or the nonlinearity of the link function.

Discussion Summary

The main contributions are a Monte Carlo approximation of the gradient using Fisher’s identity and an asymptotic analysis yielding a post-hoc correction to restore the correct posterior covariance. The correction accounts for both sources of variability, unlike SGLD-CV which fails to account for likelihood approximation noise. p. 2

Where the paper says it

Our contributions are threefold: (i) a Monte Carlo gradient estimator based on Fisher’s identity that enables SGMCMC in the presence of intractable marginal likelihoods; (ii) an asymptotic correction that adjusts for variance introduced by both minibatching and gradient approximation; and (iii) empirical results demonstrating accurate uncertainty quantification and substantial speedups over MCMC with the full dataset. Comparison of our method with the control variate SGLD designed for independent data show that failing to account for the additional noise from likelihood approximation leads to biased uncertainty estimates, underscoring the need for methods built specifically for GLMMs.

This method shows the utility of tailoring variance correction to specific dependent-data models. p. 1
Where the paper says it

We further demonstrate the method’s practical utility in an analysis of electronic health records data, where accounting for variance inflation materially changes scientific conclusions. Keywords: Scalable inference, Mixed models, Big data, Uncertainty quantification MSC Classification: 62F15 , 62R07 , 62J05 1 Introduction Generalized linear mixed models (GLMMs) are widely used to model dependent data with clustered or repeated measures, particularly in biomedical and social sciences where such structures are common [1, 2].

Conclusion

The proposed method is a viable technique for scalable Bayesian inference in GLMMs that maintains computational efficiency as n increases. It achieves accurate uncertainty quantification across various GLMM settings through the combination of SGLD and the derived variance correction. p. 2

Where the paper says it

In this paper, we introduce the first SGLD-based algorithm tailored for scalable Bayesian inference in GLMMs. Our contributions are threefold: (i) a Monte Carlo gradient estimator based on Fisher’s identity that enables SGMCMC in the presence of intractable marginal likelihoods; (ii) an asymptotic correction that adjusts for variance introduced by both minibatching and gradient approximation; and (iii) empirical results demonstrating accurate uncertainty quantification and substantial speedups over MCMC with the full dataset.

The results are robust to the choice of minibatch size S or sample size n. p. 7
Where the paper says it

Proof See Section 1 of the online supplementary materials [32]. □ The covariance of hˆ S (Ω) is given by S −1Ψ(Ω) and becomes smaller with increases in minibatch size (S → n).

This method is a promising alternative to uncorrected SGLD when the sample size becomes large. p. 0
Where the paper says it

Scalable Bayesian inference for GLMMs is challenging due to an intractable marginal likelihood and a high computational cost incurred by conventional Markov chain Monte Carlo (MCMC) methods. We develop a stochastic gradient MCMC (SGMCMC) algorithm tailored to GLMMs that enables accurate posterior inference in the large-sample regime.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop a stochastic gradient Markov chain Monte Carlo algorithm tailored specifically for Generalized Linear Mixed Models that enables accurate posterior inference in large sample regimes. This system can perform scalable Bayesian inference for GLMMs by utilizing Fisher's identity to construct a Monte Carlo estimator of the marginal log-likelihood gradient.

  2. Implement an asymptotic correction for the posterior samples obtained from the SGLD algorithm to adjust for variability introduced by both data subsampling and gradient approximation. This allows the AI system to derive properly calibrated posterior uncertainty that outperforms existing control variate methods in settings with a large number of groups.

  3. Create a scalable inference framework capable of analyzing electronic health records data, where accounting for variance inflation materially changes scientific conclusions. This improved system will provide accurate posterior means and variances in complex datasets by yielding results comparable to Gibbs sampling when the sample size is large.

  4. Design an algorithm that effectively handles intractable integrals appearing in GLMMs by using Monte Carlo estimators based on Fisher's identity, allowing SGLD updates for models with intractable marginal gradients. This capability expands the scope of scalable Bayesian inference to a wider class of non-Gaussian GLMMs.

  5. Implement a double loop algorithm that utilizes an ergodic average with respect to S Markov chains to compute stochastic gradients, addressing the issue where uncorrected SGLD will tend to have very large errors in approximating the targeted posterior when using biased stochastic gradients. This allows for more robust sampling even when standard unbiased estimators are not feasible.

  6. Integrate a Lyapunov equation solver into the algorithm to derive a post-hoc correction for posterior samples, allowing the system to solve a general Lyapunov equation of form (15) with d x d matrices and obtain corrected samples Theta k that have asymptotically, the proper mean and covariance.

  7. Enhance data handling for count outcome data by adapting the method to Poisson GLMMs, mitigating the issue where SGLD-CV may over-correct by estimating a marginal posterior variance much lower than NUTS. This results in improved accuracy for posterior variance estimates in count models compared to logistic GLMMs.

Abstract

Generalized linear mixed models (GLMMs) are widely used for analyzing correlated data, particularly in large-scale biomedical and social science applications. Scalable Bayesian inference for GLMMs is challenging due to an intractable marginal likelihood and a high computational cost incurred by conventional Markov chain Monte Carlo (MCMC) methods. We develop a stochastic gradient MCMC (SGMCMC) algorithm tailored to GLMMs that enables accurate posterior inference in the large-sample regime. Our approach uses Fisher's identity to construct a (biased) Monte Carlo estimator of the gradient of the marginal log-likelihood, making SGMCMC feasible when direct gradient computation is impossible. We analyze the additional variability, introduced by both data subsampling and gradient approximation, to derive a post-hoc covariance correction that yields properly calibrated posterior uncertainty. We show through simulated studies that the proposed method provides accurate posterior means and variances in settings with a large number of groups, outperforming existing approaches, including control variate methods. We further demonstrate the method's practical utility in an analysis of electronic health records data, where accounting for variance inflation materially changes scientific conclusions.

Sources

Related papers