Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models

arXiv:2606.24357 · stat.ME, math.ST, stat.CO, stat.TH · Submitted 2026-06-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models".

Jane: Here's the gist of the paper. Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models are presented, proposing a fully Bayesian hierarchical and conjugate GLM framework applicable to any distribution in the exponential family.

Tom: This approach models a binary inclusion indicator (z) that directly encodes covariate inclusion within the linear predictor (eta = X(beta z)), simultaneously performing variable selection and parameter estimation.

Jane: The framework utilizes conjugate priors consistent with the exponential family structure, leading to closed-form posterior distributions characterized by a Gibbs sampling scheme for inference.

Tom: Theoretical guarantees establish strong posterior consistency: Theorem 1 confirms that the posterior probability concentrates on the true model (P(z = z* X, y, tau) to 1 as n to infinity), and Theorem 2 ensures that the posterior distribution of active regression coefficients (beta(1)) concentrates around the true active coefficient vector beta*(1).

Paper summary: Lu: The authors are Lucia Filippozzi, Iñigo Urteaga, and Claudio Agostinelli from the Department of Mathematics at the University of Trento Lu. They’ve built something that sits in a specific sweet spot between traditional Bayesian variable selection methods and more flexible shrinkage priors Lu.

Tom: The title itself is really telling—"Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models"—it immediately tells you the three main components: conjugate priors, Bayesian variable selection, and generalized linear models.

Jane: When we put it together, what this paper means is that they’ve found a way to make the complicated task of selecting which predictors matter in a model mathematically rigorous and computationally tractable for any distribution in that family Jane. It moves beyond just fitting the data to actually understanding which inputs drive the output.

Meng: I see it as giving us a more reliable tool when we are building models for things like risk assessment or predictive maintenance where knowing *why* a prediction is high or low matters more than just getting the final number right Meng. It’s about interpretability built directly into the selection process.

Lalam: The implication here is that for any statistical model that fits an exponential family, we now have a framework that promises to find the correct set of predictors with high confidence, and it does so through a very structured mathematical approach Lalam. It’s about making sure you don't pick irrelevant stuff just because it looked good in the initial fit.

Tom: So what this paper means for the world is that we can start applying this kind of rigorous variable selection to a much broader range of real-world data, not just textbook examples Tom. It’s about moving from a process where you guess which features are important to one where the math guides you toward the correct set.

Jane: Exactly. It’s less about finding some clever trick and more about building a sturdy mathematical structure that guarantees you get there, provided your data fits that exponential family Jane. That's what this work is all about: providing consistency for variable selection in this specific but very important class of models.

Conclusion: Tom: So, we’re wrapping up on this paper, "Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models." Basically, they’ve taken a general class of models—the exponential family—and built a whole framework that lets you select which inputs matter while estimating the parameters at the same time.

Jane: That sounds intense. So, if I’m driving and I want to know which features in my data are actually useful for predicting something, this approach helps me figure out that without having to guess everything randomly?

Lu: Exactly. The core idea is using priors that fit the structure of the exponential family itself. This leads to these closed-form posterior distributions, which means we can get a solid answer, not just a rough estimate.

Meng: But what about practical use? If I’m running this at scale on real traffic data, does it actually keep up? Does it crash when the number of variables gets huge?

Tom: The paper shows strong consistency guarantees. They prove that as your sample size grows, the method reliably picks the right set of variables and estimates those coefficients correctly. It’s mathematically consistent.

Jane: Consistent is good, but what does that mean for a person just trying to build a predictive model? Does it make the whole process simpler or just more complicated?

Lalam: For me, it means we're moving away from guesswork on which predictors to keep. We’re using math that respects the underlying data structure. It’s about building confidence in those selections because the math forces them to be sensible within that family of models.

Lu: The theoretical work is pretty thorough, covering different levels of model complexity. They show it holds up across various scenarios where you might have a lot of variables or maybe some complicated relationships between them.

Tom: It seems like they’ve really nailed the mechanics on making variable selection and parameter estimation happen together smoothly for any distribution in that exponential family structure.

Jane: So, if this framework works well for Normal, Poisson, or Binomial models, what does that tell us about the limits of what we can do with standard statistical modeling tools?

Lalam: It tells us that for these specific model types, we have a very reliable way to find those active coefficients. It’s a big step toward making selection less of an art and more of an applied science.

Lucia Filippozzi, §nigo Urteaga, § Claudio Agostinelli

University of Trento · Fondazione Bruno Kessler (FBK) · Basque Center for Applied Mathematics (BCAM) · Basque Foundation for Science (Ikerbasque)

stat.ME, math.ST, stat.CO, stat.TH

Submitted: 2026-06-23

Updated: 2026-11-04

Comments: New introduction, reorganization of numerical results, and minor changes to the Proof of Theorem 1 (Section B - Supplementary material). 27 pages (without Supplementary Material)

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models are presented…

Key concepts

Exponential Family
A class of probability distributions that share a specific functional form. This structure is crucial because the proposed conjugate priors are designed to work naturally with this mathematical family of models, simplifying the calculation of posterior distributions.
Variable Selection ($z$)
This involves determining which covariates in the model are actually important or 'active.' The paper uses a binary indicator variable ($z$) to directly encode whether a specific covariate is included in the linear predictor, effectively performing selection during parameter estimation.
Conjugate Priors
These are special prior probability distributions that have a mathematical relationship with the likelihood function of the model. Using conjugate priors simplifies inference by allowing for closed-form posterior distributions, which can be solved efficiently using Gibbs sampling.

Terminology

Summary

Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models are presented, proposing a fully Bayesian hierarchical and conjugate GLM framework applicable to any distribution in the exponential family. p. 0 This approach models a binary inclusion indicator (z) that directly encodes covariate inclusion within the linear predictor (eta = X(beta z)), simultaneously performing variable selection and parameter estimation. p. 0 The framework utilizes conjugate priors consistent with the exponential family structure, leading to closed-form posterior distributions characterized by a Gibbs sampling scheme for inference. p. 4

Theoretical guarantees establish strong posterior consistency: Theorem 1 confirms that the posterior probability concentrates on the true model (P(z = z* X, y, tau) to 1 as n to infinity), and Theorem 2 ensures that the posterior distribution of active regression coefficients (beta(1)) concentrates around the true active coefficient vector beta*(1). p. 7 Further theoretical results (Theorems 13, 14, and 15) establish consistency across different model complexity regimes (M 1, M 2, M 3) under specified conditions. p. 7

Empirical validation across various GLM families (Normal, Poisson, Binomial) demonstrates competitive performance. p. 0 In the Poisson setting (Setting C), the method achieves high accuracy in recovering sparsity patterns and estimating active coefficients comparable to oracle models. p. 1 For the Linear Model (Setting A and B), results confirm reliable feature identification even with noisy or correlated covariates. p. 4 The Relative Mean Squared Error (RelMSE) for estimated active coefficients decreases steadily as sample size (n) increases, reflecting asymptotic consistency. p. 6 In the Logistic Model, similar high accuracy is observed across settings, and estimation error metrics also decrease with increasing n. p. 1

The effect of covariate correlation (gamma) was assessed by varying it in controlled experiments. p. 3 In both Poisson and Linear models, selection accuracy decreases as correlation increases because the distinction between informative and redundant covariates becomes less clear in posterior inclusion probabilities. Overall, BayesVS-GLM achieves very high global accuracy in covariate selection across diverse settings, including noisy or correlated covariates, performing well on real-world datasets (Table 2). p. 2

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Bayesian Variable Selection in Generalized Linear Models can perform simultaneous variable selection and parameter estimation by modeling a binary inclusion indicator that directly encodes covariate inclusion in the linear predictor, providing a valid post-model Bayesian selection procedure.

  2. The system can establish posterior consistency for both the inclusion indicators and active regression coefficients, which is a key theoretical guarantee for reliable variable selection methods.

  3. It can derive an efficient Gibbs Sampling algorithm with a corresponding R package implementation to perform posterior inference efficiently using fully conjugate Bayesian hierarchical models.

  4. The system supports inference on any distribution in the exponential family by exploiting the conjugate prior structure of Generalized Linear Models, unifying various selection perspectives into a single framework.

  5. For linear models, it can derive Normal posterior distributions for active coefficients and Inverse-Gamma posterior distributions for scale parameters like tau when conjugacy is ensured.

  6. It can approximate intractable posterior densities using Laplace approximation and draw samples via Laplace within Sampling Importance Resampling (SIR) to facilitate inference in non-Gaussian settings.

  7. The system demonstrates competitive predictive performance on synthetic and real-world datasets, showing successful covariate selection and estimation across different GLM instances like Poisson, Logistic, and Linear models.

Sources

Related papers