Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models
summary
The gist
Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models are presented…
In short
This work proposes a Bayesian framework for variable selection in Generalized Linear Models using conjugate priors tailored to exponential family distributions. It models covariate inclusion directly within the linear predictor, allowing simultaneous variable selection and parameter estimation. The method provides strong theoretical guarantees for posterior consistency and achieves high accuracy empirically across Normal, Poisson, and Binomial GLM settings.
Key concepts
- Exponential Family
- A class of probability distributions that share a specific functional form. This structure is crucial because the proposed conjugate priors are designed to work naturally with this mathematical family of models, simplifying the calculation of posterior distributions.
- Variable Selection ($z$)
- This involves determining which covariates in the model are actually important or 'active.' The paper uses a binary indicator variable ($z$) to directly encode whether a specific covariate is included in the linear predictor, effectively performing selection during parameter estimation.
- Conjugate Priors
- These are special prior probability distributions that have a mathematical relationship with the likelihood function of the model. Using conjugate priors simplifies inference by allowing for closed-form posterior distributions, which can be solved efficiently using Gibbs sampling.
Terminology used across episodes
This episode discusses
- Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models · Paper Radio
- High-Dimensional Bayesian Regularised Regression with the BayesReg Package
- Centralized and Federated Heart Disease Classification Models Using UCI Dataset and their Shapley-value Based Interpretability
The paper
Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models · Read on arXiv
Lucia Filippozzi, §nigo Urteaga, § Claudio Agostinelli
University of Trento · Fondazione Bruno Kessler (FBK) · Basque Center for Applied Mathematics (BCAM) · Basque Foundation for Science (Ikerbasque)
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models".
Jane: Here's the gist of the paper. Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models are presented, proposing a fully Bayesian hierarchical and conjugate GLM framework applicable to any distribution in the exponential family.
Tom: This approach models a binary inclusion indicator (z) that directly encodes covariate inclusion within the linear predictor (eta = X(beta z)), simultaneously performing variable selection and parameter estimation.
Jane: The framework utilizes conjugate priors consistent with the exponential family structure, leading to closed-form posterior distributions characterized by a Gibbs sampling scheme for inference.
Tom: Theoretical guarantees establish strong posterior consistency: Theorem 1 confirms that the posterior probability concentrates on the true model (P(z = z* X, y, tau) to 1 as n to infinity), and Theorem 2 ensures that the posterior distribution of active regression coefficients (beta(1)) concentrates around the true active coefficient vector beta*(1).
Paper summary: Lu: The authors are Lucia Filippozzi, Iñigo Urteaga, and Claudio Agostinelli from the Department of Mathematics at the University of Trento Lu. They’ve built something that sits in a specific sweet spot between traditional Bayesian variable selection methods and more flexible shrinkage priors Lu.
Tom: The title itself is really telling—"Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models"—it immediately tells you the three main components: conjugate priors, Bayesian variable selection, and generalized linear models.
Jane: When we put it together, what this paper means is that they’ve found a way to make the complicated task of selecting which predictors matter in a model mathematically rigorous and computationally tractable for any distribution in that family Jane. It moves beyond just fitting the data to actually understanding which inputs drive the output.
Meng: I see it as giving us a more reliable tool when we are building models for things like risk assessment or predictive maintenance where knowing *why* a prediction is high or low matters more than just getting the final number right Meng. It’s about interpretability built directly into the selection process.
Lalam: The implication here is that for any statistical model that fits an exponential family, we now have a framework that promises to find the correct set of predictors with high confidence, and it does so through a very structured mathematical approach Lalam. It’s about making sure you don't pick irrelevant stuff just because it looked good in the initial fit.
Tom: So what this paper means for the world is that we can start applying this kind of rigorous variable selection to a much broader range of real-world data, not just textbook examples Tom. It’s about moving from a process where you guess which features are important to one where the math guides you toward the correct set.
Jane: Exactly. It’s less about finding some clever trick and more about building a sturdy mathematical structure that guarantees you get there, provided your data fits that exponential family Jane. That's what this work is all about: providing consistency for variable selection in this specific but very important class of models.
Conclusion: Tom: So, we’re wrapping up on this paper, "Conjugate priors for Bayesian Variable Selection in exponential family Generalized Linear Models." Basically, they’ve taken a general class of models—the exponential family—and built a whole framework that lets you select which inputs matter while estimating the parameters at the same time.
Jane: That sounds intense. So, if I’m driving and I want to know which features in my data are actually useful for predicting something, this approach helps me figure out that without having to guess everything randomly?
Lu: Exactly. The core idea is using priors that fit the structure of the exponential family itself. This leads to these closed-form posterior distributions, which means we can get a solid answer, not just a rough estimate.
Meng: But what about practical use? If I’m running this at scale on real traffic data, does it actually keep up? Does it crash when the number of variables gets huge?
Tom: The paper shows strong consistency guarantees. They prove that as your sample size grows, the method reliably picks the right set of variables and estimates those coefficients correctly. It’s mathematically consistent.
Jane: Consistent is good, but what does that mean for a person just trying to build a predictive model? Does it make the whole process simpler or just more complicated?
Lalam: For me, it means we're moving away from guesswork on which predictors to keep. We’re using math that respects the underlying data structure. It’s about building confidence in those selections because the math forces them to be sensible within that family of models.
Lu: The theoretical work is pretty thorough, covering different levels of model complexity. They show it holds up across various scenarios where you might have a lot of variables or maybe some complicated relationships between them.
Tom: It seems like they’ve really nailed the mechanics on making variable selection and parameter estimation happen together smoothly for any distribution in that exponential family structure.
Jane: So, if this framework works well for Normal, Poisson, or Binomial models, what does that tell us about the limits of what we can do with standard statistical modeling tools?
Lalam: It tells us that for these specific model types, we have a very reliable way to find those active coefficients. It’s a big step toward making selection less of an art and more of an applied science.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates