Bayesian MI-LASSO for Variable Selection on Multiply-Imputed Data

arXiv:2211.00114 · stat.ME · Submitted 2022-10-31 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Bayesian MI-LASSO for Variable Selection on Multiply-Imputed Data".

Jane: Here's the gist of the paper. Multiple imputation is widely used for handling missing data in realworld applications.

Tom: In this work, we focus on variable selection using multiply-imputed data.

Jane: MI-LASSO, one of the most commonly used approaches to this problem, regards the same variable across all separate imputed datasets as a group variable and exploits the group LASSO to yield a consistent variable selection across all the multiply-imputed datasets.

Tom: In this paper, we extend MI-LASSO to a Bayesian framework and propose four Bayesian MI-LASSO models for variable selection on multiply-imputed data.

Paper summary: Jane: So to wrap up, we've looked at the paper called Bayesian MI-LASSO for Variable Selection on Multiply-Imputed Data. It proposes extending MI-LASSO into a Bayesian framework with four different models, including shrinkage and Spike-Slab priors, all aimed at making variable selection more consistent across different imputed datasets >

Tom: The authors are arguing that this method provides a robust way to do variable selection on multiply-imputed data by using a four-step projection predictive procedure instead of ad hoc thresholding >

Lu: The key is the extension into the Bayesian framework, placing those shared priors on each grouped coefficient vector beta j, which enables consistent selection across all imputations >

Meng: It seems like for practical use, they are recommending starting with Horseshoe or ARD because they don's require tuning and deliver strong performance >

Jane: And that’s the big picture: we have a framework here for handling missing data and variable selection that offers better consistency than standard methods >

Tom: The implication is that if you're dealing with complex missing data scenarios, this paper gives you a solid blueprint for building systems where variable selection doesn't suddenly change depending on which dataset you look at >

Conclusion: Tom: So, we're wrapping up this look at "Bayesian MI-LASSO for Variable Selection on Multiply-Imputed Data." Essentially, they took standard variable selection methods and put them inside a Bayesian framework to handle data that has missing pieces in multiple places.

Jane: The authors are using these four different Bayesian models—three shrinkage priors and one Spike-Slab prior—to figure out which variables are actually important when you have that missing data problem.

Lu: They introduce this idea of placing a shared, penalized prior on each group of coefficients, which is the core mechanism that lets them find consistent variable selection across all those different imputed datasets.

Meng: From an engineering standpoint, what they're doing here is moving away from guessing thresholds and instead using a projection predictive procedure to make sure the variable selection stays stable even when you're making inferences after selecting your variables.

Tom: It sounds like they are showing that this Bayesian approach isn't just a theoretical exercise; it actually delivers better performance than MI-LASSO or other alternatives in terms of accuracy and error rates.

Jane: And the big implication for us is that if you're working with complex missing data—like in medical studies or environmental science—this method offers a much more robust way to get reliable results about which variables are driving the outcome.

Lu: The paper shows that even though they have four different model choices, they all perform quite well, and the choice between shrinkage and Spike-Slab depends on whether you want higher sensitivity or higher specificity for your variable selection.

Tom: It's interesting how they tie this together with a specific real-world study on dioxin exposure data from the University of Michigan; it shows these methods work in practice, not just on simulated data.

Jane: And that leads us to the next part where we'll look at what those specific results actually mean for understanding those environmental exposures.

Columbia University · Rutgers University

stat.ME

Submitted: 2022-10-31

Updated: 2026-11-05

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: Multiple imputation is widely used for handling missing data in realworld applications.

Key concepts

Multiple Imputation (MI)
MI is a technique used to handle missing data by creating several complete datasets based on statistical models. The analysis is then performed on each dataset, and the results are combined to provide more accurate inferences than using a single imputed dataset.
MI-LASSO
This method uses multiple imputation combined with the LASSO regularization technique for variable selection. It treats variables within groups as related and uses a group LASSO approach to select consistent variables across all the different imputed datasets.
Bayesian MI-LASSO
This extension incorporates a Bayesian framework by placing shared, penalized priors on grouped coefficient vectors. This structure allows the model to incorporate group-level information through these priors, leading to coherent uncertainty estimates and consistent variable selection across imputations.

Terminology

Summary

Multiple imputation is widely used for handling missing data in realworld applications. p. 0 In this work, we focus on variable selection using multiply-imputed data. p. 1 MI-LASSO, one of the most commonly used approaches to this problem, regards the same variable across all separate imputed datasets as a group variable and exploits the group LASSO to yield a consistent variable selection across all the multiply-imputed datasets. p. 0 In this paper, we extend MI-LASSO to a Bayesian framework and propose four Bayesian MI-LASSO models for variable selection on multiply-imputed data. p. 0

Bayesian Framework Extension

The authors extend MI-LASSO to a Bayesian framework, proposing four Bayesian MI-LASSO models, including three shrinkage prior-based and one Spike-Slab prior-based methods. This extension allows for the incorporation of group-level information through shared penalized priors and provides coherent uncertainty estimates. p. 1 The key idea of Bayesian MI-LASSO is to place a shared, penalized prior on j-th grouped coefficient vector βj, i. p. 3i.d ∼ πβj(·), for j = 1,..., p. This structure enables consistent variable selection across all imputations. p. 3

Four Bayesian MI-LASSO Models

The four proposed Bayesian MI-LASSO models include three shrinkage-based models and one Spike-and-Slab type model. The shrinkage models apply continuous priors that concentrate mass near zero, promoting shrinkage without forcing exact sparsity. p. 4 Specifically, the first model specifies Multi-Laplace priors for grouped regression coefficients. p. 4 The second shrinkage model leverages Horseshoe priors for group coefficients, introducing an additional global parameter τ to control overall shrinkage. p. 4 The last shrinkage model extends the grouped Automatic Relevance Determination (ARD) model into the Bayesian MI-LASSO framework. p. 5

Projection Predictive Variable Selection Procedure

To address the challenge of variable selection across posterior draws and post-selection inference in the Bayesian variable selection setting, a flexible four-step procedure for projection predictive variable selection is developed. This approach avoids ad hoc thresholding and facilitates valid post-selection inference by leveraging predictive projections and a modified Bayesian information criterion. p. 1 The procedure involves generating a candidate collection of variable selection subsets using a scaled neighborhood criterion, projecting the posterior draws from the full model onto each candidate subset, evaluating each projected model using the BICm, and finally providing post-selection inference. p. 8

Simulation Study Results

Simulation studies showed that the Bayesian MI-LASSO outperformed MI-LASSO and other alternative approaches, achieving higher specificity and lower mean squared error across a range of settings. The four proposed Bayesian MI-LASSO models demonstrated consistently strong and similar performance, outperforming the alternatives in variable selection and estimation accuracy across all settings. p. 1 In the University of Michigan Dioxin Exposure Study case study, all covariates selected by the Bayesian MI-LASSO models were statistically significant with 95% probability intervals excluding zero. p. 0

Case Study Application

The application to the University of Michigan Dioxin Exposure Study showed that all four Bayesian MI-LASSO models selected a common set of 10 variables along with the intercept. The results were consistent with simulation findings, where the Multi-Laplace and Spike-and-Slab models tended to yield more parsimonious selections with slightly higher specificity, whereas the Horseshoe and ARD models favored greater sensitivity by including more relevant variables. p. 0 Furthermore, all four models achieved modified BIC values below -2.4, indicating good model fit and the estimated degrees of freedom reflect differences in shrinkage. p. 2

Discussion and Recommendations

The authors recommend starting with the Horseshoe or ARD priors as they require no tuning, deliver strong performance, and tend to favor slightly higher sensitivity across simulation scenarios. The four-step selection procedure is proposed as a novel and robust framework that avoids arbitrary thresholds and supports valid inference by projecting the full-model posterior onto reduced models. p. 8 The default prior specifications are recommended for general use, with the caveat to avoid extremely small v or b values which may lead to instability and. p. 6 The framework is applicable to both shrinkage and Spike-and-Slab Bayesian MI-LASSO models, requiring only a grid of pre-specified thresholds between 0 and 1. p. 8

Future Work Directions

One natural extension is to explore alternative Bayesian shrinkage or Spike-and-Slab priors beyond those studied in this paper. Another direction is to extend the current framework to accommodate different outcome types, such as binary or ordinal responses, by incorporating latent variable models. p. 0 The authors also note that joint modeling of imputation and variable selection could better capture uncertainty across the whole process. p. 7 The theoretical side suggests that selection consistency for some grouped Spike-and-Slab models is difficult to obtain when the imputation model is misspecified. p. 9

Acknowledgments and Funding

The authors would like to thank Dr. David Garabrant and Xiaohui Jiang for providing the UMDES data used in the application. p. 0 The work was supported by the National Institute of Environmental Health Sciences of the National Institutes of Health under Award Number R01ES035784.

Supplementary Material Details

The supplementary material provides details and theoretical analysis of the Gibbs samplers for the Bayesian MI-LASSO models, derives the degrees of freedom estimates, and presents additional information on the simulation study and real data application. This includes details on Harris ergodicity for all four models. p. 7 The derivation of the degrees of freedom estimators is also provided in Supplementary Materials S3 and S2. p. 6 The results for the UMDES data application are summarized in Table 5. p. 1 The comparison of performance across different settings is provided in Table 3 <ref:2211. p. 1 p. 1 p. 100114#pg8>. The results under high-missingness settings are shown in Table 4 <ref:2211. p. 0 p. 0 p. 000114#pg8>. The results for moderate missingness are provided in Web Table 2 <ref:2211. p. 6 p. 6 p. 600114#pg6>. The results for the binary covariate simulation are shown in Table 4 <ref:2211.00114#pg10>. The results under correlated covariates reinforce patterns observed in Simulation 1 <ref:2211. p. 100114#pg4> and Simulation 3 <ref:2211. p. 000114#pg10>. The results for the UMDES data application are summarized in Table 5 <ref:2211. p. 1 p. 100114#pg7>. The comparison of performance across different settings is provided in Table 3 <ref:2211.00114#pg8>. The results under high-missingness settings are shown in Table 4 <ref:2211.00114#pg9>. The results for moderate missingness are provided in Web Table 2 <ref:2211.00114#pg6>.

References

The paper cites numerous works related to multiple imputation, Bayesian methods, and variable selection consistency <ref:2211.00114#pg4> and <ref:2211. p. 000114#pg7>. Key references include Chen and Wang (2013) for MI-LASSO, Carvalho et al. p. 1 (2009) for the Horseshoe prior, Mitchell and Beauchamp (1988) for Spike-and-Slab priors, and Piironen and Vehtari (2020) for projection predictive inference <ref:2211. p. 500114#pg7>. The paper also references work on simulation methods like those by Gelman and Rubin (1992) and MCMC convergence diagnostics like the improved Rˆ statistic <ref:2211. p. 700114#pg9> and Vehtari et al. p. 7 (2021).

Data Description

The UMDES dataset includes serum dioxin measurements from blood samples, dioxin levels from household dust and soil, and detailed participant-level information on demographics, health status, residential history, occupational and recreational activities, lifetime food consumption, and current diet <ref:2211.00114#pg7>. The study focuses on 448 individuals residing in or near the Tittabawassee River floodplain between the Dow Chemical plant in Midland and the confluence with the Shiawassee River in Saginaw <ref:2211.00114#pg7>. The dataset includes 63 continuous and 68 binary covariates capturing demographic and exposure-related characteristics <ref:2211.00114#pg7>.

Variable Selection Results Summary

The analysis showed that all four Bayesian MI-LASSO models selected a common set of 10 variables along with the intercept <ref:2211.00114#pg7> and the Spike-and-Laplace model additionally included months of breast-feeding <ref:2211.00114#pg7>. The Horseshoe and ARD models each selected two further variables <ref:2211. p. 500114#pg7>. All four models achieved modified BIC values below -2.4, indicating good model fit <ref:2211. p. 000114#pg7> and the estimated degrees of freedom reflect differences in shrinkage <ref:2211. p. 000114#pg7>. The Bayesian MI-LASSO models yielded consistent coefficient directions across all four specifications, providing a coherent interpretation of predictor effects <ref:2211. p. 100114#pg7>.

Coefficient Interpretation

For example, the association between age and serum TCDD concentration differs by gender: as individuals age, TCDD levels tend to increase more in females than in males <ref:2211.00114#pg7>. Higher BMI is also associated with elevated serum TCDD levels, while weight gain, months of breast-feeding, and ever smoking are negatively associated with TCDD concentration <ref:2211.00114#pg7>. The analysis identified several residential, occupational, recreational, and property-use factors that likely reflect major exposure pathways <ref:2211.00114#pg7>.

Model Comparison Summary

The Bayesian MI-LASSO models outperformed MI-LASSO and other existing methods in both variable selection and coefficient estimation across most scenarios <ref:2211.00114#pg4>. Notably, the Bayesian MI-LASSO also outperformed standard LASSO and performed comparably to Bayesian LASSO applied to fully observed data without missing data <ref:2211. p. 100114#pg4>. In some settings, they even achieved slightly lower MSE than the corresponding Bayesian LASSO <ref:2211. p. 000114#pg4>. The Horseshoe and ARD priors exhibited slightly higher sensitivity, while the Multi-Laplace and Spike-and-Slab priors showed slightly higher specificity <ref:2211. p. 000114#pg4>.

Variable Selection Mechanism Comparison

MI-LASSO tends to favor higher sensitivity, selecting more variables by including a group variable if any of its components is significant <ref:2211.00114#pg4>. In contrast, the Bayesian MI-LASSO models favor higher specificity, selecting a group variable only when the pooled posterior draws provide strong overall evidence of significance <ref:2211. p. 700114#pg4>. The Multi-Laplace and Spike-and-Slab priors yielded more parsimonious selections <ref:2211. p. 500114#pg7>.

Post-Selection Inference Robustness

The proposed four-step procedure avoids arbitrary thresholds, identifies stable and parsimonious subsets, and enables valid post-selection inference by projecting the full-model posterior onto reduced models. This framework applies to both shrinkage and Spike-and-Slab Bayesian MI-LASSO models and requires only a grid of pre-specified thresholds between 0 and 1. p. 8 The estimated degrees of freedom remain unbiased Monte Carlo estimators and can be applied directly and the modified BIC used for model selection can be replaced with alternative information criteria. p. 2 The procedure is robust across a range of hyperparameters, with default settings performing consistently well <ref:2211. p. 0 p. 000114#pg7>.

Practical Considerations

Among all Bayesian MI-LASSO models, the authors recommend starting with the Horseshoe or ARD priors as they require no tuning, deliver strong performance, and tend to favor slightly higher sensitivity across simulation scenarios <ref:2211.00114#pg7>. The Multi-Laplace and Spike-and-Slab models are recommended to specify hyperparameters by matching the marginal prior variance of the coefficients to the regression variance <ref:2211. p. 600114#pg7>. The final variable selection and projected coefficients remain consistent across a range of prior settings, except under extreme hyperparameter values <ref:2211. p. 000114#pg7>.

Future Work

One natural extension is to explore alternative Bayesian shrinkage or Spike-and-Slab priors beyond those studied in this paper <ref:2211.00114#pg7>. Another direction is to extend the current framework to accommodate different outcome types, such as binary or ordinal responses, by incorporating latent variable models <ref:2211.00114#pg7>. Jointly modeling imputation and variable selection could better capture uncertainty across the whole process <ref:2211. p. 700114#pg7> and selection consistency is difficult to obtain when the imputation model is misspecified <ref:2211. p. 800114#pg7>.

Acknowledgments

The authors would like to thank Dr. David Garabrant and Xiaohui Jiang for providing the UMDES data used in the application <ref:2211. p. 000114#pg7>. The work was supported by the National Institute of Environmental Health Sciences of the National Institutes of Health under Award Number R01ES035784 <ref:2211.00114#pg7>.

The supplementary material provides details and theoretical analysis of the Gibbs samplers for the Bayesian MI-LASSO models, derives the degrees of freedom estimates, and presents additional information on the simulation study and real data application <ref:2211.00114#pg7>. This includes details on Harris ergodicity for all four models <ref:2211. p. 700114#pg9>. The derivation of the degrees of freedom estimators is also provided in Supplementary Materials S3 <ref:2211. p. 600114#pg10> and S2 <ref:2211. p. 000114#pg10>. The results for the UMDES data application are summarized in Table 5 <ref:2211.00114#pg7>. The comparison of performance across different settings is provided in Table 3 <ref:2211.00114#pg8>. The results under high-missingness settings are shown in Table 4 <ref:2211.00114#pg9>. The results for moderate missingness are provided in Web Table 2 <ref:2211.00114#pg6>.

The proposed four-step procedure avoids arbitrary thresholds, identifies stable and parsimonious subsets, and enables valid post-selection inference by projecting the full-model posterior onto reduced models. This framework applies to both shrinkage and Spike-and-Slab Bayesian MI-LASSO models and requires only a grid of pre-specified thresholds between 0 and 1 <ref:2211. p. 800114#pg7>. The estimated degrees of freedom remain unbiased Monte Carlo estimators and can be applied directly <ref:2211. p. 600114#pg8> and the modified BIC used for model selection can be replaced with alternative information criteria <ref:2211. p. 200114#pg8>. The procedure is robust across a range of hyperparameters, with default settings performing consistently well <ref:2211.00114#pg7>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Bayesian MI-LASSO for Variable Selection on Multiply-Imputed Data can perform variable selection consistently across multiple imputed datasets by treating variables as a group and applying a group LASSO penalty to yield a consistent variable selection across all the multiply-imputed datasets.

  2. The improved system can incorporate prior knowledge about variable importance directly into Bayesian models through informative priors, which allows for improved variable selection and overall model performance when experts suggest certain covariates are more relevant.

  3. The system can combine posterior samples from each imputed dataset by pooling them directly, which naturally captures both within- and between-imputation variability, offering a flexible solution over traditional methods relying solely on Rubin's rules.

  4. A four-step projection predictive variable selection procedure avoids ad hoc thresholding and facilitates valid post-selection inference by projecting the posterior distribution of the full model onto a reduced subspace defined by candidate submodels.

  5. The system can generate candidate variable selection subsets using a scaled neighborhood criterion, which is compatible with both shrinkage and Spike-and-Slab models and allows for a grid of pre-specified thresholds between 0 and 1.

  6. The system can evaluate the selected model subset by minimizing the modified Bayesian information criterion BICm, defined as log X D d=1 y d - x d [s]E[β d s data]2 / (Dn)! based on posterior means of projected coefficients.

  7. The improved system provides post-selection inference by projecting each posterior draw onto the selected subset and reporting 95% equal-tailed credible intervals for uncertainty, ensuring that excluded variables are set to zero across all posterior draws.

Related papers