Variable selection in linear regression models: choosing the best subset is not always the best choice

summary

The gist

Variable selection in linear regression models: choosing the best subset is not always the best choice Best subset selection (BSS) is often considered the intuitive ’gold standard’…

In short

The study compared Best Subset Selection (BSS) against Lasso and Elastic Net for variable selection in linear regression across various settings. BSS performed best only when signal-to-noise ratios were high and predictors were uncorrelated. In settings with correlated predictors or low signal, alternatives like Enet or Lasso are preferred.

Key concepts

Best Subset Selection (BSS)
BSS is a method that tries every possible combination of predictor variables to find the subset that yields the best model performance. It's considered intuitive but is computationally very difficult for large datasets.
Lasso and Elastic Net
These are alternative methods used for variable selection in high-dimensional settings. Lasso uses a penalty to shrink coefficients, while Elastic Net combines L1 (Lasso) and L2 (Ridge) penalties. They are often faster than BSS but might not perform as well under certain conditions.
Signal-to-Noise Ratio
This ratio measures how much the true underlying relationship between variables is visible compared to random noise in the data. A high ratio means the true predictors are clearly distinguishable, which is a favorable condition for BSS success.
Correlation Between Predictors
This refers to how much different predictor variables move together. When predictors are highly correlated, methods like Elastic Net or Ridge-weighted Enet often perform better than BSS because they handle redundancy more effectively.

Terminology used across episodes

This episode discusses

The paper

Variable selection in linear regression models: choosing the best subset is not always the best choice · Read on arXiv

Leibniz Institute for Prevention Research & Epidemiology – BIPS

Variable selection in linear regression settings is a much discussed problem. Best subset selection (BSS) is often considered the intuitive 'gold standard', with its use being restricted only by its NP-hard nature. Alternatives such as the least absolute shrinkage and selection operator (Lasso) or the elastic net (Enet) have become methods of choice in high-dimensional settings. A recent proposal represents BSS as a mixed integer optimization problem so that much larger problems have become feasible in reasonable computation time. We present an extensive neutral comparison assessing the variable selection performance, in linear regressions, of BSS compared to forward stepwise selection (FSS), Lasso and Enet. The simulation study considers a wide range of settings that are challenging with regard to dimensionality (with respect to the number of observations and variables), signal-to-noise ratios and correlations between predictors. As main measure of performance, we used the best possible F1-score for each method to ensure a fair comparison irrespective of any criterion for choosing the tuning parameters, and results were confirmed by alternative performance measures. Somewhat surprisingly, it was only in settings where the signal-to-noise ratio was high and the variables were (nearly) uncorrelated that BSS reliably outperformed the other methods, even in low-dimensional settings. Further, the FSS's performance was nearly identical to BSS. Our results shed new light on the usual presumption of BSS being, in principle, the best choice for variable selection. Especially for correlated variables, alternatives like Enet are faster and appear to perform better in practical settings.

DOI: 10.1002/bimj.202200209

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Variable selection in linear regression models".

Jane: Here's the gist of the paper. Variable selection in linear regression settings is a much discussed problem where Best Subset Selection (BSS) is often considered the intuitive ’gold standard’, with its use being restricted only by its NP-hard nature Alternatives such as the least absolute shrinkage and selection operator (Lasso) or the elastic net (Enet) have become methods of choice in high-dimensional settings A recent proposal represents BSS as a mixed integer optimization problem so that much larger problems have become feasible in reasonable computation time The simulation study considers a wide range of settings that are challenging with regard to dimensionality (with respect to the number of observations and variables), signal-to-noise ratios and correlations between predictors Somewhat surprisingly, it was only in settings where the signal-to-noise ratio was high and the variables were (nearly) uncorrelated that BSS reliably outperformed the other methods, even in low-dimensional settings Comparison of Selection Methods The paper presents an extensive neutral comparison assessing the variable selection performance, in linear regressions, of BSS compared to forward stepwise selection (FSS), Lasso and Enet The main measure of performance used was the best possible F1-score for each method to ensure a fair comparison irrespective of any criterion for choosing the tuning parameters Further, the FSS’s performance was nearly identical to BSS The results shed new light on the usual presumption of BSS being, in principle, the best choice for variable selection Especially for correlated variables, alternatives like Enet are faster and appear to perform better in practical settings Performance Under Different Conditions The results show that BSS reliably outperformed the other methods only in settings with a high signal-to-noise ratio and when the variables were uncorrelated Even in a low-dimensional synthetic data setting, with the number of observations ten times the number of variables, the selection performance of BSS drops dramatically if the true predictors are moderately correlated In those cases, BSS is even outperformed by the Lasso, which is known to be inconsistent for variable selection when the true predictors are highly correlated Interestingly, the much simpler FSS achieves similar performance as BSS in almost all settings and, in some, performs even slightly better Role of Signal-to-Noise Ratio and Correlation The authors found that BSS reliably outperformed the other methods only in settings with a high signal-to-noise ratio and when the variables were uncorrelated The results for the synthetic settings are corroborated by those for the semi-synthetic data settings with the real-world correlation structures Again, the BSS and FSS performances are similar and, especially in the high-dimensional setting, the performance of BSS does not improve much with a high signal-to-noise ratio Even in a low-dimensional setting, BSS is clearly inferior to the Enet and the Lasso when the signal-to-noise ratio is low to moderate Limitations of Best Subset Selection The problem was noticeable in all our simulations settings, not only in the settings investigated for certification, whenever the signal-to-noise ratio was not high or the true direct predictors were correlated This suggests that the objective function (3) and the discrete nature of the optimisation does not easily allow BSS to distinguish between further true predictors and false ones in such settings In summary, the L0-norm penalization seems only appropriate in situations where a high signal-to-noise ratio and (nearly) uncorrelated true predictors are plausible Alternatively, based on our empirical results, using a Ridge-weighted Enet seems a good choice for settings with correlated predictors

Title and authors: Tom: So we're looking at this paper from the Leibniz Institute for Prevention Research and Epidemiology about variable selection in linear regression models: choosing the best subset is not always the best choice. It looks like they’re taking something we usually think is a perfect standard, Best Subset Selection or BSS, and putting it through a real stress test against other popular methods like Lasso and Elastic Net.

Jane: Right, so BSS is often seen as the gold standard because if you assume there are s predictors and you look at all possible combinations up to size k, the true model must be one of those candidates, making BSS the obvious choice for variable selection <ref:2302.12034#pg2>. But this paper seems to suggest that's not always true in practice.

Lu: The core idea here is that while BSS sounds intuitive because it checks every possible subset, a naive implementation quickly becomes computationally infeasible when you have many variables, which is the main reason people look for alternatives like Lasso or Elastic Net <ref:2302.12034#pg1>. This paper sets up a comparison to see how these methods actually perform in real high-dimensional settings.

Meng: And what I'm curious about is how they handle the scale of the problem; when you have thousands of variables, BSS becomes practically impossible unless you reformulate it as a mixed integer optimization problem to make it feasible computationally <ref:2302.12034#pg1>. That’s a practical improvement for anyone trying to deploy these models.

Lalam: The paper presents an extensive neutral comparison assessing the variable selection performance in linear regressions between BSS, forward stepwise selection, Lasso, and Elastic Net using the best possible F1-score as their main measure of performance <ref:2302.12034#pg2>. It’s a fair way to compare them regardless of how you tune those parameters.

Tom: So what they found is that their results shed new light on the usual presumption that BSS is in principle the best choice for variable selection <ref:2302.12034#pg2>. They looked at a wide range of settings, challenging things like dimensionality and signal-to-noise ratios, and surprisingly, BSS only reliably outperformed the other methods when the signal-to-noise ratio was high and the variables were nearly uncorrelated.

Jane: That’s a big caveat there; it seems BSS isn't universally superior <ref:2302.12034#pg2>. They also found that for correlated variables, alternatives like Elastic Net are often faster and seem to perform better in practical settings <ref:2302.12034#pg2>. This makes sense if you’re dealing with real-world data where predictors are usually not perfectly independent.

Title and authors: Lu: The paper notes that even in a low-dimensional synthetic data setting, with the number of observations ten times the number of variables, BSS's selection performance drops dramatically if the true predictors are moderately correlated <ref:2302.12034#pg2>. And in those cases, BSS can actually be outperformed by Lasso, which is known to be inconsistent when those true predictors are highly correlated <ref:2302.12034#pg2>.

Meng: If we’re talking about deployment, that inconsistency with Lasso when predictors are correlated is a major red flag for us at scale <ref:2302.12034#pg2>. It means you can't just pick one method and hope it works everywhere without checking the data structure first.

Lalam: The authors also found that BSS and forward stepwise selection actually achieve nearly identical performance <ref:2302.12034#pg2>, and in some cases, forward stepwise selection even performs slightly better in almost all settings <ref:2302.12034#pg2>. That’s an interesting finding when you think about how simple methods can sometimes beat the complex ones.

Tom: So we have this trade-off: BSS is theoretically sound but computationally hard, while simpler methods like forward stepwise selection are surprisingly competitive, even better in some cases <ref:2302.12034#pg2>. They clearly aren't the only tool we have to consider when selecting predictors.

Jane: The paper then goes on to discuss their improvements, specifically suggesting that using a Ridge-weighted Elastic Net seems like a good choice for settings where the true predictors are correlated <ref:2302.12034#pg3>. This is a concrete recommendation for when you suspect your variables aren't perfectly independent <ref:2302.12034#pg3>.

Lu: That points to the implication that for settings with correlated predictors, we should probably move away from BSS and look towards methods like Elastic Net because they appear more practical and faster in those scenarios <ref:2302.12034#pg2>. This aligns with their suggestion that Enet can be faster and perform better in practice <ref:2302.12034#pg2>.

Meng: From an engineering standpoint, if I have a pipeline that needs to run on production data where multicollinearity is expected, recommending Ridge-weighted Elastic Net makes sense because it has known stability properties <ref:2302.12034#pg3>. It’s less likely to break at scale than relying solely on the naive BSS approach <ref:2302.12034#pg1>.

Lalam: The main limitation they point out is that BSS and forward selection don't distinguish between true predictors and false ones well when the signal-to-noise ratio isn't high or when the true direct predictors are correlated <ref:2302.12034#pg3>. This suggests that the L0 norm penalization, which BSS uses, might only be appropriate when we have high signal quality and nearly uncorrelated true predictors <ref:2302.12034#pg3>.

Tom: So in summary, while BSS is a good starting point conceptually because it’s the gold standard, this paper shows it’s not always the best practical choice <ref:2302.12034#pg2>. The real takeaway is that if you're dealing with correlated variables or high dimensionality, you should probably look at Elastic Net or Lasso instead <ref:2302.12034#pg2>.

Title and authors: Jane: Exactly. They also highlighted that the computational feasibility of BSS has been improved by formulating it as a mixed integer optimization problem, allowing it to handle problems with thousands of variables in a way that is reasonable <ref:2302.12034#pg1>. That makes the gold standard method actually usable for much larger problems now.

Lu: So, the big picture here is shifting: instead of always defaulting to BSS, we need a more nuanced approach based on whether our data has high signal quality and how correlated the predictors are <ref:2302.12034#pg3>. This guides us toward methods that are faster and better when those assumptions aren't perfectly met <ref:2302.12034#pg2>.

Meng: For deployment, that means we should test Elastic Net first in scenarios where we know multicollinearity might be present, because it seems to have better practical performance in those situations <ref:2302.12034#pg2>.

Lalam: The paper "Variable selection in linear regression models: choosing the best subset is not always the best choice" gives us a clear path forward: use BSS only when you have very high signal and uncorrelated variables, and otherwise, consider Elastic Net or Ridge-weighted versions for better performance with correlated predictors <ref:2302.12034#pg3>.

Tom: It’s interesting how they framed this comparison—comparing it directly against forward stepwise selection shows that the simpler methods are often just as good as BSS in many situations <ref:2302.12034#pg2>. That's a bit surprising, but it grounds the discussion in practical performance metrics rather than just theoretical perfection.

Jane: It really does show that the intuitive choice isn't always the one that wins in a real-world setting <ref:2302.12034#pg2>. We’ve seen this pattern with other papers too, where methods like Lasso or Enet prove to be more robust when you introduce complexities like correlation or high dimensionality <ref:2302.12034#pg2>.

Lu: I think what this paper really contributes is making the performance landscape clearer by showing exactly *when* BSS fails and *why*, which helps researchers choose the right tool for their specific data structure <ref:2302.12034#pg3>.

Meng: For those of us shipping models, knowing that Elastic Net is often faster and better when variables are correlated means we can design our pre-processing steps to favor those methods in those cases <ref:2302.12034#pg2>. That’s actionable information.

Lalam: So the final thought from this paper is a practical one: stop treating BSS as an automatic winner, especially if your signal-to-noise ratio is low or your predictors are likely to be correlated <ref:2302.12034#pg3>. This empirical comparison provides the evidence for that shift in strategy <ref:2302.12034#pg2>.

The paper's summary: Tom: So, we’ve been looking at how Best Subset Selection, or BSS, usually wins when you pick variables for a linear regression model because it checks every possible combination <ref:2302.12034#pg2>. But this paper is basically saying that intuition can be misleading in high-dimensional settings.

Jane: Right. It sets up a real test comparing BSS against other popular methods like Lasso and Elastic Net using the best possible F1-score as their performance measure <ref:2302.12034#pg2>. The main finding is that BSS only really beats the others when you have a very high signal-to-noise ratio and your predictors aren't correlated at all.

Tom: That’s wild because we’re often told BSS is the gold standard for selection, but this paper shows it actually falls behind in many real situations <ref:2302.12034#pg2>. They found that when things get complex—like when the true predictors are moderately correlated—BSS performance drops significantly.

Jane: And they pointed out that alternatives like Elastic Net seem much more practical and often perform better when you’re dealing with those correlated variables <ref:2302.12034#pg2>. They also noted that forward stepwise selection, which is way simpler than BSS, actually performs almost identically to BSS in almost every setting <ref:2302.12034#pg2>.

Tom: So it’s not just about the theoretical best choice anymore; it's about what works on actual data structures <ref:2302.12034#pg3>. The authors flag a limitation: if your signal-to-noise ratio isn't high or your true predictors are correlated, BSS gets confused and can’t tell the difference between real predictors and false ones <ref:2302.12034#pg3>.

Jane: That suggests that the mathematical way BSS tries to penalize complexity might just not be robust enough for messy, real-world data <ref:2302.12034#pg3>. They specifically suggest that if you have correlated predictors, using a Ridge-weighted Elastic Net is a better choice than BSS because it handles those correlations more gracefully <ref:2302.12034#pg3>.

Tom: And on the practical side, they did something smart—they reformulated BSS as a mixed integer optimization problem so that even huge problems with thousands of variables become computationally feasible in reasonable time <ref:2302.12034#pg1>. So, the gold standard method is now actually usable for bigger datasets <ref:2302.12034#pg1>.

Jane: It really shifts how we think about variable selection. Instead of blindly trusting one method, researchers need to look at the data structure first—how high is the signal quality and how correlated are the variables—to decide whether they should use BSS, Lasso, or Elastic Net <ref:2302.12034#pg2>.

Tom: Exactly. The implication for anyone building a model is that you shouldn't always default to BSS; if you have correlated inputs, look toward Elastic Net because it’s faster and seems more reliable in those situations <ref:2302.12034#pg2>.

Jane: This gives us a clearer direction for choosing the right tool, moving away from just picking the method with the fanciest name <ref:2302.12034#pg2>. And that leaves us wondering how this idea of tuning methods based on data structure impacts how we handle missing data in other types of analysis—like those multiple imputation techniques we talked about earlier <ref:2302.12034#pg1>.

The paper's improvements: Tom: So, we’ve heard that BSS isn't always the best choice for variable selection because it struggles when things get messy with correlated predictors or low signal quality <ref:2302.12034#pg3>. Now, the paper isn't just pointing out problems; they’re proposing actual fixes for those problems <ref:2302.12034#pg3>.

Jane: They suggest that instead of just using one method, we should use Ridge-weighted Elastic Net when we suspect our predictors are correlated because it seems much better in those scenarios <ref:2302.12034#pg3>. That’s a concrete recommendation for handling real-world data complexity <ref:2302.12034#pg3>.

Tom: That makes sense from an engineering standpoint because Enet is often faster and more stable when dealing with multicollinearity, which means less overhead when we’re shipping models at scale <ref:2302.12034#pg2>. It moves us toward a more robust setup than just relying on the naive BSS approach <ref:2302.12034#pg1>.

Jane: And they also mentioned that by reformulating BSS as a mixed integer optimization problem, it became feasible to run it even on massive problems with thousands of variables in a reasonable amount of time <ref:2302.12034#pg1>. So the gold standard method is now much more accessible computationally <ref:2302.12034#pg3>.

Tom: That’s a big win for deployment because feasibility is everything; if we can run the selection process quickly, we can iterate on our models faster without burning through all our compute resources <ref:2302.12034#pg1>.

Jane: It really changes the playbook for model building: don't assume BSS is always the answer; instead, check your data structure and pick a method tailored to those conditions <ref:2302.12034#pg2>. This gives us a much more nuanced way to choose our tools in practice <ref:2302.12034#pg3>.

Tom: So we’ve moved from just knowing what BSS does to knowing when *not* to use it, and what specific alternatives—like Ridge-weighted Elastic Net—to deploy instead for better results <ref:2302.12034#pg2>. That’s actionable information for anyone trying to build a production system on arXiv papers <ref:2302.12034#pg3>.

Jane: It’s exciting because it shows that the theoretical gold standard is actually more like a starting point than an automatic winner in applied settings <ref:2302.12034#pg2>. This opens up so many possibilities for designing better preprocessing pipelines based on the data itself <ref:2302.12034#pg3>.

Conclusion: Tom: So we’ve spent some time looking at how Best Subset Selection, or BSS, isn't always the best choice when picking variables for linear regression models because it struggles with complex data structures <ref:2302.12034#pg3>.

Jane: Right. The main idea is that we need to stop treating BSS as the automatic winner and start matching the method to our specific data characteristics <ref:2302.12034#pg2>.

Tom: It really boils down to signal quality and how correlated the predictors are—if those conditions aren't met, BSS just fails and gets inconsistent results <ref:2302.12034#pg3>.

Jane: And they showed that for those messy situations, methods like Ridge-weighted Elastic Net seem much more practical and perform better <ref:2302.12034#pg3>. It’s a big shift from just picking the method with the fanciest name <ref:2302.12034#pg2>.

Tom: I liked seeing how they framed it—comparing BSS to forward stepwise selection and showing that sometimes those simpler methods are almost as good as the complex ones in most settings <ref:2302.12034#pg2>.

Jane: And they did a solid job making BSS actually usable on large problems by turning it into a mixed integer optimization problem so we can handle thousands of variables <ref:2302.12034#pg1>. That's a practical improvement for anyone trying to deploy models right now <ref:2302.12034#pg1>.

Tom: So the paper, "Variable selection in linear regression models: choosing the best subset is not always the best choice," gives us a clear rule: check your signal and correlation first before defaulting to BSS <ref:2302.12034#pg3>.

Jane: Exactly. It’s about moving toward a more data-aware approach rather than a purely theoretical one <ref:2302.12034#pg2>. This makes model selection much smarter when you’re working with real, imperfect data <ref:2302.12034#pg3>.

Tom: For me, it means I need to be more careful about my settings and not just pick the most complicated sounding technique blindly <ref:2302.12034#pg2>.

Jane: It opens up so many avenues for how we design our pipelines going forward, focusing on what the data actually tells us <ref:2302.12034#pg3>. This leads me to think about how this affects imputation strategies, like those multiple imputation techniques we talked about earlier <ref:2302.12034#pg1>.

Tom: I'm ready for the next paper—I want to see if they can tackle missing data challenges with these new selection insights <ref:2302.12034#pg2>.

Jane: Let’s see what's next on arXiv and figure out how this selection insight fits into those bigger missing data puzzles <ref:2302.12034#pg1>.

Lu: I think this work is important because it’s grounded in empirical results showing where the assumptions of simpler methods break down, which helps us understand the representation learning landscape better <ref:2302.12034#pg2>.

Meng: From my side, if we can deploy a more robust selection strategy that accounts for correlation, it means less time debugging models that fail at scale due to poor variable choices <ref:2302.12034#pg1>.

Lalam: I'm confident in this direction; the paper's findings about when to use Elastic Net versus BSS are very solid based on the evidence they presented <ref:2302.12034#pg3>.

More episodes

← Home