Model Selection in High-Dimensional Linear Regression using Boosting with Multiple Testing
summary
The gist
The paper proposes Boosting with Multiple Testing (BMT), a novel method that integrates forward stepwise variable selection (boosting) with a multiple testing framework derived from Chudik et al.
In short
The paper introduces Boosting with Multiple Testing (BMT), a method combining variable selection boosting with multiple testing. BMT sequentially adds only the most significant variable while filtering candidates using a multiple testing framework. This approach aims to select true signals accurately while avoiding noise and proxy variables, offering better performance than existing methods like OCMT and Lasso.
Key concepts
- Boosting with Multiple Testing (BMT)
- A novel method that combines sequential variable selection (boosting) with a multiple testing filter. It selects the most important variable at each step based on previous selections, while simultaneously filtering remaining candidates to control false discoveries from noise and proxy variables.
- Oracle-type Properties
- This means BMT performs very well relative to an ideal model that includes all true signals and excludes pure noise. The theory shows that BMT tends to select the correct true model and achieves standard, optimal error rates for prediction and coefficient estimation.
- Signal-to-Proxy Dominance Condition
- A specific condition under which BMT is guaranteed to recover the full true model. This condition is tailored to the relationship between signal variables and proxy variables, making it a less strict requirement than global conditions needed for Lasso consistency.
Terminology used across episodes
This episode discusses
- Model Selection in High-Dimensional Linear Regression using Boosting with Multiple Testing · Paper Radio
- Selective Sequential Model Selection
The paper
Model Selection in High-Dimensional Linear Regression using Boosting with Multiple Testing · Read on arXiv
George Kapetanios, Vasilis Sarafidis, Alexia Ventouri
King’s College, London · Brunel University London
This paper studies model selection in high dimensional regression settings. We propose a Boosting with Multiple Testing (BMT) approach designed to address proxy signal contamination in environments with strongly correlated regressors, including settings driven by latent common factors. At each stage, at most one regressor is selected conditional on those already included, using a multiple testing threshold for the remaining candidates. Under stated dependence, detectability and design conditions, BMT recovers the exact true model with probability approaching one. A separate extension permits growing signal counts under additional restrictions. For fixed numbers of signals and proxies, we also establish recovery of an approximating model containing all true signals and no noise variables under a separate set of conditions. Following exact selection, fixed dimensional least squares has the oracle asymptotic distribution under the stated moment conditions. A theoretical comparison identifies designs and tuning sequences for which BMT is consistent while the Lasso is not. In the main Monte Carlo comparison, BMT has the highest median Matthews correlation coefficient and exact recovery rate and the lowest median coefficient root mean squared error among the compared specifications. Applications to U.S. emissions and inflation show that BMT combines sparse specifications with favourable forecast performance, including the lowest emissions forecasting errors in the reported comparisons and the best one quarter ahead forecasts in the baseline inflation comparison.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Model Selection in High-Dimensional Linear Regression using Boosting with Multiple Testing".
Warren: Here's the gist of the paper. The paper proposes Boosting with Multiple Testing (BMT), a novel method that integrates forward stepwise variable selection (boosting) with a multiple testing framework derived from Chudik et al.
Ruth: (2018).
Warren: The core mechanism involves sequentially adding only the most significant regressor at each stage, conditional on previously selected variables, while simultaneously applying a family-wise multiple testing filter to the remaining candidates.
Ruth: This approach is designed to maintain strong screening properties while operating in a less greedy manner concerning proxy and noise variables.
Paper summary: Ruth: Okay, so to recap where we are: this paper proposes Boosting with Multiple Testing, or BMT. The core thesis is that you can combine forward stepwise variable selection with a multiple testing filter from Chudik et al., two thousand eighteen <ref:2602.19705#pg1>. This keeps the good screening properties but makes the process less greedy concerning proxy and noise variables.
Warren: It claims that using sharp probability inequalities for processes called heterogeneous strongly mixing, they can prove BMT has oracle type properties relative to an approximating model that includes all true signals and excludes pure noise variables.
Iris: The mechanism is sequential: you add only the most significant regressor conditionally, while applying a family-wise multiple testing filter to the remaining candidates at every stage. This is designed to maintain strong screening properties while being less greedy about proxy and noise variables.
Ruth: And they establish three main theoretical results from this framework. First, under assumptions similar to Chudik et al., BMT has oracle type properties relative to an approximating model containing all true signals and at most a finite number of proxy signals.
Warren: The second result is about the comparison with OCMT. They show that the sequential updating in BMT is less greedy than OCMT because there’s a positive probability of stopping before all proxy signal regressors are included.
Iris: And under a tractable signal-to-proxy dominance condition, BMT asymptotically recovers the full true model, while OCMT tends to retain all proxy variables that pass their marginal testing step. That's a big distinction in how they handle those non-essential predictors.
Ruth: The third result connects the selection process to estimation: the post-selection least squares estimator from the BMT-selected model is asymptotically equivalent to an infeasible oracle estimator if you knew the true model upfront.
Warren: So, what this means for us right now is that this method provides a principled way to handle model selection in high dimensions by balancing accuracy and stability across sequential steps. It’s not just one trick; it builds on established frameworks while refining the selection criteria for real-world complexity.
Iris: And the work is robust because it handles dependence structures like heterogeneous strongly mixing processes, and even allows for polynomial probability tails, which is a nice relaxation of standard assumptions.
Conclusion: Ruth: So looking at this paper, "Model Selection in High-Dimensional Linear Regression using Boosting with Multiple Testing," the authors are showing how to make the variable selection process more stable when you’re dealing with huge datasets where you have many predictors.
Warren: They are proposing BMT as a way to manage that complexity by combining boosting's ability to iteratively pick variables with a multiple testing filter that controls greediness in proxy variables.
Iris: The main implication is that this method gives us confidence that the selection process will converge towards the correct model structure under reasonable assumptions, even when things get complicated.
Ruth: It’s about moving away from just relying on penalized regression or purely greedy boosting algorithms when you have many strongly correlated variables; BMT offers a way to keep the estimation stable and interpretable.
Warren: For someone running these models, it means they can expect better model selection accuracy and more stable coefficient estimates than with current methods like OCMT or standard Lasso when multicollinearity is severe.
Iris: The limitation they state is that the recovery of the true model under their signal-to-proxy dominance condition is local and tailored specifically to that proxy-signal setting, which isn't as broad as global conditions needed for Lasso consistency.
Ruth: That’s a fair point. So, while BMT is powerful, you need to be mindful of the specific conditions under which it achieves its best results. It's a tool with defined boundaries rather than a magic solution for every single high-dimensional problem out there.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19