Bias-Aware External-Model-Assisted Inference in High-Dimensional Regression

arXiv:2606.15602 · stat.ME · Submitted 2026-06-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Bias-Aware External-Model-Assisted Inference in High-Dimensional Regression".

Jane: Here's the gist of the paper. Detailed Research Summary: Debiased External-model-Assisted Lasso (DEAL) This paper introduces Debiased External-model-Assisted Lasso (DEAL) as a novel methodology designed to overcome significant structural limitations present in Prediction-Powered Inference (PPI) and its variant, PPI++, particularly when applied to high-dimensional semi-supervised linear regression involving external, potentially transfer-learned estimators.

Tom: Core Problem Addressed: The fundamental weakness of existing methods like PPI/PPI++ lies in the use of a prediction rectifier to ensure validity.

Jane: When both the working model and the predictor are linear, this rectifier exactly cancels the predictor's contribution to the estimation, effectively collapsing both PPI and PPI++ onto Ordinary Least Squares (OLS), irrespective of how good or bad the external estimator is.

Tom: This mechanism can lead to inflated variance when estimators are near-oracle.

Title and authors: Tom: Exactly. And the paper sets out to show that this whole rectifier approach isn't the right way to go for high-dimensional semi-supervised linear regression, because it collapses everything onto a baseline estimator >

Jane: So what’s the main thing they propose instead? It sounds like they ditch that prediction rectifier and use something different when dealing with an external model and unlabeled covariates >

Lu: They propose something called Debiased External-model-Assisted Lasso or DEAL. Instead of routing the external estimator through a prediction rectifier, DEAL routes it into the variance of a debiased estimator >

Meng: Routing it into the variance is interesting from an engineering standpoint, because that sounds like they’re treating the external model's output not as a mean to be corrected by a prediction-based control variate but as something influencing the uncertainty itself >

Tom: So when you route it into the variance, what does that actually enable them to do that PPI couldn't >

The paper's summary: Jane: Well, they describe DEAL as a bias-aware shrinkage step that treats the one-step correction as if it were a noisy observation of an unknown bias vector >

Lu: That lets them adapt to three different scenarios: when you are only using the target Lasso, when the external model is near-oracle but has some bias, and when it’s just biased but still gives you useful information >

Tom: So the overall process involves a few steps: they start with a bias-aware initializer, then they do this pseudo-label imputation using those unlabeled covariates before running a stacked Lasso fitting on that enlarged dataset >

Meng: I need to know what the cost of that enlargement is. If you're adding thousands of imputed responses just to fit the model, how does that affect deployment at scale >

Jane: They do address it by showing their coordinate-wise asymptotic normality under sub-Gaussian design assumptions, which gives them explicit variance estimates based on that enlarged design >

Lu: The paper shows that this method provides coordinate-wise asymptotic normality with an explicit, adaptive variance, and a key finding is that at a common unlabeled budget, the confidence intervals generated by DEAL are strictly shorter than those of target-only debiased Lasso and strictly shorter than PPI when you use a linear labeler >

The paper's improvements: Tom: So they claim efficiency gains come from the enlarged design's variance itself, not just from how accurate those imputed responses are, which is a big point for model deployment >

Jane: They also show that this method performs better than PPI and PPI++ in empirical tests across five different applications, showing median confidence interval length ratios ranging from about zero point two three to zero point five three compared to debiased Lasso >

Meng: That's a fifty percent reduction in the CI length for some cases, but how robust is this against covariate shift? We're often dealing with unlabeled data from populations that don't exactly match the target distribution >

Lu: They address that by showing a one-sided modification of the Stage-four debiasing precision matrix restores asymptotic validity in the shifted regime, meaning it maintains nominal coverage even when your unlabeled data comes from a different population >

Tom: That’s significant because it means you don't have to perfectly control the shift in your unlabeled covariates for the inference to still work >

Conclusion: Jane: So to wrap up, this paper on Bias-Aware External-Model-Assisted Inference in High-Dimensional Regression suggests that routing external information into the variance of a debiased estimator offers a way around the structural weakness of prediction rectifiers >

Tom: It means we can get better confidence intervals because we’re using the unlabeled design to sharpen the estimated precision matrix, and this works even when your external model is imperfect or non-linear >

Lu: And they extend it by showing you can adapt it to labelers that aren't in a linear coefficient form by using the labeler’s population linear projection on the target distribution and its associated residual >

Meng: From a production standpoint, if we can dynamically select the optimal unlabeled sample size based on a variance-balance rule that tracks the bias-variance trade-off, that gives us some real control over deployment risk >

Lalam: I'm confident in this direction. This method is more robust against model uncertainty or covariate shift because it leverages a channel that the rectifier simply can't access >

Hongzhe Zhang, Hanxuan Ye, Hongzhe Li

University of Pennsylvania

stat.ME

Submitted: 2026-06-14

Updated: 2026-10-28

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: Debiased External-model-Assisted Lasso (DEAL) This paper introduces Debiased External-model-Assisted Lasso (DEAL) as a novel methodology designed to overcome significant structural limitations…

Key concepts

Prediction Rectifier Limitation
Existing methods use a prediction rectifier to ensure validity. When both models are linear, this step cancels the external predictor's contribution entirely, collapsing the method to Ordinary Least Squares (OLS). This limits how well external information can be used.
Variance of Debiased Estimator
DEAL routes external information into the variance calculation rather than a prediction mean. This bypasses the limitation of rectifiers, allowing the method to utilize richer data structures. It treats one-step corrections as noisy observations of an unknown bias vector, enabling adaptation across different estimation regimes.
Enlarged Design Efficiency
By incorporating external models and unlabeled covariates into the variance structure, DEAL effectively enlarges the design used for final debiasing. This enlarged design leads to tighter confidence intervals—up to 50% shorter than standard debiased Lasso—demonstrating superior efficiency gains.
Covariate Shift Robustness
The method remains robust against covariate shift in unlabeled data. A specific modification of the final precision matrix restores asymptotic validity even when the unlabeled data comes from a population different from the target distribution, ensuring nominal coverage.

Terminology

Summary

Debiased External-model-Assisted Lasso (DEAL)

This paper introduces Debiased External-model-Assisted Lasso (DEAL) as a novel methodology designed to overcome significant structural limitations present in Prediction-Powered Inference (PPI) and its variant, PPI++, particularly when applied to high-dimensional semi-supervised linear regression involving external, potentially transfer-learned estimators.

Core Problem Addressed:

The fundamental weakness of existing methods like PPI/PPI++ lies in the use of a prediction rectifier to ensure validity. When both the working model and the predictor are linear, this rectifier exactly cancels the predictor's contribution to the estimation, effectively collapsing both PPI and PPI++ onto Ordinary Least Squares (OLS), irrespective of how good or bad the external estimator is. This mechanism can lead to inflated variance when estimators are near-oracle. p. 0

Where the paper says it

In a correctly specified linear model, however, this rectifier cancels the predictor exactly: PPI and its power-tuned variant PPI++ both reduce to ordinary least squares regardless of predictor quality, and can inflate variance when the predictor is near the oracle regression function. Genuine efficiency must therefore come from a mechanism other than the rectifier.

DEAL's Novel Approach:

DEAL departs from this rectifier-based approach by fundamentally changing how it incorporates the external estimator and unlabeled covariates. Instead of routing these components through a prediction rectifier, DEAL routes them into the variance of a debiased estimator. This is achieved via a bias-aware shrinkage step that treats the one-step correction as a noisy observation of an unknown bias vector, allowing for adaptation across three distinct regimes: target-only Lasso, near-oracle external estimator, and biased-but-informative one. p. 0

Where the paper says it

Rather than rectifying predictions, DEAL routes the external estimator and the unlabeled covariates into the variance of a debiased estimator. A bias-aware shrinkage step treats the one-step correction as a noisy observation of an unknown bias vector and adapts, through a single cross-fitted tuning parameter, across three regimes—target-only Lasso, a near-oracle external estimator, and a biased-but-informative one; pseudo-labels are then imputed on the unlabeled covariates, a stacked Lasso is fitted, and a final debiasing step is applied.

The procedure involves several key stages:

  1. Bias-Aware Initializer: Selecting a shrinkage level based on a data-driven rule.

  2. Pseudo-Label Imputation: Utilizing unlabeled covariates to impute responses for the fitting process.

  3. Stacked Lasso Fitting: Employing a stacked Lasso on enlarged data incorporating the imputed values and external information.

  4. Final Debiasing Step: Applying Javanmard–Montanari precision matrices for final debiasing.

Theoretical Guarantees and Efficiency Gains:

The authors establish coordinate-wise asymptotic normality with an explicit, adaptive variance under sub-Gaussian design assumptions. A crucial theoretical finding is that at a common unlabeled budget, the confidence intervals generated by DEAL are strictly shorter than those of target-only debiased Lasso (DL), and strictly shorter than PPI/PPI++ when using a linear labeler, and under full unlabeled saturation with a non-linear one. The efficiency gain is explicitly drawn from the enlarged design's variance, not solely from the accuracy of the imputed responses.

Key Structural Contrasts:

The paper highlights a critical structural difference: PPI/PPI++ act on the first moment of the prediction residual, which lies within the linear span of X under a linear labeler, making them asymptotically equivalent to DL. In contrast, DEAL routes covariates through the variance of a debiased estimator—a channel that the rectifier cannot reach. p. 0

Where the paper says it

We study high-dimensional semi-supervised linear regression with an external, possibly transfer-learned, initial estimator, and propose the Debiased External-model-Assisted Lasso (DEAL). Rather than rectifying predictions, DEAL routes the external estimator and the unlabeled covariates into the variance of a debiased estimator.

This allows DEAL to behave as if it had drawn a strictly larger labeled sample. p. 1
Where the paper says it

We observe a small labeled target sample (X0, y0), abundant unlabeled target covariates X˜, and an external estimator βˆ ext that may itself be the output of a transfer-learning procedure. The proposed estimator—the Debiased External-model-Assisted Lasso, abbreviated DEAL—departs from the rectifier in two ways.

Empirical Validation:

Empirical validation across five diverse applications (astronomy, materials chemistry, oncology) confirms these theoretical advantages. The empirical findings show that DEAL significantly tightens confidence intervals compared to DL and PPI++, with median CI-length ratios ranging from approximately 0. p. 0

Where the paper says it

In a Monte Carlo study the DEAL intervals run from roughly 0. 87 of the debiased-Lasso length at nominal coverage, and a real-data portfolio of five applications spanning astronomy, materials chemistry, and oncology—the last using a large-language-model oracle—tightens intervals in every case (median ratios 0.

23 to 0.53, whereas PPI/PPI++ do not show material tightening. p. 0
Where the paper says it

87 of the debiased-Lasso length at nominal coverage, and a real-data portfolio of five applications spanning astronomy, materials chemistry, and oncology—the last using a large-language-model oracle—tightens intervals in every case (median ratios 0. 53), whereas debiased Lasso, PPI, and PPI++ do not materially tighten.

The key takeaway is that the efficiency gain stems from the unlabeled design's ability to sharpen the estimated precision matrix, making DEAL largely indifferent to labeler quality and robust against model uncertainty or covariate shift (the one-sided modification of debiasing preserves coverage under shift).

Conclusion:

DEAL provides a more robust and efficient framework for high-dimensional semi-supervised regression. Its success is rooted in drawing efficiency from the unlabeled design's sharpening of the estimated precision matrix, offering superior performance over rectifier-based methods by accessing information channels inaccessible to them.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Bias-aware external-model-assisted inference can be used to extract valid inference from prediction models even when they are imperfect or non-linear, because the channel that the rectifier cannot reach is the variance of a debiased estimator. This allows AI systems to gain genuine efficiency by routing information through design rather than relying on a potentially biased mean.

  2. The system can perform better than current prediction-powered inference (PPI) and PPI++ in high-dimensional settings because the external model and the unlabeled covariates enlarge the design that the final debiasing step inverts. This means an AI system can achieve confidence interval length reductions of up to 50 percent compared to debiased Lasso.

  3. The AI system's performance becomes robust against covariate shift in unlabeled data, as a one-sided modification of the Stage-4 debiasing precision matrix restores asymptotic validity in the shifted regime. This allows for deployment where unlabeled data comes from populations that differ from the target distribution, maintaining nominal coverage.

  4. The system can be adapted to labelers that do not take a linear-coefficient form by using the labeler’s population linear projection on the target distribution and its associated residual as inputs. This capability extends inference to more complex, non-linear conditional means without losing asymptotic normality or CI length dominance over benchmarks like PPI++.

  5. The system can dynamically select the optimal unlabeled sample size based on a variance-balance rule that tracks the bias–variance trade-off. This means the AI can intelligently decide how much unlabeled data to use, maximizing accuracy by stopping when the second-order bias variance reaches a fixed finite level.

Abstract

In high-dimensional semi-supervised linear regression, prediction-powered inference (PPI) corrects an external predictor with a rectifier estimated from the labeled data. In a linear model, however, this rectifier cancels the predictor: PPI and PPI++ reduce to ordinary least squares and can inflate variance when the predictor is close to the oracle. We propose the Debiased External-model-Assisted Lasso (DEAL), which routes the external estimator and the unlabeled covariates into the variance of a debiased estimator, with a bias-aware, cross-fitted shrinkage step that adapts across target-only, near-oracle, and biased-but-informative regimes. We prove coordinate-wise asymptotic normality with an adaptive variance, extend validity to the projection parameter under misspecification and nonlinear labelers, and show that, at a common unlabeled budget, DEAL intervals are shorter than those of debiased Lasso, PPI, and PPI++; a shift-aware variant preserves coverage under covariate shift. In simulations, DEAL intervals are 0.49-0.87 of the debiased-Lasso length, and across six real-data applications spanning astronomy, chemistry, proteomics, and oncology, the last using a large-language-model oracle, they tighten in every case, with median length ratios of 0.23-0.53.

Sources

Related papers