Inference with weights: Residualization produces short, valid intervals for varying estimands and varying resampling processes

arXiv:2507.19607 · stat.ME · Submitted 2025-07-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Inference with weights".

Jane: Here's the gist of the paper. Weighting procedures are used in observational causal inference to adjust for covariate imbalance within the sample.

Tom: The authors examine and recommend the use of robust standard errors from a weighted regression that additionally includes the balancing covariates and their interactions with treatment.

Jane: They show that these standard errors are more precise and asymmetrically correct for weights that achieve exact balance under multiple common resampling frameworks, including design-based and model-based inference, as well as superpopulation sampling with a finite sample correction.

Tom: Gains to precision can be quite significant when the balancing weights adjust for prognostic covariates.

Paper summary: Tom: So we’re diving deeper into what exactly they are proposing in "Inference with weights: Residualization produces short, valid intervals for varying estimands and varying resampling processes." The authors start by pointing out that common practice is to estimate robust standard errors from a weighted regression of the outcome on the treatment.

Jane: But they immediately flag that this common practice can inflate variance estimates, leading to standard errors and confidence intervals that are overly conservative. They want us to look at a different way of doing things.

Lu: The central thesis is recommending the use of robust standard errors from a weighted regression that does not just use the weights but also incorporates the balancing covariates and their interactions with the treatment. This is what they call residualized standard errors, which they describe as "linearized" in survey literature.

Meng: So, what's the motivation for this specific regression setup? Why include those interactions with the treatment if we're already adjusting for imbalance?

Tom: They argue that this specific structure helps isolate the variation in the outcome that actually influences our point estimate, which is what they call residualization.

Jane: So, it’s like they are trying to get rid of all that noise from the covariates by focusing only on what truly matters for the effect estimate.

Lu: They show that this approach can be adapted to get variance estimates for both the average treatment effect on all units or ATE, and they even showed how these can be adapted to obtain variance estimates for the average treatment effect on the treated or ATT.

Meng: That means this isn't just a tool for one specific type of effect; it has versatility across different ways we think about causal effects.

Tom: And they show that these standard errors are smaller, but they still maintain nominal or even higher coverage under multiple common resampling frameworks, including design-based and model-based inference.

Jane: That’s a key part—the precision gain is there, but the intervals aren't just narrower; they stay correct across different ways you might be doing your inference.

Lu: The paper details how the required variance computation shifts depending on whether you are operating under a design-based or model-based setting, and that’s important because it shows the method isn't universally applied without modification.

Meng: It sounds like the method has to change its internal logic based on whether we're thinking about survey design or a statistical model.

Tom: And they also extend this for superpopulation inference by adding a correction to account for uncertainty in those covariate means due to factors like Berk et al. (two thousand thirteen) and Negi and Wooldridge (two thousand twenty-one), and Ding (two thousand twenty-three) <ref:2507.19607#pg1>.

Jane: So, the paper covers a lot of ground, moving from the basic idea of correcting inflated variance to showing how this method scales up for superpopulation sampling too.

Conclusion: Tom: We’re wrapping up this discussion on "Inference with weights: Residualization produces short, valid intervals for varying estimands and varying resampling processes." The authors argue that residualization is appropriate regardless of the origin of the weights, whether they're exact balancing, approximate balancing, or even Inverse Propensity Score Weighting.

Jane: And it works for a wide variety of targets too—things like ATT versus ATE—and whether you are using a design-based or model-based framework.

Lu: The big implication is that this approach is familiar and relatively easy to implement, but the authors admit there's still incomplete consensus about the underlying theoretical motivations for these choices.

Meng: So, while the empirical evidence shows significant gains in precision—reductions of ten to forty-five percent in estimated standard errors—the theoretical justification for choosing residualization over other methods is still being debated.

Tom: That’s right. The results are strong and the method performs well across simulations and applications, but the underlying theory isn't settled yet.

Jane: So, in simple terms, it means you can get much tighter estimates for your causal effects if you use this residualized standard error approach.

Lu: But remember what they say about the limitations—the method doesn't solve every theoretical question about weight uncertainty and there are still some open questions out there.

Meng: So, the practical lesson here is that it’s a strong tool to try in your modeling pipeline if you need those precision gains, but be ready for more theoretical discussion later.

Tom: Exactly. The paper demonstrates that the performance depends on how strongly the covariates relate to the outcome, as studies with higher R squared Y on X tend to have higher reductions in standard errors through residualization.

Jane: So while it’s a powerful tool for getting better numbers, we need to keep an eye out for those lingering theoretical debates as researchers continue to use it.

UC Berkeley · UCLA

stat.ME

Submitted: 2025-07-25

Updated: 2026-11-12

Project page: https://ngreifer.github.io/WeightIt/articles/estimating-effects.html

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 99/100

The gist: Weighting procedures are used in observational causal inference to adjust for covariate imbalance within the sample.

Key concepts

Weighting Procedures
These are methods used to adjust for imbalances between groups in observational studies, often by assigning weights to observations. While useful for balancing covariates, standard weighting methods can inflate variance estimates, leading to overly wide confidence intervals.
Residualization
This is a proposed method of calculating standard errors that involves using weights in a specific weighted regression. The goal is to isolate the variation in the outcome (Y) that is conditionally dependent on the covariates (X), which are then balanced. This focuses on the variation truly influenced by treatment.
Design-Based vs. Model-Based Inference
This refers to two different philosophical ways of performing statistical inference. Design-based methods rely on the study's original design, while model-based methods use a statistical model to estimate parameters. The paper shows that the best residualization technique depends on which framework is being used.
SATE and ATT
These are specific types of causal effect estimates: SATE (Single Treatment Effect) and ATT (Average Treatment Effect). The paper examines how the required balancing procedure and the appropriate variance estimation method change depending on which specific causal quantity you are trying to estimate.

Terminology

Summary

Weighting procedures are used in observational causal inference to adjust for covariate imbalance within the sample. p. 0

Where the paper says it

Inference with weights: Residualization produces short, valid intervals for varying estimands and varying resampling processes Erin Hartman∗ Chad Hazlett Arisa Sadeghpour UC Berkeley UCLA UC Berkeley July 29, 2025 Abstract Weighting procedures are used in observational causal inference to adjust for covariate imbalance within the sample. Common practice for inference is to estimate robust standard errors from a weighted regression of outcome on treatment.

The authors examine and recommend the use of robust standard errors from a weighted regression that additionally includes the balancing covariates and their interactions with treatment. p. 0
Where the paper says it

However, it is well known that weighting can inflate variance estimates, sometimes significantly, leading to standard errors and confidence intervals that are overly conservative. We instead examine and recommend the use of robust standard errors from a weighted regression that additionally includes the balancing covariates and their interactions with treatment.

They show that these standard errors are more precise and asymmetrically correct for weights that achieve exact balance under multiple common resampling frameworks, including design-based and model-based inference, as well as superpopulation sampling with a finite sample correction. p. 0
Where the paper says it

We instead examine and recommend the use of robust standard errors from a weighted regression that additionally includes the balancing covariates and their interactions with treatment. We show that these standard errors are more precise and asymptotically correct for weights that achieve exact balance under multiple common resampling frameworks, including design-based and model-based inference, as well as superpopulation sampling with a finite sample correction.

Gains to precision can be quite significant when the balancing weights adjust for prognostic covariates. p. 0
Where the paper says it

We show that these standard errors are more precise and asymptotically correct for weights that achieve exact balance under multiple common resampling frameworks, including design-based and model-based inference, as well as superpopulation sampling with a finite sample correction. Gains to precision can be quite significant when the balancing weights adjust for prognostic covariates.

The Core Problem Addressed

Weighting procedures in observational causal inference are popular for covariate adjustment, but standard methods often inflate variance estimates, leading to overly conservative standard errors and confidence intervals. Investigators worry about incorporating uncertainty in the weights themselves and whether the chosen standard error is appropriate for sample- or population-based estimands under different resampling frameworks. p. 1

Where the paper says it

Second, weighting can inflate variance estimates relative to the unweighted sample, sometimes significantly, and investigators may worry whether this is appropriate or problematic. Third, such a standard error may seem not to recognize special properties achieved by the weights, such as the degree to which differences between groups have already been minimized by the weights.

The paper clarifies this question by considering both design-based and model-based philosophies, as well as questions of inference for sample- and population-based target estimands. p. 1
Where the paper says it

We first find that the question itself is ill-posed without clearly defining the types of uncertainty the investigator is hoping to account for in their variance estimate. We thus consider both the design-based and model-based philosophies, as well as questions of inference for sample- and population-based target estimands.

The Proposed Solution: Residualization

The authors propose using residualized standard errors, which are described as linearized standard errors in the survey literature, to address the uncertainty in variance estimation. This approach involves employing weights in a weighted “Lin-style” (Lin, 2013) regression of Y on treatment, the centered covariates balanced upon (ϕ˜(X)), and the interaction of these. p. 8

Where the paper says it

We offer here a brief summary of our proposal, with details below. Our proposal is to employ the weights in a weighted “Lin-style” (Lin, 2013) regression of Y on treatment, the centered covariates balanced upon (ϕ˜(X)), and the interaction of these, then use a heteroskedasticity-robust (e.

The central concept motivating this choice is the use of “residualization” to isolate the variation in Y conditionally on X, as only this variation influences the point estimate. p. 9
Where the paper says it

Residualization. The central concept motivating this choice is the use of “residualization” to isolate the variaion in Y conditionally on X, as only this variation influences the point estimate.

Framework Dependence and Estimand Specificity

The appropriate residualized variance estimation method depends on whether one is operating under a design-based or model-based framework. The paper details how the required variance computation changes between these settings, such as using the conditional variance V[ˆτ X, Z] in the model-based setting versus relying on residual variation in the design-based setting. p. 8

Where the paper says it

Similarly, in the model-based perspective, we can imagine a first-stage random sampling of {Zi , Xi , ϵi} from the superpopulation, then a second stage redrawing or marginalization over ϵi within the selected sample. Again, for estimation of variance in this setting we will effectively start with the ATE or ATT variance and then consider additional variation due to having drawn Xi and Zi .

Furthermore, it distinguishes between different target estimands like ATE and ATT and how this affects the required balancing procedure. p. 9
Where the paper says it

We recommend this approach regardless of the origins of the weights (exact balancing, approximate balancing, or IPW), across a variety of estimands (e. ATE), and whether operating under a design- or model-based framework, though we note required adjustments when considering inference for (super)population vs.

Empirical Validation and Findings

Simulation evidence demonstrates that the residualized standard error is smaller, but still conservative or correct under each resampling framework, producing nominal or higher coverage. Gains to precision compared to nonresidualizing estimators can be quite significant, showing reductions of 10-30% in estimated standard errors. p. 2

Where the paper says it

Gains to precision compared to nonresidualizing estimators (e. variance estimators built directly on the weighted difference in means or weighted outcome regression without covariates) can be quite significant, especially when the weights adjust for prognostic covariates.

Empirical applications across three studies show these gains, with the residualized standard error being the most precise estimator that achieves near nominal coverage. p. 2
Where the paper says it

We show that these standard errors are smaller, but still conservative or correct under each resampling framework, producing nominal or higher coverage. Gains to precision compared to nonresidualizing estimators (e.

Conclusion and Future Directions

The authors conclude that residualization is appropriate regardless of the origins of the weights (exact balancing, approximate balancing, or IPW), across a variety of estimands (e.g., ATT vs. ATE), and whether operating under a design- or model-based framework. p. 9

Where the paper says it

We recommend this approach regardless of the origins of the weights (exact balancing, approximate balancing, or IPW), across a variety of estimands (e. ATE), and whether operating under a design- or model-based framework, though we note required adjustments when considering inference for (super)population vs.

While this approach is familiar and easy to implement, there remains incomplete consensus regarding the theoretical motivations for these choices. The analysis suggests that M-estimation and similar approaches intended to incorporate uncertainty in the weights are not necessary for the (S)ATE and (S)ATT. p. 8
Where the paper says it

Again, for estimation of variance in this setting we will effectively start with the ATE or ATT variance and then consider additional variation due to having drawn Xi and Zi . The estimands of interest in the superpopulation, which we denote the “population” estimands, are: τP AT E = E[Y (1) − Y (0)] τP AT T = E[Y (1) − Y (0) | Z = 1] 4 Quantifying sampling uncertainty Here we develop our proposal and justify the use of a weighted linear model for variance estimation.

The results can be adjusted to account for uncertainty over p(X, Z) in an unseen superpopulation using a specific correction. p. 2
Where the paper says it

With adjustments to the weights and form of the estimator, these can be adapted to obtain variance estimates for the average treatment effect on the treated (ATT). These can further be extended for “superpopulation” inference using an adjustment to account for uncertainty in the covariate means due to Berk et al.

The theoretical expectations regarding these estimators are born out in simulations and demonstrated by application. p. 2
Where the paper says it

variance estimators built directly on the weighted difference in means or weighted outcome regression without covariates) can be quite significant, especially when the weights adjust for prognostic covariates. Our empirical application and simulations demonstrate reductions of 10-30% in the estimated standard errors, bringing them closer to empirical standard errors.

Simulation Evidence

The paper replicates simulations under three common paradigms—design-based, model-based, and superpopulation—and evaluates four standard error estimators: Neyman, Pooled, Separate, and Separate + M-estimation. The results show that the residualized estimator is the most precise estimator that achieves near nominal coverage for both SATE and ATT under design-based settings. p. 0

Where the paper says it

We instead examine and recommend the use of robust standard errors from a weighted regression that additionally includes the balancing covariates and their interactions with treatment. We show that these standard errors are more precise and asymptotically correct for weights that achieve exact balance under multiple common resampling frameworks, including design-based and model-based inference, as well as superpopulation sampling with a finite sample correction.

For superpopulation quantities, the Separate + Correction approach yields standard errors considerably larger than uncorrected ones. p. 0
Where the paper says it

We instead examine and recommend the use of robust standard errors from a weighted regression that additionally includes the balancing covariates and their interactions with treatment. We show that these standard errors are more precise and asymptotically correct for weights that achieve exact balance under multiple common resampling frameworks, including design-based and model-based inference, as well as superpopulation sampling with a finite sample correction.

Empirical Applications

The precision gains are demonstrated using data from three empirical studies, including Ladd and Lenz (2009), Chan et al. (2016), and Black and Owens (2016). The results show that the residualized standard error is significantly smaller than the commonly used unresidualized estimator, with reductions on the order of 10-45% in estimated standard error depending on the data-generating process. The performance depends on the strength of the relationship of covariates with the outcome, as studies with higher R2 Y ∼X tend to have higher reductions in standard errors through residualization. p. 2

Where the paper says it

variance estimators built directly on the weighted difference in means or weighted outcome regression without covariates) can be quite significant, especially when the weights adjust for prognostic covariates. Our empirical application and simulations demonstrate reductions of 10-30% in the estimated standard errors, bringing them closer to empirical standard errors.

Diagnostics and Weighting Effects

The paper includes diagnostics showing how covariates contribute most to the reduction in standard errors through residualizing, revealing which covariates are more prognostic. For Inverse Propensity Score Weighting (IPW), the balance achieved is best for Chan et al. p. 4

Where the paper says it

In our discussion below, this has the important implication that IPW thus falls into the category of approximate weighting approaches (those that does not produce exact finite-sample covariate balance), for which our motivating logic differs slightly from the case of exact balancing weights. Matching methods Matching methods, in which treatment and control groups are adjusted to render treatment orthogonal to measured covariates in the observed sample by finding ”matches” (usually a type of nearest neighbor) for each unit, are a related approach to inverse propensity score weighting (Stuart, 2010).

(2016) and the noncontender subgroup analysis of Black and Owens (2016). p. 0
Where the paper says it

For procedures that balance only approximately or in expectation, such as inverse propensity weighting or approximate balancing weights, our proposed method improves precision by reducing residuals through augmentation with the parametric model. We demonstrate our approach through simulation and re-analysis of multiple empirical studies.

The point estimates can change between unadjusted and adjusted regressions when using IPW because IPW does not exactly balance, but controlling for highly prognostic covariates cleans up this residual imbalance. p. 4
Where the paper says it

In our discussion below, this has the important implication that IPW thus falls into the category of approximate weighting approaches (those that does not produce exact finite-sample covariate balance), for which our motivating logic differs slightly from the case of exact balancing weights. Matching methods Matching methods, in which treatment and control groups are adjusted to render treatment orthogonal to measured covariates in the observed sample by finding ”matches” (usually a type of nearest neighbor) for each unit, are a related approach to inverse propensity score weighting (Stuart, 2010).

The superpopulation corrected standard errors are considerably larger than the uncorrected, residualized standard errors. p. 2
Where the paper says it

estimate are well-known as “linearized” standard errors in the survey literature. We refer to them as “residualized” standard errors, which provides a more general and intuitively descriptive name.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Residualized standard errors can be estimated using a weighted linear model including interaction terms, which effectively weakens the specification assumption to allow Y (1) and Y (0) to be “separately” (and differently) linear in phi(X). This allows for variance estimation that appropriately takes credit for the stabilizing impact of the weights.

  2. The proposed method is applicable across various weighting procedures, including exact balancing weights, approximate balancing weights, or any other weighting procedure, and is recommended regardless of the origin of the weights.

  3. The system can estimate superpopulation average treatment effect by adding a correction to robust standard errors using Equation (7), which accounts for uncertainty in the covariate means due to sampling variability.

  4. The AI system can provide coverage rates that are nominal or conservative under residualization with separate models, demonstrating that this approach is smaller, but still conservative or correct under each resampling framework.

  5. The system can be configured to utilize different estimation methods like Neyman, Pooled, Separate, and Separate plus M-estimation standard errors to compare performance across design-based and model-based frameworks.

Abstract

Weighting is widely used in observational causal inference, yet basic questions remain about how to estimate the variance of weighted effect estimates. For example, when and how should sampling uncertainty in the weights be propagated into the final result? Should these weights reduce rather than inflate variance? Does the choice of weighted variance estimator depend on the target estimand and the assumed source of randomness? We show that a single, familiar estimator addresses these concerns across settings: robust standard errors from a weighted regression of the outcome on treatment, the centered balancing covariates, and their interaction with treatment. These residualized standard errors are valid and often substantially smaller than those from weighted regression on treatment alone, with reductions of up to 30% when covariates are prognostic. We establish their justification under design-based, model-based, and superpopulation frameworks, for both ATE and ATT estimands, and for exact balancing, approximate balancing, and inverse propensity weights. For inference to the superpopulation, we employ a simple correction. We further show that M-estimation yields nearly identical results when constructed to answer the same questions, and that the key issue is not "uncertainty in the weights" given a sample, but uncertainty over the sample. Simulations and three empirical applications confirm our analyses.

Sources

Related papers