Sensitivity Analysis for Linear Estimators

arXiv:2309.06305 · econ.EM, stat.ME · Submitted 2023-09-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Sensitivity Analysis for Linear Estimators".

Warren: Here's the gist of the paper. Sensitivity analysis for linear estimators proposes a novel framework that measures identification failure by comparing an observed distribution to a hypothetical target distribution that would identify the causal parameter of interest.

Ruth: This approach yields generalized bounds for Average Potential Outcome (APO), Regression Discontinuity (RD), and instrumental variables (IV) exclusion failure designs, consistently estimating sharp bounds under plausible conditions.

Warren: Framework Overview The proposed framework proceeds by defining a target distribution that point-identifies the causal estimand of interest.

Ruth: It then considers a structural model implying restrictions on the divergence between the target population and the observed population.

Paper summary: Ruth: We're digging deeper into how this framework works, specifically looking at its core thesis outlined in "Sensitivity Analysis for Linear Estimators" >

Warren: They are proposing a novel sensitivity analysis framework where you measure identification failure by comparing the observed distribution to a hypothetical target distribution that would point to the causal parameter of interest >

Iris: The authors claim this approach yields generalized bounds for Average Potential Outcome, Regression Discontinuity, and instrumental variables exclusion failure designs >

Ruth: They suggest that under plausible conditions, this method consistently estimates sharp bounds for these designs, even when things are a bit challenging >

Warren: What's important is that their partial identification results extend the APO context to allow even unbounded likelihood ratios, which is a big step in what they can handle >

Iris: They also offer a simple closed form characterization of the identified set, which means you get an explicit way to define what's identifiable, even with those unbounded changes in measure >

Ruth: And they link this work to the literature on sensitivity analysis for IPW estimators, suggesting it's an extension of previous ideas where you start by assuming point identification and then relax those assumptions >

Warren: It seems like they are building a unified way to handle different types of distributional restrictions that cause identification problems across several major causal inference setups >

Conclusion: Ruth: So, looking at this work by Dorn and Yap, "Sensitivity Analysis for Linear Estimators", it seems to be about developing a systematic way to quantify how far an estimator can be from being perfectly identified >

Warren: Essentially, they're giving us a sensitivity analysis that tells us the degree of identification failure by measuring the gap between what we observe and what would identify the causal parameter >

Iris: The implication for me is that it gives us a concrete way to check our models, not just assume they work well under some nice conditions >

Ruth: Right. It moves identification failure from being an abstract problem to something you can actually measure with bounds on the outcome, like Average Potential Outcome or RD designs >

Warren: And for someone who only listens to the show, it means that when you use these common estimators, you can get a bound on how much your estimate might swing based on how far your actual data is from what would be perfectly identifying >

Iris: It's about moving beyond just checking if an estimator works under standard asymptotics and instead giving us a measure of its reliability across different levels of identification failure >

Ruth: That's the gist of it. It’s a new tool for understanding the limits of our statistical tools when we apply them to real-world data on causal questions >

Jacob Dorn, Luther Yap

econ.EM, stat.ME

Submitted: 2023-09-12

Updated: 2026-11-13

Comments: Previously circulated as Sensitivity Analysis for Linear Estimands

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: Sensitivity analysis for linear estimators proposes a novel framework that measures identification failure by comparing an observed distribution to a hypothetical target distribution that would…

Key concepts

Target Distribution
This is a specific distribution designed to point-identify the exact causal parameter you are interested in. The framework compares your actual observed data distribution against this ideal target. The difference between them reveals how much uncertainty or 'failure' exists in identifying the true causal effect.
Radon-Nikodym Derivative
This is a statistical measure that describes how one probability distribution relates to another. In this context, it measures the restriction on the relationship between the target population and your observed population. Analyzing its largest and smallest values helps define sharp boundaries for what can be reliably identified.
Identification Failure Bounds
These are mathematical limits that define the set of possible causal estimates you can reliably calculate. The framework uses restrictions on these derivatives to establish sharp upper bounds, showing how much uncertainty is inherent in estimating parameters like APO or CATE.

Terminology

Summary

Sensitivity analysis for linear estimators proposes a novel framework that measures identification failure by comparing an observed distribution to a hypothetical target distribution that would identify the causal parameter of interest. p. 0

Where the paper says it

Abstract We propose a novel sensitivity analysis framework for linear estimators with identification failures that can be viewed as seeing the wrong outcome distribution. Our approach measures the degree of identification failure through the change in measure between the observed distribution and a hypothetical target distribution that would identify the causal parameter of interest.

This approach yields generalized bounds for Average Potential Outcome (APO), Regression Discontinuity (RD), and instrumental variables (IV) exclusion failure designs, consistently estimating sharp bounds under plausible conditions. p. 0
Where the paper says it

Our approach measures the degree of identification failure through the change in measure between the observed distribution and a hypothetical target distribution that would identify the causal parameter of interest. The framework yields a sensitivity analysis that generalizes existing bounds for Average Potential Outcome (APO), Regression Discontinuity (RD), and instrumental variables (IV) exclusion failure designs.

Framework Overview

The proposed framework proceeds by defining a target distribution that point-identifies the causal estimand of interest. It then considers a structural model implying restrictions on the divergence between the target population and the observed population. p. 0

Where the paper says it

First, we define a target distribution: a synthetic distribution over observed variables that would enable a standard estimator to point-identify the causal estimand of interest. Second, we consider a structural model that implies restrictions on the divergence between the target population and the population the practitioner observes.

The framework leverages statistical and distributionally robust optimization literature to estimate bounds implied by these restrictions, especially when they correspond to a family of restrictions on the Radon-Nikodym derivative. p. 0
Where the paper says it

Third, we leverage work from the literature in statistics and distributionally robust optimization to estimate bounds implied by the restrictions from the second step. The framework is especially powerful when the implied restrictions correspond to a family of restrictions on the Radon-Nikodym derivative (the generalization of likelihood ratio to allow point masses) between the distribution of observables under the target distribution and under the distribution the observed data is drawn from.

The approach typically yields a sensitivity analysis where under the strongest restriction, bounds reduce to the standard point estimate, and under the weakest restriction, they reduce to a worst-case exercise. p. 1
Where the paper says it

Our proposed framework has several empirical advantages. The approach typically yields a sensitivity analysis: under the strongest restriction, the bounds reduce to the standard point estimate; under the weakest restriction, the bounds reduce to a worst-case exercise.

Partial Identification Bounds

The framework characterizes sharp bounds on the identified set using an L∞ restriction on the largest and smallest Radon-Nikodym derivative. The identified set is defined as I(w, w¯) = EQ[λ(R)Y] Q ∈ M(w, w¯). p. 1

Where the paper says it

, 2019; Dorn and Guo, 2023). We extend this literature to obtain a simple closed-form characterization of the identified set, even with unbounded changes in measure.

The sharp upper bound is given by Theorem 1, which states that the sharp upper bound on the identified set satisfies ψ+(w, w¯) = EPObs [λ(R)Y + (λ(R)Y − Q+(R))a(¯w(R), w(R), λ(R)Y)]. p. 5
Where the paper says it

(2) Further, the identified set is convex. The theorem states that the sharp upper bound can be written as a closed-form moment: after the constituent components such as λ(.

Applications and Results

The framework is applied to three novel applications:

  1. Bounds on the APO with treatment selected on an unobserved confounder, nesting unconfoundedness and Manski-type bounds. p. 1

    Where the paper says it

    In our first application, we study bounds on the APO with a treatment that may be selected on an unobserved confounder. Our framework nests unconfoundedness, Manski-type bounds that restrict only the support of the unobserved potential outcomes, and Tan (2006)’s Marginal Sensitivity Model as special cases.

  2. Bounds on causal effects from sharp RD with one-sided manipulation, allowing meaningful bounds on the Conditional Average Treatment Effect (CATE). p. 1

    Where the paper says it

    (2020) propose a worst-case bound on the conditional average treatment effect for nonmanipulators: a Conditional Local Average Treatment Effect (CLATE). We show that a stronger restriction on manipulation choice allows us to obtain meaningful bounds on the more standard conditional average treatment effect (CATE), which to the best of our knowledge has been an open issue in the literature.

  3. Treatment effects with an instrument that fails the exclusion restriction, contributing a sensitivity analysis for a measure of exclusion failure that is unit-free. p. 1

    Where the paper says it

    When the instrument is allowed to affect the outcome directly, we consider estimation of a generic weighted average of local average treatment effects (LATEs) across instrument values. In the continuous outcome case, we contribute a sensitivity analysis for a measure of exclusion failure that is unit-free.

Estimation and Inference

The paper provides theoretical guarantees for the consistency of a plug-in estimator, showing that natural plug-in estimators achieve standard asymptotics under reasonable conditions. The second part of Proposition 6 states that even if the quantile is not estimated correctly, there exists some Tˆ such that Tˆ ≥ Tˆ∗ and Tˆ∗ p→T. p. 6

Where the paper says it

Even if the researcher fails to consistently estimate the additional nuisance parameter Q+, they can still easily estimate valid bounds: Lemma 3. Suppose Assumptions Support and Moments hold and there is a putative quantile function Q¯ such that the expectation of |ϕ +(w, w, r, y ¯ | Q¯)| exists.

Inference uses a standard percentile bootstrap, which achieves coverage at least as large as nominal under standard conditions. p. 1
Where the paper says it

We implement a simple plug-in estimator and percentile bootstrap that leverage a fixed grid of quantile estimates across bootstraps and values of c. We find that a plug-in boostrap approach achieves excellent bias and coverage properties for small c; increasingly over-covers as c grows towards 0.

Simulation Performance

In simulations of the conditional c-dependence environment, the median bound estimates generally track the true bounds. The coverage results show that under nominal coverage, the coverage rate would be 93. p. 1

Where the paper says it

We find that a plug-in boostrap approach achieves excellent bias and coverage properties for small c; increasingly over-covers as c grows towards 0. 1; and nearly achieves nominal coverage of the real line at the most difficult c = 0.

6% to 96.3% with 95% probability. p. 5
Where the paper says it

In particular: Definition 4. The threshold probability is τ (R) = w¯(R)−1 w¯(R)−w(R) .

The bounds with quantile forest estimates remain valid but tend to be more conservative. p. 1
Where the paper says it

The plug-in estimator is consistent for the sharp bounds when given consistent estimates of the primary analysis nuisance functions and a certain conditional outcome quantile function. When the conditional outcome quantile function is inconsistent, the plug-in estimator yields valid bounds that are too wide rather than too narrow.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved sensitivity analysis for linear estimators can quantify identification failure by measuring the change in measure between the observed distribution and a hypothetical target distribution that would identify the causal parameter of interest. This allows for bounds on Average Potential Outcome (APO), Regression Discontinuity (RD), and instrumental variables (IV) exclusion failure designs, even when likelihood ratios are unbounded.

  2. The framework can be implemented to yield sharp and tractable analytic bounds by deriving them as a closed form moment of the observed data under specific distributional restrictions, which simplifies estimation compared to general optimization problems.

  3. The system can estimate valid bounds even when the conditional outcome quantile function is inconsistent by using a plug-in estimator that yields valid bounds that are too wide rather than too narrow.

  4. The framework enables novel results for CATE under RD with manipulation and for IV models with exclusion failure, providing sensitivity analysis where previous literature lacked meaningful quantification.

  5. The system can construct a worst-case distribution Q+ over the structural model space by defining observable functions like V (x, t, y) to maximize the required conditional expectations within M(Lambda, Lambda).

  6. The plug-in estimator for the upper bound can be implemented using linear quantile regression and random forests with three-fold cross validation, allowing for estimation of bounds even when dealing with delicate problems where the identified set becomes unbounded at extreme sensitivity parameters.

  7. The system provides coverage guarantees via a standard percentile bootstrap procedure, ensuring that confidence intervals for the identified set achieve nominal coverage rates under standard conditions and offer robustness even if quantile estimators tend to an inconsistent limit.

Abstract

We propose a framework for building sensitivity analyses for linear estimators. We measure the degree of identification failure by the change of measure between the observed distribution and a hypothetical target distribution under which a standard estimator would identify the parameter of interest. Under L infinity bounds on that derivative, the identified set has closed-form sharp bounds that require no numerical optimization and that are computable and continuous when the derivative is unbounded, nesting Manski-type worst-case bounds and existing characterizations that require bounded changes of measure. The framework nests several selection models for average potential outcomes, yielding the established worst-case bounds of each design. The framework yields a new model family for regression discontinuity (RD) that delivers the first sensitivity analysis on the conditional average treatment effect under manipulation. We show that a bootstrap procedure yields valid confidence intervals. An empirical application shows that a modest restriction on manipulation delivers bounds far more informative than worst-case bounds.

Sources

Related papers