The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes

summary

The gist

When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper…

In short

The paper investigates inference costs when a missing data variable's relevance weakens and is replaced by a shadow variable. It shows this cost depends on whether the tilting variable has an atom at its essential supremum, determining the effective sample size and the reach of scanned sets.

Key concepts

Shadow Variable
A covariate that predicts an outcome but is excluded from the response mechanism. It is used to restore identification when a primary outcome is missing not at random.
Relevance
The strength of a shadow variable, treated as a yes-or-no condition initially. The paper treats this relevance continuously by scaling its contribution to moment conditions, showing how it affects inference.
Effective Sample Size (ESS)
A measure that governs the practical failures of empirical likelihood methods. It is determined exactly by the tilting variable's properties and never exceeds the total respondents, indicating how much information is truly available for inference.

Terminology used across episodes

This episode discusses

The paper

The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes · Read on arXiv

Michael Büler, Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences

Münster School of Business, FH Münster University of Applied Sciences · Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences

When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism. The literature treats the first requirement, relevance, as a yes-or-no condition. We make it continuous, parametrizing it by a scalar that scales the shadow variable's entire contribution to the moment conditions, and ask what inference costs as that scalar vanishes. One functional of the inverse weight governs the answer. We characterize it exactly, show that it fixes the effective sample size, and use it to sharpen the rates of the estimator that avoids the weight. The characterization is concrete. Whether the functional saturates or diverges turns not on boundedness of the outcome but on whether the tilting variable has an atom at its essential supremum: a binary outcome saturates, a uniform one does not. It also bounds how far a scanned set reaches, and that reach grows only as the square root of the logarithm of the sample size, so a thousandfold increase buys one unit of tilt while the set to be scanned widens. A reanalyzed credit-scoring study shows the cost: a characteristic an optimizing selector uses cannot satisfy exclusion.

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "The price of a weak shadow variable".

Jane: Here's the gist of the paper. When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper investigates what inference costs as the relevance of this shadow variable vanishes, finding that this cost depends on whether the tilting variable has an atom at its essential supremum The nature of weak relevance The authors treat the first requirement for a shadow variable, relevance, as a yes-or-no condition and then make it continuous by parametrizing it with a scalar that scales the shadow variable’s entire contribution to the moment conditions They characterize this functional exactly and show that it fixes the effective sample size, which grows like the square root of the logarithm of the sample size, while a thousandfold increase in sample size buys one unit of tilt This characterization is concrete and shows that whether the functional saturates or diverges depends on whether the tilting variable has an atom at its essential supremum The role of identification and effective sample size The second moment of the inverse weight decomposes exactly into a term involving a quadratic divergence of an exponential tilt, which yields a sharp dichotomy where the functional is capped when the tilting variable has an atom at its essential supremum This decomposition determines the effective sample size exactly, which is given by ESS(gamma) = n E(pi0) + 2 pr(R = 0) + pr(R = 0)2Λ(gamma) The effective sample size never exceeds the respondents, ESS ≤ n pr(R = 1), with equality when the weight is degenerate The reach of the scanned set The reach of a scanned confidence set grows like (log n)1/2 and is free of ρ, whereas the set that must be scanned widens without limit as ρ falls The procedure prescribed to reveal nonidentification therefore conceals it by giving no sign of having done so The reach grows like (log n)1/2 and does not depend on ρ, and the functional Λ itself exceeds the largest representable number once logΛ > log(DBL MAX) ≐ 709.78 irrespective of n Diagnostic tools for analysis The paper provides a checklist for analyzing results, including relevance, exclusion, and overlap The effective sample size is the master quantity that governs the practical failures of empirical likelihood and is flat in the identification strength A candidate shadow variable should be placed on the relevance–exclusion plane before it is adopted The factorization of the identifying signal through a scalar at every point of the identified set shows that relevance is a global property, and every quantity downstream can be followed as a function of it The range one must scan grows as relevance falls while the range one can scan does not The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The choice of regularization rate should be adaptive, as the optimal ridge rises as relevance falls Data preparation The data preparation involves several conventions to reproduce the analysis, including recoding durations and age levels These preparations are necessary to reconcile published tables with the analysis performed in the paper The reproduction of these details is essential for applying the diagnostics correctly Simulation and coverage The simulation compares four procedures across different relevance levels and sample sizes, showing that at low relevance, the unregularized representer estimator is worse than doing nothing The Anderson–Rubin set holds its nominal level in every cell, which is what identification-robust means The coverage of the nominal 95 per cent interval remains nominal throughout, while the Wald interval falls as relevance decreases The scanned set is therefore honestly unbounded exactly where the data remain uninformative The measured reach exceeds (log n)1/2 throughout and by a widening margin, indicating that the empirical effective sample size overstates how far the scan can be carried The signature Dufour (1997) describes is visible in the Anderson–Rubin set holding its nominal level while the Wald interval falls as relevance falls Application to credit scoring The application to consumer credit shows that no usable shadow variable existed in either data set, with the most relevant characteristic—the credit score—violating exclusion in almost exactly equal measure The effective sample size tells the same story, provided it is evaluated at the estimate rather than at ignorability Screening all characteristics shows this was not an unlucky choice, as the most relevant characteristic has a ratio of 0.95: relevant and effectively excluded The failure is invisible in a coefficient table and fatal to any inverse-weighted method The diagnostic is reported as a property of that analysis rather than as a correction applied to it The choice of the excluded characteristic by its insignificance in the response model is what a formal selection procedure also does The relevance and exclusion are not in tension by accident, but under an aligned selector they are incompatible The search for a shadow variable should be directed at characteristics the scorecard could not use in the form in which they matter The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The difference between the two cases is whether the analyst is told References Andrews, D.

Tom: W.

Jane: K.

Tom: and Cheng, X.

Paper summary: Lu: We've covered how relevance is treated continuously, how it fixes the effective sample size based on an atom condition, and what this means for practical diagnostics like relevance and exclusion planes. The title of the paper, "The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes," really captures that trade-off between identification and information.

Tom: It shows that when you’re dealing with nonignorable missing data, the effort to restore identification through a shadow variable has a measurable cost, and that cost isn't just theoretical; it depends on whether the variable has an atom at its essential supremum.

Jane: So for someone listening who isn't reading the math, what does this actually change for their day-to-day work in building predictive models? It seems to suggest that simply finding a predictor isn't enough; you have to understand its specific mathematical properties regarding its extreme values.

Lu: It emphasizes that relevance is a global property; every quantity downstream can be followed as a function of it, and the range you must scan grows as relevance falls while the range you actually can scan does not.

Meng: It means we need to stop blindly screening features for identification and start looking at how those features interact with the response model in terms of exclusion and relevance. That’s a necessary shift from just feature engineering to actual identification diagnostics.

Tom: And it points us toward adaptive regularization; the optimal ridge rate should rise as relevance falls, which means our tuning strategy needs to change dynamically depending on how well our shadow variable is actually working.

Jane: So the paper makes it clear that inference under nonignorable missingness isn't a fixed problem; the cost of identification changes based on these underlying structural properties of the variables we use to try and fix it.

Conclusion: Tom: So, we’re wrapping up this deep dive on "The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes." Basically, they found that if you try to fix missing data using a shadow variable—a predictor you exclude from the main calculation—the cost of making that fix depends entirely on how relevant that predictor actually is.

Jane: So, what does "relevance vanishing" mean in plain English? Like, when a variable becomes so weak in predicting the outcome that it barely matters anymore?

Lu: Exactly. The paper treats relevance like a switch—it’s either yes or no—but they make it continuous by scaling how much that shadow variable contributes to the whole picture.

Meng: So, if you want identification, you need this variable to be relevant enough to matter, right? And the cost of that identification scales based on whether this variable has a specific mathematical property called an atom at its essential supremum.

Tom: That’s where it gets technical but super important. They show that when relevance drops low, the effort you put into finding that shadow variable doesn't give you much more information about the data itself; it just gives you a tiny bit of identification strength for a huge cost in terms of how many samples you effectively have.

Jane: That means if we’re trying to build something with missing data, and our best guess for a helpful predictor is weak, we’re essentially paying a high price for that guess.

Lu: Right. The analysis shows that this cost relates directly to the effective sample size, which can grow like the square root of the logarithm of your total sample size, but only if you have enough relevance to start with.

Meng: From a production standpoint, that sounds rough because we want reliable estimates at scale. If identification strength is low and we need more samples just to get a decent result, that’s not scalable.

Tom: The authors point out that the method they use for scanning—the range you have to look in—stays small, around the square root of the log of your sample size, even when relevance falls. So you can’t just keep widening your search area hoping to find something better.

Jane: But what does that tell us about how we should actually approach these missing data problems in AI and data science?

Lu: It suggests that before you start hunting for a shadow variable, you need to check its relevance against the exclusion conditions first; it’s not just about finding *a* predictor, it's about finding one that isn't too weak.

More episodes

← Home