The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "The price of a weak shadow variable".
Jane: Here's the gist of the paper. When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper investigates what inference costs as the relevance of this shadow variable vanishes, finding that this cost depends on whether the tilting variable has an atom at its essential supremum The nature of weak relevance The authors treat the first requirement for a shadow variable, relevance, as a yes-or-no condition and then make it continuous by parametrizing it with a scalar that scales the shadow variable’s entire contribution to the moment conditions They characterize this functional exactly and show that it fixes the effective sample size, which grows like the square root of the logarithm of the sample size, while a thousandfold increase in sample size buys one unit of tilt This characterization is concrete and shows that whether the functional saturates or diverges depends on whether the tilting variable has an atom at its essential supremum The role of identification and effective sample size The second moment of the inverse weight decomposes exactly into a term involving a quadratic divergence of an exponential tilt, which yields a sharp dichotomy where the functional is capped when the tilting variable has an atom at its essential supremum This decomposition determines the effective sample size exactly, which is given by ESS(gamma) = n E(pi0) + 2 pr(R = 0) + pr(R = 0)2Λ(gamma) The effective sample size never exceeds the respondents, ESS ≤ n pr(R = 1), with equality when the weight is degenerate The reach of the scanned set The reach of a scanned confidence set grows like (log n)1/2 and is free of ρ, whereas the set that must be scanned widens without limit as ρ falls The procedure prescribed to reveal nonidentification therefore conceals it by giving no sign of having done so The reach grows like (log n)1/2 and does not depend on ρ, and the functional Λ itself exceeds the largest representable number once logΛ > log(DBL MAX) ≐ 709.78 irrespective of n Diagnostic tools for analysis The paper provides a checklist for analyzing results, including relevance, exclusion, and overlap The effective sample size is the master quantity that governs the practical failures of empirical likelihood and is flat in the identification strength A candidate shadow variable should be placed on the relevance–exclusion plane before it is adopted The factorization of the identifying signal through a scalar at every point of the identified set shows that relevance is a global property, and every quantity downstream can be followed as a function of it The range one must scan grows as relevance falls while the range one can scan does not The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The choice of regularization rate should be adaptive, as the optimal ridge rises as relevance falls Data preparation The data preparation involves several conventions to reproduce the analysis, including recoding durations and age levels These preparations are necessary to reconcile published tables with the analysis performed in the paper The reproduction of these details is essential for applying the diagnostics correctly Simulation and coverage The simulation compares four procedures across different relevance levels and sample sizes, showing that at low relevance, the unregularized representer estimator is worse than doing nothing The Anderson–Rubin set holds its nominal level in every cell, which is what identification-robust means The coverage of the nominal 95 per cent interval remains nominal throughout, while the Wald interval falls as relevance decreases The scanned set is therefore honestly unbounded exactly where the data remain uninformative The measured reach exceeds (log n)1/2 throughout and by a widening margin, indicating that the empirical effective sample size overstates how far the scan can be carried The signature Dufour (1997) describes is visible in the Anderson–Rubin set holding its nominal level while the Wald interval falls as relevance falls Application to credit scoring The application to consumer credit shows that no usable shadow variable existed in either data set, with the most relevant characteristic—the credit score—violating exclusion in almost exactly equal measure The effective sample size tells the same story, provided it is evaluated at the estimate rather than at ignorability Screening all characteristics shows this was not an unlucky choice, as the most relevant characteristic has a ratio of 0.95: relevant and effectively excluded The failure is invisible in a coefficient table and fatal to any inverse-weighted method The diagnostic is reported as a property of that analysis rather than as a correction applied to it The choice of the excluded characteristic by its insignificance in the response model is what a formal selection procedure also does The relevance and exclusion are not in tension by accident, but under an aligned selector they are incompatible The search for a shadow variable should be directed at characteristics the scorecard could not use in the form in which they matter The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The difference between the two cases is whether the analyst is told References Andrews, D.
Tom: W.
Jane: K.
Tom: and Cheng, X.
Paper summary: Lu: We've covered how relevance is treated continuously, how it fixes the effective sample size based on an atom condition, and what this means for practical diagnostics like relevance and exclusion planes. The title of the paper, "The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes," really captures that trade-off between identification and information.
Tom: It shows that when you’re dealing with nonignorable missing data, the effort to restore identification through a shadow variable has a measurable cost, and that cost isn't just theoretical; it depends on whether the variable has an atom at its essential supremum.
Jane: So for someone listening who isn't reading the math, what does this actually change for their day-to-day work in building predictive models? It seems to suggest that simply finding a predictor isn't enough; you have to understand its specific mathematical properties regarding its extreme values.
Lu: It emphasizes that relevance is a global property; every quantity downstream can be followed as a function of it, and the range you must scan grows as relevance falls while the range you actually can scan does not.
Meng: It means we need to stop blindly screening features for identification and start looking at how those features interact with the response model in terms of exclusion and relevance. That’s a necessary shift from just feature engineering to actual identification diagnostics.
Tom: And it points us toward adaptive regularization; the optimal ridge rate should rise as relevance falls, which means our tuning strategy needs to change dynamically depending on how well our shadow variable is actually working.
Jane: So the paper makes it clear that inference under nonignorable missingness isn't a fixed problem; the cost of identification changes based on these underlying structural properties of the variables we use to try and fix it.
Conclusion: Tom: So, we’re wrapping up this deep dive on "The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes." Basically, they found that if you try to fix missing data using a shadow variable—a predictor you exclude from the main calculation—the cost of making that fix depends entirely on how relevant that predictor actually is.
Jane: So, what does "relevance vanishing" mean in plain English? Like, when a variable becomes so weak in predicting the outcome that it barely matters anymore?
Lu: Exactly. The paper treats relevance like a switch—it’s either yes or no—but they make it continuous by scaling how much that shadow variable contributes to the whole picture.
Meng: So, if you want identification, you need this variable to be relevant enough to matter, right? And the cost of that identification scales based on whether this variable has a specific mathematical property called an atom at its essential supremum.
Tom: That’s where it gets technical but super important. They show that when relevance drops low, the effort you put into finding that shadow variable doesn't give you much more information about the data itself; it just gives you a tiny bit of identification strength for a huge cost in terms of how many samples you effectively have.
Jane: That means if we’re trying to build something with missing data, and our best guess for a helpful predictor is weak, we’re essentially paying a high price for that guess.
Lu: Right. The analysis shows that this cost relates directly to the effective sample size, which can grow like the square root of the logarithm of your total sample size, but only if you have enough relevance to start with.
Meng: From a production standpoint, that sounds rough because we want reliable estimates at scale. If identification strength is low and we need more samples just to get a decent result, that’s not scalable.
Tom: The authors point out that the method they use for scanning—the range you have to look in—stays small, around the square root of the log of your sample size, even when relevance falls. So you can’t just keep widening your search area hoping to find something better.
Jane: But what does that tell us about how we should actually approach these missing data problems in AI and data science?
Lu: It suggests that before you start hunting for a shadow variable, you need to check its relevance against the exclusion conditions first; it’s not just about finding *a* predictor, it's about finding one that isn't too weak.
Michael Büler, Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences
Münster School of Business, FH Münster University of Applied Sciences · Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences
stat.ME, math.ST, stat.AP, stat.TH
Submitted: 2026-10-10
Updated: 2026-11-09
Comments: 50 pages, 2 figures. Replication package: https://doi.org/10.5281/zenodo.23262503. v3 adds three antecedent citations, corrects a screen that coded threshold rules linearly, extends the screen to a continuous outcome, and bounds the scorecard coefficients under a bounded departure from MAR. v2 consolidated the work at the moderators' request. Two papers are under separate review
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper…
Key concepts
- Shadow Variable
- A covariate that predicts an outcome but is excluded from the response mechanism. It is used to restore identification when a primary outcome is missing not at random.
- Relevance
- The strength of a shadow variable, treated as a yes-or-no condition initially. The paper treats this relevance continuously by scaling its contribution to moment conditions, showing how it affects inference.
- Effective Sample Size (ESS)
- A measure that governs the practical failures of empirical likelihood methods. It is determined exactly by the tilting variable's properties and never exceeds the total respondents, indicating how much information is truly available for inference.
Terminology
Summary
When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper investigates what inference costs as the relevance of this shadow variable vanishes, finding that this cost depends on whether the tilting variable has an atom at its essential supremum
The nature of weak relevance
The authors treat the first requirement for a shadow variable, relevance, as a yes-or-no condition and then make it continuous by parametrizing it with a scalar that scales the shadow variable’s entire contribution to the moment conditions They characterize this functional exactly and show that it fixes the effective sample size, which grows like the square root of the logarithm of the sample size, while a thousandfold increase in sample size buys one unit of tilt This characterization is concrete and shows that whether the functional saturates or diverges depends on whether the tilting variable has an atom at its essential supremum
The role of identification and effective sample size
The second moment of the inverse weight decomposes exactly into a term involving a quadratic divergence of an exponential tilt, which yields a sharp dichotomy where the functional is capped when the tilting variable has an atom at its essential supremum This decomposition determines the effective sample size exactly, which is given by ESS(gamma) = n E(pi0) + 2 pr(R = 0) + pr(R = 0)2Λ(gamma) The effective sample size never exceeds the respondents, ESS ≤ n pr(R = 1), with equality when the weight is degenerate
The reach of the scanned set
The reach of a scanned confidence set grows like (log n)1/2 and is free of ρ, whereas the set that must be scanned widens without limit as ρ falls The procedure prescribed to reveal nonidentification therefore conceals it by giving no sign of having done so The reach grows like (log n)1/2 and does not depend on ρ, and the functional Λ itself exceeds the largest representable number once logΛ > log(DBL MAX) ≐ 709.78 irrespective of n
Diagnostic tools for analysis
The paper provides a checklist for analyzing results, including relevance, exclusion, and overlap The effective sample size is the master quantity that governs the practical failures of empirical likelihood and is flat in the identification strength A candidate shadow variable should be placed on the relevance–exclusion plane before it is adopted
Conclusion and practical implications
The factorization of the identifying signal through a scalar at every point of the identified set shows that relevance is a global property, and every quantity downstream can be followed as a function of it The range one must scan grows as relevance falls while the range one can scan does not The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The choice of regularization rate should be adaptive, as the optimal ridge rises as relevance falls
Data preparation
The data preparation involves several conventions to reproduce the analysis, including recoding durations and age levels These preparations are necessary to reconcile published tables with the analysis performed in the paper The reproduction of these details is essential for applying the diagnostics correctly
Simulation and coverage
The simulation compares four procedures across different relevance levels and sample sizes, showing that at low relevance, the unregularized representer estimator is worse than doing nothing The Anderson–Rubin set holds its nominal level in every cell, which is what identification-robust means The coverage of the nominal 95 per cent interval remains nominal throughout, while the Wald interval falls as relevance decreases The scanned set is therefore honestly unbounded exactly where the data remain uninformative The measured reach exceeds (log n)1/2 throughout and by a widening margin, indicating that the empirical effective sample size overstates how far the scan can be carried The signature Dufour (1997) describes is visible in the Anderson–Rubin set holding its nominal level while the Wald interval falls as relevance falls
Application to credit scoring
The application to consumer credit shows that no usable shadow variable existed in either data set, with the most relevant characteristic—the credit score—violating exclusion in almost exactly equal measure The effective sample size tells the same story, provided it is evaluated at the estimate rather than at ignorability Screening all characteristics shows this was not an unlucky choice, as the most relevant characteristic has a ratio of 0.95: relevant and effectively excluded The failure is invisible in a coefficient table and fatal to any inverse-weighted method The diagnostic is reported as a property of that analysis rather than as a correction applied to it The choice of the excluded characteristic by its insignificance in the response model is what a formal selection procedure also does The relevance and exclusion are not in tension by accident, but under an aligned selector they are incompatible The search for a shadow variable should be directed at characteristics the scorecard could not use in the form in which they matter The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The difference between the two cases is whether the analyst is told
References
Andrews, D. W. K. and Cheng, X. (2012). Estimation and inference with weak, semi-strong, and strong identification Beppu, K. p. 8 Move along the identified curve, so that |𝛾𝑌
| → ∞ while
𝛾𝑋(𝑠) remains bounded. (i) If 𝑌 is binary with pr(𝑌 = 1) ∈ (0, 1), then
Λ(𝛾) ⟶ 𝐴2
𝐴2
1
, 𝐴𝑘
∶= 𝐸{𝜋0
𝑒
−𝑘𝛾T
𝑋𝑋𝟙(𝑌 = 1)}, (11)
a finite constant: the tilt cannot explode, however strong the assumed departure from ignorability. (2025) give a recent
alternative in which the identifying restriction is a multiplicative non-interaction in the selection
probability rather than a completeness or rank condition on the stacked system. Beppu and
Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when
the outcome is categorical too, but that completeness can fail even in simple models once one of
the two is continuous; for that case they give a sufficient condition that can be checked on the
observed data. (2025) give a recent
alternative in which the identifying restriction is a multiplicative non-interaction in the selection
probability rather than a completeness or rank condition on the stacked system. Beppu and
Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when
the outcome is categorical too, but that completeness can fail even in simple models once one of
the two is continuous; for that case they give a sufficient condition that can be checked on the
observed data. A reanalyzed credit-scoring study shows the cost: a characteristic
an optimizing selector uses cannot satisfy exclusion. Keywords: Effective sample size · Exponential tilt · Identification-robust inference ·
Missing not at random · Nonresponse instrument. The distinction matters for what the sets mean: theirs do not shrink
with the sample, ours do, but only so slowly that Section 4 becomes the binding constraint. Exponential tilting in Caner (2010) names the member of the generalized empirical likelihood
family obtained by choosing the exponential divergence in the estimating criterion, and that
paper studies it under weak instruments. One recent line drops that condition rather than grading it: Chen
et al. (2026) call a shadow variable weak when completeness is not imposed, and obtain partial
identification with sharp bounds. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. (2025) give a recent
alternative in which the identifying restriction is a multiplicative non-interaction in the selection
probability rather than a completeness or rank condition on the stacked system. Beppu and
Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when
the outcome is categorical too, but that completeness can fail even in simple models once one of
the two is continuous; for that case they give a sufficient condition that can be checked on the
observed data. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. The price of a weak shadow variable: inference under
nonignorable missingness as relevance vanishes
Michael Bücker1,2
1Münster School of Business, FH Münster University of Applied Sciences
2
Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied
Sciences
Correspondence: michael. buecker@fh-muenster. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. Under
completeness conditions the model is then point identified, and a range of estimators is available:
semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and
Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al. (2025) give a recent
alternative in which the identifying restriction is a multiplicative non-interaction in the selection
probability rather than a completeness or rank condition on the stacked system. Beppu and
Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when
the outcome is categorical too, but that completeness can fail even in simple models once one of
the two is continuous; for that case they give a sufficient condition that can be checked on the
observed data.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive regularization based on relevance is possible by using a ridge penalty lambda star derived from an optimization over relevance and sample size, as shown in Equation 21. This allows for tuning lambda to balance the bias against variance, leading to a rule that is transportable across different designs and performs better than unregularized methods.
-
The system can diagnose infeasibility of empirical likelihood problems by checking the effective sample size ESS/n and the largest inverse weight, as detailed in Equation 25. This provides a diagnostic about whether the data do identify what it corrects for, separating issues related to identification strength from those related to overlap.
-
Inference procedures can be made robust against weak relevance by requiring that a candidate shadow variable has high relevance-to-exclusion violation ratio, as suggested by Table 6 in Section 9.2. This requires the characteristic's entire contribution to the outcome to lie outside the class optimized by the selector, rather than just being partially visible to it.
Abstract
When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism. The literature treats the first requirement, relevance, as a yes-or-no condition. We make it continuous, parametrizing it by a scalar that scales the shadow variable's entire contribution to the moment conditions, and ask what inference costs as that scalar vanishes. One functional of the inverse weight governs the answer. We characterize it exactly, show that it fixes the effective sample size, and use it to sharpen the rates of the estimator that avoids the weight. The characterization is concrete. Whether the functional saturates or diverges turns not on boundedness of the outcome but on whether the tilting variable has an atom at its essential supremum: a binary outcome saturates, a uniform one does not. It also bounds how far a scanned set reaches, and that reach grows only as the square root of the logarithm of the sample size, so a thousandfold increase buys one unit of tilt while the set to be scanned widens. A reanalyzed credit-scoring study shows the cost: a characteristic an optimizing selector uses cannot satisfy exclusion.
Sources
- AI-Generated Measurements for Identification and Inference with Missing Data: A Weak Shadow Variable Approach
- Instrument, Variable and Model Selection with Nonignorable Nonresponse
- Estimation with missing not at random binary outcomes via exponential tilts
- The Illusion of Improvement: Reject Inference Strategies in Credit Scoring
- Confidence Sets under Weak Identification: Theory and Practice
- A Multiplicative Instrumental Variable Model for Data Missing Not-at-Random
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes