The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes

arXiv:2610.14695 · stat.ME, math.ST, stat.AP, stat.TH · Submitted 2026-10-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "The price of a weak shadow variable".

Jane: Here's the gist of the paper. When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper investigates what inference costs as the relevance of this shadow variable vanishes, finding that this cost depends on whether the tilting variable has an atom at its essential supremum The nature of weak relevance The authors treat the first requirement for a shadow variable, relevance, as a yes-or-no condition and then make it continuous by parametrizing it with a scalar that scales the shadow variable’s entire contribution to the moment conditions They characterize this functional exactly and show that it fixes the effective sample size, which grows like the square root of the logarithm of the sample size, while a thousandfold increase in sample size buys one unit of tilt This characterization is concrete and shows that whether the functional saturates or diverges depends on whether the tilting variable has an atom at its essential supremum The role of identification and effective sample size The second moment of the inverse weight decomposes exactly into a term involving a quadratic divergence of an exponential tilt, which yields a sharp dichotomy where the functional is capped when the tilting variable has an atom at its essential supremum This decomposition determines the effective sample size exactly, which is given by ESS(gamma) = n E(pi0) + 2 pr(R = 0) + pr(R = 0)2Λ(gamma) The effective sample size never exceeds the respondents, ESS ≤ n pr(R = 1), with equality when the weight is degenerate The reach of the scanned set The reach of a scanned confidence set grows like (log n)1/2 and is free of ρ, whereas the set that must be scanned widens without limit as ρ falls The procedure prescribed to reveal nonidentification therefore conceals it by giving no sign of having done so The reach grows like (log n)1/2 and does not depend on ρ, and the functional Λ itself exceeds the largest representable number once logΛ > log(DBL MAX) ≐ 709.78 irrespective of n Diagnostic tools for analysis The paper provides a checklist for analyzing results, including relevance, exclusion, and overlap The effective sample size is the master quantity that governs the practical failures of empirical likelihood and is flat in the identification strength A candidate shadow variable should be placed on the relevance–exclusion plane before it is adopted The factorization of the identifying signal through a scalar at every point of the identified set shows that relevance is a global property, and every quantity downstream can be followed as a function of it The range one must scan grows as relevance falls while the range one can scan does not The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The choice of regularization rate should be adaptive, as the optimal ridge rises as relevance falls Data preparation The data preparation involves several conventions to reproduce the analysis, including recoding durations and age levels These preparations are necessary to reconcile published tables with the analysis performed in the paper The reproduction of these details is essential for applying the diagnostics correctly Simulation and coverage The simulation compares four procedures across different relevance levels and sample sizes, showing that at low relevance, the unregularized representer estimator is worse than doing nothing The Anderson–Rubin set holds its nominal level in every cell, which is what identification-robust means The coverage of the nominal 95 per cent interval remains nominal throughout, while the Wald interval falls as relevance decreases The scanned set is therefore honestly unbounded exactly where the data remain uninformative The measured reach exceeds (log n)1/2 throughout and by a widening margin, indicating that the empirical effective sample size overstates how far the scan can be carried The signature Dufour (1997) describes is visible in the Anderson–Rubin set holding its nominal level while the Wald interval falls as relevance falls Application to credit scoring The application to consumer credit shows that no usable shadow variable existed in either data set, with the most relevant characteristic—the credit score—violating exclusion in almost exactly equal measure The effective sample size tells the same story, provided it is evaluated at the estimate rather than at ignorability Screening all characteristics shows this was not an unlucky choice, as the most relevant characteristic has a ratio of 0.95: relevant and effectively excluded The failure is invisible in a coefficient table and fatal to any inverse-weighted method The diagnostic is reported as a property of that analysis rather than as a correction applied to it The choice of the excluded characteristic by its insignificance in the response model is what a formal selection procedure also does The relevance and exclusion are not in tension by accident, but under an aligned selector they are incompatible The search for a shadow variable should be directed at characteristics the scorecard could not use in the form in which they matter The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The difference between the two cases is whether the analyst is told References Andrews, D.

Tom: W.

Jane: K.

Tom: and Cheng, X.

Paper summary: Lu: We've covered how relevance is treated continuously, how it fixes the effective sample size based on an atom condition, and what this means for practical diagnostics like relevance and exclusion planes. The title of the paper, "The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes," really captures that trade-off between identification and information.

Tom: It shows that when you’re dealing with nonignorable missing data, the effort to restore identification through a shadow variable has a measurable cost, and that cost isn't just theoretical; it depends on whether the variable has an atom at its essential supremum.

Jane: So for someone listening who isn't reading the math, what does this actually change for their day-to-day work in building predictive models? It seems to suggest that simply finding a predictor isn't enough; you have to understand its specific mathematical properties regarding its extreme values.

Lu: It emphasizes that relevance is a global property; every quantity downstream can be followed as a function of it, and the range you must scan grows as relevance falls while the range you actually can scan does not.

Meng: It means we need to stop blindly screening features for identification and start looking at how those features interact with the response model in terms of exclusion and relevance. That’s a necessary shift from just feature engineering to actual identification diagnostics.

Tom: And it points us toward adaptive regularization; the optimal ridge rate should rise as relevance falls, which means our tuning strategy needs to change dynamically depending on how well our shadow variable is actually working.

Jane: So the paper makes it clear that inference under nonignorable missingness isn't a fixed problem; the cost of identification changes based on these underlying structural properties of the variables we use to try and fix it.

Conclusion: Tom: So, we’re wrapping up this deep dive on "The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes." Basically, they found that if you try to fix missing data using a shadow variable—a predictor you exclude from the main calculation—the cost of making that fix depends entirely on how relevant that predictor actually is.

Jane: So, what does "relevance vanishing" mean in plain English? Like, when a variable becomes so weak in predicting the outcome that it barely matters anymore?

Lu: Exactly. The paper treats relevance like a switch—it’s either yes or no—but they make it continuous by scaling how much that shadow variable contributes to the whole picture.

Meng: So, if you want identification, you need this variable to be relevant enough to matter, right? And the cost of that identification scales based on whether this variable has a specific mathematical property called an atom at its essential supremum.

Tom: That’s where it gets technical but super important. They show that when relevance drops low, the effort you put into finding that shadow variable doesn't give you much more information about the data itself; it just gives you a tiny bit of identification strength for a huge cost in terms of how many samples you effectively have.

Jane: That means if we’re trying to build something with missing data, and our best guess for a helpful predictor is weak, we’re essentially paying a high price for that guess.

Lu: Right. The analysis shows that this cost relates directly to the effective sample size, which can grow like the square root of the logarithm of your total sample size, but only if you have enough relevance to start with.

Meng: From a production standpoint, that sounds rough because we want reliable estimates at scale. If identification strength is low and we need more samples just to get a decent result, that’s not scalable.

Tom: The authors point out that the method they use for scanning—the range you have to look in—stays small, around the square root of the log of your sample size, even when relevance falls. So you can’t just keep widening your search area hoping to find something better.

Jane: But what does that tell us about how we should actually approach these missing data problems in AI and data science?

Lu: It suggests that before you start hunting for a shadow variable, you need to check its relevance against the exclusion conditions first; it’s not just about finding *a* predictor, it's about finding one that isn't too weak.

Michael Büler, Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences

Münster School of Business, FH Münster University of Applied Sciences · Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences

stat.ME, math.ST, stat.AP, stat.TH

Submitted: 2026-10-10

Updated: 2026-11-09

Comments: 50 pages, 2 figures. Replication package: https://doi.org/10.5281/zenodo.23262503. v3 adds three antecedent citations, corrects a screen that coded threshold rules linearly, extends the screen to a continuous outcome, and bounds the scorecard coefficients under a bounded departure from MAR. v2 consolidated the work at the moderators' request. Two papers are under separate review

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper…

Key concepts

Shadow Variable
A covariate that predicts an outcome but is excluded from the response mechanism. It is used to restore identification when a primary outcome is missing not at random.
Relevance
The strength of a shadow variable, treated as a yes-or-no condition initially. The paper treats this relevance continuously by scaling its contribution to moment conditions, showing how it affects inference.
Effective Sample Size (ESS)
A measure that governs the practical failures of empirical likelihood methods. It is determined exactly by the tilting variable's properties and never exceeds the total respondents, indicating how much information is truly available for inference.

Terminology

Summary

When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism The paper investigates what inference costs as the relevance of this shadow variable vanishes, finding that this cost depends on whether the tilting variable has an atom at its essential supremum

The nature of weak relevance

The authors treat the first requirement for a shadow variable, relevance, as a yes-or-no condition and then make it continuous by parametrizing it with a scalar that scales the shadow variable’s entire contribution to the moment conditions They characterize this functional exactly and show that it fixes the effective sample size, which grows like the square root of the logarithm of the sample size, while a thousandfold increase in sample size buys one unit of tilt This characterization is concrete and shows that whether the functional saturates or diverges depends on whether the tilting variable has an atom at its essential supremum

The role of identification and effective sample size

The second moment of the inverse weight decomposes exactly into a term involving a quadratic divergence of an exponential tilt, which yields a sharp dichotomy where the functional is capped when the tilting variable has an atom at its essential supremum This decomposition determines the effective sample size exactly, which is given by ESS(gamma) = n E(pi0) + 2 pr(R = 0) + pr(R = 0)2Λ(gamma) The effective sample size never exceeds the respondents, ESS ≤ n pr(R = 1), with equality when the weight is degenerate

The reach of the scanned set

The reach of a scanned confidence set grows like (log n)1/2 and is free of ρ, whereas the set that must be scanned widens without limit as ρ falls The procedure prescribed to reveal nonidentification therefore conceals it by giving no sign of having done so The reach grows like (log n)1/2 and does not depend on ρ, and the functional Λ itself exceeds the largest representable number once logΛ > log(DBL MAX) ≐ 709.78 irrespective of n

Diagnostic tools for analysis

The paper provides a checklist for analyzing results, including relevance, exclusion, and overlap The effective sample size is the master quantity that governs the practical failures of empirical likelihood and is flat in the identification strength A candidate shadow variable should be placed on the relevance–exclusion plane before it is adopted

Conclusion and practical implications

The factorization of the identifying signal through a scalar at every point of the identified set shows that relevance is a global property, and every quantity downstream can be followed as a function of it The range one must scan grows as relevance falls while the range one can scan does not The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The choice of regularization rate should be adaptive, as the optimal ridge rises as relevance falls

Data preparation

The data preparation involves several conventions to reproduce the analysis, including recoding durations and age levels These preparations are necessary to reconcile published tables with the analysis performed in the paper The reproduction of these details is essential for applying the diagnostics correctly

Simulation and coverage

The simulation compares four procedures across different relevance levels and sample sizes, showing that at low relevance, the unregularized representer estimator is worse than doing nothing The Anderson–Rubin set holds its nominal level in every cell, which is what identification-robust means The coverage of the nominal 95 per cent interval remains nominal throughout, while the Wald interval falls as relevance decreases The scanned set is therefore honestly unbounded exactly where the data remain uninformative The measured reach exceeds (log n)1/2 throughout and by a widening margin, indicating that the empirical effective sample size overstates how far the scan can be carried The signature Dufour (1997) describes is visible in the Anderson–Rubin set holding its nominal level while the Wald interval falls as relevance falls

Application to credit scoring

The application to consumer credit shows that no usable shadow variable existed in either data set, with the most relevant characteristic—the credit score—violating exclusion in almost exactly equal measure The effective sample size tells the same story, provided it is evaluated at the estimate rather than at ignorability Screening all characteristics shows this was not an unlucky choice, as the most relevant characteristic has a ratio of 0.95: relevant and effectively excluded The failure is invisible in a coefficient table and fatal to any inverse-weighted method The diagnostic is reported as a property of that analysis rather than as a correction applied to it The choice of the excluded characteristic by its insignificance in the response model is what a formal selection procedure also does The relevance and exclusion are not in tension by accident, but under an aligned selector they are incompatible The search for a shadow variable should be directed at characteristics the scorecard could not use in the form in which they matter The binary case is the benign case because saturation is not identification, and the behavior Dufour (1997) prescribes The difference between the two cases is whether the analyst is told

References

Andrews, D. W. K. and Cheng, X. (2012). Estimation and inference with weak, semi-strong, and strong identification Beppu, K. p. 8

Where the paper says it

Move along the identified curve, so that |𝛾𝑌 | → ∞ while 𝛾𝑋(𝑠) remains bounded. (i) If 𝑌 is binary with pr(𝑌 = 1) ∈ (0, 1), then Λ(𝛾) ⟶ 𝐴2 𝐴2 1 , 𝐴𝑘 ∶= 𝐸{𝜋0 𝑒 −𝑘𝛾T 𝑋𝑋𝟙(𝑌 = 1)}, (11) a finite constant: the tilt cannot explode, however strong the assumed departure from ignorability.

and Morikawa, K. p. 2
Where the paper says it

(2025) give a recent alternative in which the identifying restriction is a multiplicative non-interaction in the selection probability rather than a completeness or rank condition on the stacked system. Beppu and Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when the outcome is categorical too, but that completeness can fail even in simple models once one of the two is continuous; for that case they give a sufficient condition that can be checked on the observed data.

(2024). Verifiable identification condition for nonignorable nonresponse data with categorical instrumental variables Bücker, M. p. 2
Where the paper says it

(2025) give a recent alternative in which the identifying restriction is a multiplicative non-interaction in the selection probability rather than a completeness or rank condition on the stacked system. Beppu and Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when the outcome is categorical too, but that completeness can fail even in simple models once one of the two is continuous; for that case they give a sufficient condition that can be checked on the observed data.

, van Kampen, M., and Krämer, W. (2013). Reject inference in consumer credit scoring with nonignorable missing data Caner, M. p. 0
Where the paper says it

A reanalyzed credit-scoring study shows the cost: a characteristic an optimizing selector uses cannot satisfy exclusion. Keywords: Effective sample size · Exponential tilt · Identification-robust inference · Missing not at random · Nonresponse instrument.

(2010). Exponential tilting with weak instruments: estimation and testing Chen, H. p. 4
Where the paper says it

The distinction matters for what the sets mean: theirs do not shrink with the sample, ours do, but only so slowly that Section 4 becomes the binding constraint. Exponential tilting in Caner (2010) names the member of the generalized empirical likelihood family obtained by choosing the exponential divergence in the estimating criterion, and that paper studies it under weak instruments.

, Simchi-Levi, D., and Xiong, R. (2026). AI-generated measurements for identification and inference with missing data: a weak shadow variable approach Chen, J. p. 1
Where the paper says it

One recent line drops that condition rather than grading it: Chen et al. (2026) call a shadow variable weak when completeness is not imposed, and obtain partial identification with sharp bounds.

and Fang, F. p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

(2019). Semiparametric likelihood for estimating equations with nonignorable nonresponse Chen, J. p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

and Shao, J. p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

(2025). Instrument, variable and model selection with nonignorable nonresponse Zhao, J. p. 2
Where the paper says it

(2025) give a recent alternative in which the identifying restriction is a multiplicative non-interaction in the selection probability rather than a completeness or rank condition on the stacked system. Beppu and Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when the outcome is categorical too, but that completeness can fail even in simple models once one of the two is continuous; for that case they give a sufficient condition that can be checked on the observed data.

and Shao, J. (2015). Semiparametric pseudo-likelihoods in generalized linear models with nonignorable missing data Zhao, P. p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

(2019). Empirical likelihood and Wilks phenomenon for data with nonignorable missingness Zhao, P. p. 0
Where the paper says it

The price of a weak shadow variable: inference under nonignorable missingness as relevance vanishes Michael Bücker1,2 1Münster School of Business, FH Münster University of Applied Sciences 2 Institute for Process Management and Digital Transformation (IPD), FH Münster University of Applied Sciences Correspondence: michael. buecker@fh-muenster.

, Wang, L., and Shao, J. p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

p. 0
Where the paper says it

Under completeness conditions the model is then point identified, and a range of estimators is available: semiparametric likelihood (Qin et al. , 2002; Chen and Fang, 2019), pseudo-likelihood (Zhao and Shao, 2015), and estimators that avoid the inverse weight altogether (Li et al.

(2021). Sufficient dimension reduction and instrument search for data with nonignorable nonresponse Zhao, P. p. 2
Where the paper says it

(2025) give a recent alternative in which the identifying restriction is a multiplicative non-interaction in the selection probability rather than a completeness or rank condition on the stacked system. Beppu and Morikawa (2024) show that for a categorical instrument the two conditions are equivalent when the outcome is categorical too, but that completeness can fail even in simple models once one of the two is continuous; for that case they give a sufficient condition that can be checked on the observed data.

, Wang, L., and Shao, J. (2021).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive regularization based on relevance is possible by using a ridge penalty lambda star derived from an optimization over relevance and sample size, as shown in Equation 21. This allows for tuning lambda to balance the bias against variance, leading to a rule that is transportable across different designs and performs better than unregularized methods.

  2. The system can diagnose infeasibility of empirical likelihood problems by checking the effective sample size ESS/n and the largest inverse weight, as detailed in Equation 25. This provides a diagnostic about whether the data do identify what it corrects for, separating issues related to identification strength from those related to overlap.

  3. Inference procedures can be made robust against weak relevance by requiring that a candidate shadow variable has high relevance-to-exclusion violation ratio, as suggested by Table 6 in Section 9.2. This requires the characteristic's entire contribution to the outcome to lie outside the class optimized by the selector, rather than just being partially visible to it.

Abstract

When an outcome is missing not at random, identification is commonly restored by a shadow variable: a covariate that predicts the outcome but is excluded from the response mechanism. The literature treats the first requirement, relevance, as a yes-or-no condition. We make it continuous, parametrizing it by a scalar that scales the shadow variable's entire contribution to the moment conditions, and ask what inference costs as that scalar vanishes. One functional of the inverse weight governs the answer. We characterize it exactly, show that it fixes the effective sample size, and use it to sharpen the rates of the estimator that avoids the weight. The characterization is concrete. Whether the functional saturates or diverges turns not on boundedness of the outcome but on whether the tilting variable has an atom at its essential supremum: a binary outcome saturates, a uniform one does not. It also bounds how far a scanned set reaches, and that reach grows only as the square root of the logarithm of the sample size, so a thousandfold increase buys one unit of tilt while the set to be scanned widens. A reanalyzed credit-scoring study shows the cost: a characteristic an optimizing selector uses cannot satisfy exclusion.

Sources

Related papers