Non-Invariance in Nested Prediction Models under Selective Predictor Availability
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Non-Invariance in Nested Prediction Models under Selective Predictor Availability".
Jane: Here's the gist of the paper. Non-invariance in nested prediction models under selective predictor availability decomposes into components due to omission of the additional predictor, potential residual non-invariance, and their interaction <ref:2609.02836#pg18>.
Tom: This framework provides a formal basis for understanding how selective predictor measurement affects the generalisability of clinical prediction models using routinely collected health data <ref:2609.02836#pg15>.
Jane: Formal Framework and Setting The paper introduces nested prediction models under differential predictor availability, defining a restricted model based on available predictors X and an extended model including an additional selectively measured predictor Z <ref:2609.02836#pg16>.
Tom: The setting is formalized using a Directed Acyclic Graph (DAG) (D1) to illustrate the causal structures underlying these models under differential predictor availability <ref:2609.02836#pg18>.
Paper summary: Tom: So, wrapping up this discussion on "Non-Invariance in Nested Prediction Models under Selective Predictor Availability." The main point is that when you select your patients based on whether a predictor Z is available or not, the difference in prediction accuracy between those groups isn't just random noise.
Jane: It’s systematically decomposed into three parts: the omission of Z, any leftover non-invariance even with Z included, and how those factors interact. So it shows that we have a formal way to break down *why* a model might perform differently across populations.
Lu: And the most concrete result is that this total non-invariance is fully transferred into the imputation bias when you impute Z based on data from the selected population <ref:2609.02836#pg18>.
Meng: So, for us in deployment, it means that using standard imputation based on what we have in the selected group carries this non-invariance forward as bias into our extended model. You can't just ignore that transfer when you try to generalize your prediction model.
Tom: The implication is that while you can use these imputation strategies to make the extended model computable, it doesn't fix the underlying identification problem; it just moves the non-invariance around as bias into a new place.
Jane: It’s about understanding that concerns about generalisability aren't automatically reasons to discard a model validated in an external population; it depends entirely on the selection mechanism used for that validation <ref:2609.02836#pg18>.
Lu: The paper provides a principled framework for this tension between what’s available and what’s needed to build models that work across different patient groups.
Meng: We need to pay attention to how we identify the bias directly from the observable data, because the paper shows that it equals the unobservable bias you'd introduce by imputing Z <ref:2609.02836#pg18>.
Tom: It’s a shift from just saying "the model is off" to understanding *why* it’s off and *how much* we have to account for that specific missing data piece in our deployment strategy.
Conclusion: Tom: So we’re wrapping up on this paper about non-invariance in nested prediction models under selective predictor availability, and who wrote it, let's see.
Jane: It’s by Sperrin, Riley, Collins, and Martin. It sounds like a mouthful for a radio show.
Lu: They focus on how omitting a predictor Z selectively changes the model's performance between different patient groups.
Meng: So they’re looking at how we build models when we don't always have all the data points we need for everyone.
Tom: Basically, it breaks down why a model might look different when you only give it some of its inputs, and how that difference comes from missing information or interaction effects.
Jane: The main thing they show is that this non-invariance usually boils down to the bias introduced when you try to guess the missing predictor based on what you know about the selected group.
Lu: They formalize this by showing that if there's no extra info, the non-invariance in your small model actually equals a specific imputation bias.
Meng: So it’s not just a theoretical issue; it means if you use standard imputation methods on routine data, you’re carrying that error forward into your final prediction. That’s a deployment headache.
Tom: Exactly. It links the math—the non-invariance—directly to the practical problem of imputing Z, which is what we actually have to do in clinical settings.
Jane: And they suggest that while you can use those imputation methods to make your extended model work, it doesn't magically fix the underlying identification issue.
Lu: It’s a question of identification, and it shows that observable non-invariance is measurable directly from the data we already have, which is pretty promising for validation.
Tom: Pretty promising, but it also highlights that you need a careful strategy when validating models across different populations; just because a model works in one group doesn't mean it works everywhere.
Jane: It sets up the next big question for us about how we should actually test if those imputation strategies are actually working in the real world.
School of Life Course & Population Sciences · Department of Population Health Sciences, King’s College London
stat.ME
Submitted: 2026-09-02
Updated: 2026-11-12
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 77/100
The gist: Non-invariance in nested prediction models under selective predictor availability decomposes into components due to omission of the additional predictor, potential residual non-invariance…
Key concepts
- Nested Prediction Models
- These are prediction models built in layers. A restricted model uses only available data (predictors X), while an extended model includes an extra predictor (Z) that might be selectively measured based on patient information. The paper examines how the performance difference between these two models changes depending on which patients you are looking at.
- Non-Invariance
- This refers to the fact that a prediction model's performance (its accuracy or bias) is not the same when applied to different populations, even if both populations share some predictors. In this context, it specifically measures how much the restricted model's error changes when comparing selected versus target patient groups.
- Selective Predictor Availability
- This occurs when a clinical predictor (like ACR) is only measured if certain conditions are met or based on other patient information. This means the availability of a predictor is not random but depends on the specific patient, creating selection bias that affects how models generalize to the broader population.
Terminology
Summary
Non-invariance in nested prediction models under selective predictor availability decomposes into components due to omission of the additional predictor, potential residual non-invariance, and their interaction <ref:2609. p. 002836#pg18>. This framework provides a formal basis for understanding how selective predictor measurement affects the generalisability of clinical prediction models using routinely collected health data <ref:2609. p. 002836#pg15>.
Formal Framework and Setting
The paper introduces nested prediction models under differential predictor availability, defining a restricted model based on available predictors X and an extended model including an additional selectively measured predictor Z <ref:2609.02836#pg16>. The setting is formalized using a Directed Acyclic Graph (DAG) (D1) to illustrate the causal structures underlying these models under differential predictor availability <ref:2609. p. 102836#pg18>. Selective measurement occurs when the decision to measure a clinical predictor depends on patient information, denoted by S being dependent on predictors X or other manifestations of the underlying disease process W <ref:2609. p. 202836#pg18>.
Decomposition of Non-Invariance
The non-invariance in the restricted model between selected and target populations decomposes into components due to omission of the additional predictor, potential residual non-invariance, and their interaction <ref:2609.02836#pg18>. The decomposition is expressed as ∆X,S = ∆exp ZX + ∆res Z,X + ∆int Z,X <ref:2609. p. 602836#pg19>. The first term represents non-invariance attributable to omission of Z, resolved by inclusion of Z in the extended model <ref:2609. p. 602836#pg18>.
Imputation Bias and Transfer
Imputation based on the conditional distribution of the additional predictor in the selected population transfers restricted-model non-invariance to the imputed extended model as imputation bias <ref:2609.02836#pg18>. Specifically, this bias is shown to be equal to ∆X,S = P(Y X, S = 1) − P(Y X) <ref:2609. p. 002836#pg20>. This identification result links the observable restricted-model non-invariance directly to the unobservable bias introduced by imputing Z from the selected population <ref:2609. p. 002836#pg18>.
Application to Kidney Failure Risk Equation
The framework is illustrated using the Kidney Failure Risk Equation (KFRE), where albumin-to-creatinine ratio (ACR) availability is selectively measured in routine clinical practice <ref:2609.02836#pg18>. The analysis of the KFRE showed that ACR availability was associated with age, sex, eGFR and diabetes, and the restricted three-variable model showed evidence of non-invariance <ref:2609. p. 002836#pg18>.
Collider Structure and Selection Bias
When selection operates through a collider structure involving multiple routes of selection, the total non-invariance in the restricted model can be decomposed into subgroup-specific components <ref:2609.02836#pg19>. This generic collider structure is compatible with both an internal route of selection and an external route operating through manifestations of the underlying disease process <ref:2609. p. 802836#pg18>.
Implications for Validation
The results provide a principled framework for understanding how selective predictor measurement gives rise to model non-invariance and its implications for the generalisability of clinical prediction models to their intended target populations <ref:2609.02836#pg18>. The study highlights that concerns regarding the generalisability of a model validated in an external population may not rule out its transportability to other populations selected through identical selection mechanisms <ref:2609. p. 102836#pg18>.
Appendix on Imputation Bias
The imputation bias at the conditional level is shown to be equal to δX,Z = πX,Zδimp X,Z,S=1 + (1 − πX,Z)δimp X,Z,S=0 <ref:2609.02836#pg18>. This final result shows that non-invariance in the restricted model is fully transferred to imputation bias <ref:2609. p. 902836#pg18>. The imputation bias over the target population is ultimately found to be equal to ∆X,S = P(Y X, S = 1) − P(Y X) <ref:2609. p. 002836#pg18>.
Appendix on Full Clinical DAG
The full clinical DAG for the KFRE is presented in (D3), showing the relationships between UCKD, IDM, eGFR, ACR, S and ESRD <ref:2609.02836#pg18>. This structure is consistent with routine clinical practice where ACR testing is frequently prompted by specific clinical indications such as diabetes <ref:2609. p. 002836#pg18>. The pattern observed is consistent with two colliding routes of selection implying three patient profiles <ref:2609. p. 702836#pg18>.
References
[1] B. A. Goldstein, N. A. Bhavsar, M. Phelan, and M. J. Pencina Controlling for Informed Presence Bias Due to the Number of Health Encounters in an Electronic Health Record
in American Journal of Epidemiology 184 (11 Dec 2016) pp 847–855 <ref:2609.02836#pg18>.
[2] B. A. Goldstein, A. M. Navar, M. J. Pencina, and J. P Ioannidis Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review
in Journal of the American Medical Informatics Association 24.1 (2017) pp 198–208 <ref:2609. p. 002836#pg18>.
[3] M. A Hernán, S. Hernández-Díaz, and J. M Robins A structural approach to selection bias
in Epidemiology 15 (5 Sept 2004) pp 615–625 <ref:2609.02836#pg18>.
[4] H Lu, G S Gonsalves, and D Westreich Selection bias requires selection: the case of collider stratification bias
in American journal of epidemiology 193.3 (2024) pp 407–409 <ref:2609. p. 002836#pg18>.
[5] D G Altman and P Royston What do we mean by validating a prognostic model?
in Statistics in medicine 19.4 (2000) pp 453–473 <ref:2609. p. 002836#pg18>.
[6] K G M Moons, P Royston, Y Vergouwe, D E Grobbee, and D G Altman Prognosis and prognostic research: what why and how
in BMJ 338 (2009) p 375 <ref:2609.02836#pg18>.
[7] E W Steyerberg Applications of prediction models
in Clinical Prediction Models A Practical Approach to Development Validation and Updating New York Springer 2009 pp 11–31 <ref:2609.02836#pg18>.
[8] E W Steyerberg, K G Moons, D A van der Windt, J A Hayden, P Perel, S Schroter et al Prognosis Research Strategy PROGRESS 3: prognostic model research
in PLoS medicine 10.2 (2013) e1001381 <ref:2609. p. 002836#pg18>.
[9] R F Wolff, K G Moons, R D Riley, P F Whiting, M Westwood, G S Collins et al PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
in https://doi.org/10.7326/M18-1376 170 (1 Jan 2019) pp 51–58 <ref:2609. p. 002836#pg18>.
[10] O Efthimiou, M Seo, K Chalkou, T Debray, M Egger, and G Salanti Developing clinical prediction models: a step-by-step guide
in BMJ 386 (2024) <ref:2609.02836#pg18>.
[11] R Sisk, L Lin, M Sperrin, J K Barrett, B Tom, K Diaz-Ordaz et al Informative presence and observation in routine health data: a review of methodology for clinical risk prediction
in Journal of the American Medical Informatics Association 28.1 (2021) pp 155–166 <ref:2609. p. 002836#pg18>.
[12] B J Wells, K M Chagin, A S Nowacki, and M W Kattan Strategies for handling missing data in electronic health record derived data
in Egems 1.3 (2013) p 1035 <ref:2609. p. 002836#pg18>.
[13] J M Madden, M D Lakoma, D Rusinak, C Y Lu, and S B Soumerai Missing clinical and behavioral health data in a large electronic health record EHR system
in Journal of the American Medical Informatics Association 23.6 (2016) pp 1143–1149 <ref:2609. p. 002836#pg18>.
[14] J Pearl and E Bareinboim Transportability of causal and statistical relations: A formal approach
in Proceedings of the AAAI Conference on Artificial Intelligence Vol 25. 1 (2011) pp 247–254 <ref:2609. p. 002836#pg18>.
[15] G S Collins, P Dhiman, J Ma, M M Schlussel, L Archer, B Van Calster et al Evaluation of clinical prediction models (part 1): from development to external validation
in bmj 384 (2024) <ref:2609.02836#pg18>.
[16] N Tangri, L A Stevens, J Griffith, H Tighiouart, O Djurdjev, D Naimark et al A predictive model for progression of chronic kidney disease to kidney failure
in Jama 305.15 (2011) pp 1553–1559 <ref:2609. p. 002836#pg18>.
[17] S D Fraser, J Parkes, D Culliford, M Santer, and P Roderick Timeliness in chronic kidney disease and albuminuria identification: a retrospective cohort study
in BMC family practice 16.1 (2015) p 18 <ref:2609. p. 002836#pg18>.
[18] C D Chu, F Xia, Y Du, R Singh, D S Tuot, J A Lamprea-Montealegre et al Estimated prevalence and testing for albuminuria in US adults at risk for chronic kidney disease
in JAMA network open 6.7 (2023) e2326230–e2326230.
[19] D B Ketema, H Wallace, W Hailu, S V Badve, P Ronksley, B L Neuen et al Quality of care for people with chronic kidney disease: a systematic review and meta-analysis
in BMJ open 15.9 (2025) e102044.
[20] J J Heckman Sample selection bias as a specification error
in Econometrica Journal of the econometric society (1979) pp 153–161 <ref:2609.02836#pg18>.
[21] S van Buuren Flexible Imputation of Missing Data 2nd ed Boca Raton FL Chapman and Hall CRC 2018 <ref:2609. p. 002836#pg18>.
[23] M A Hernán The Hazards of Hazard Ratios
in Epidemiology 21.1 (2010) pp 13–15 <ref:2609. p. 002836#pg18>.
[24] R Oaxaca Male-Female Wage Differentials in Urban Labor Markets
in International Economic Review 14.3 (1973) pp 693–709 <ref:2609. p. 002836#pg18>.
[25] A S Blinder Wage Discrimination: Reduced Form and Structural Estimates
in Journal of Human Resources 8.4 (1973) pp 436–455 <ref:2609. p. 002836#pg18>.
[26] M Sperrin, R D Riley, G S Collins, and G P Martin Targeted validation: validating clinical prediction models in their intended population and setting
in Diagnostic and prognostic research 6.1 (2022) p 24.
[27] K Ploddi, M Sperrin, G P Martin, and M M O’Connell Scoping review of methodology for aiding generalisability and transportability of clinical prediction models
in arXiv preprint arXiv:2412.04275 (2024).
[30] N Tangri, M E Grams, A S Levey, J Coresh, L J Appel, B C Astor et al Multinational assessment of accuracy of equations for predicting risk of kidney failure: a meta-analysis
in Jama 315.2 (2016) pp 164–174.
--- Page 3 ---
U W X Z S Y (D1)
Appendix on Residual Prognostic Information
Under invariance of the extended model, non-invariance of the restricted model implies that Z both carries residual prognostic information and is selectively measured, i.e: Z ̸⊥⊥ Y X and Z ̸⊥⊥ S X. This leads to the decomposition ∆X,S = ∆exp ZX + ∆res Z,X + ∆int Z,X. p. 6 The final result shows that non-invariance in the restricted model is fully transferred to imputation bias. p. 9
Appendix on Imputation Bias at Conditional Level
At the conditional level, δimp X,Z,S=0 = P(Y X, Z, S ˆ = 0) − P(Y X, Z, S = 0). This is expressed as δX,Z (1 − πX,Z). p. 4 The imputation bias over the target population is found to be ∆imp X,Z = P(Y X, S = 1) − P(Y X). p. 9 This final result shows that non-invariance in the restricted model is fully transferred to imputation bias. p. 9
The full clinical DAG for the KFRE is shown in (D3) with UCKD, IDM, Age Sex eGFR ACR S ESRD. This structure is consistent with routine clinical practice where ACR testing is frequently prompted by specific clinical indications such as diabetes. p. 0 The pattern observed is consistent with two colliding routes of selection implying three patient profiles. p. 7
Appendix on Collider Structure
The generic collider structure is compatible with both an internal route of selection and an external route operating through manifestations of the underlying disease process. This structure results in a fragmented selected population with potentially heterogeneous outcome risk. p. 2 The decomposition of total non-invariance is given by δX,S = π⊤ ·XδX,S,. p. 8 This generic collider structure is compatible with an internal route of selection and an external route operating through manifestations of the underlying disease process. p. 8
Appendix on Identification
The results show that observable restricted-model non-invariance is estimable directly from observed values of Y, X and S in both selected and non-selected patients. This identifiable quantity equals the otherwise unobservable bias introduced by imputing Z from the selected population. p. 0 This links the present work to Heckman’s treatment of sample selection as a specification problem and to related work on the identifiability of causal and statistical relations across populations. p. 4
Appendix on Missing-Data Literature
Under A1, the conditional distribution of Z in selected patients equals that in the target population, so standard imputation based on P(ZI, S = 1) is unbiased <ref:2609. p. 002836#pg18>. Under ¬ A1 (MNAR), imputation based on the selected-population distribution introduces bias <ref:2609. p. 0 p. 002836#pg18>. The present framework extends this distinction by showing that, when Z is MNAR, the resulting imputation bias is exactly identified: it equals the observable restricted-model non-invariance <ref:2609. p. 002836#pg18>. Under A1, the conditional distribution of Z in selected patients equals that in the target population, so standard imputation based on P(ZI, S = 1) is unbiased <ref:2609.02836#pg18>. Under ¬ A1 (MNAR), imputation based on the selected-population distribution introduces bias <ref:2609.02836#pg18>.
Appendix on Practical Strategy
Imputation strategies based on the conditional distribution of the predictor in the selected population can be used to restore the applicability of the extended model to the non-selected population. Although this constitutes a pragmatic strategy for reducing the cost of unavailable predictor measurements, it does not resolve underlying non-identification — it merely renders the extended model computable, at the cost of carrying restricted-model non-invariance forward as bias. p. 5 The results presented here can also be framed as a question of identification. p. 0
Appendix on Calibration
In practice, although recalibration may improve operational performance, it may not fully restore distributional invariance unless separate calibration strategies are adopted for the selected and non-selected populations. This opens several directions for future methodological research. p. 1
Appendix on Structural Definition
The proposed framework builds upon current approaches to the development and validation of clinical prediction models intended for clinically defined target populations. p. 1 The proposed nested structure of prediction models explicitly accounts for this tension. p. 1 This in turn allows the formalisation of how selective predictor measurement induces model non-invariance between selected patients and the target population. p. 4 The proposed framework builds upon current approaches to the development and validation of clinical prediction models intended for clinically defined target populations.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Formal framework for non-invariance decomposition: The system can decompose restricted model non-invariance into "non-invariance due to omission of a selectively measured predictor, resolved in the selected population after inclusion of this predictor, and potential residual non-invariance due to omission of other predictors and their interaction."
-
Imputation bias quantification: The system can quantify imputation bias as
the observable restricted model non-invariance,
showing thatnon-invariance in the restricted model is fully transferred to the imputed extended model in the form of imputation bias.
-
Collider structure handling: The system can explicitly account for selection operating through multiple routes by decomposing total non-invariance into subgroup-specific components, represented by
the k-specific non-invariance, with k in X W, W X, X n W.
-
Model transportability assessment: The system can distinguish between generalisability and transportability to identify that
models developed in a selected population may well be transportable to other populations selected through identical selection mechanisms.
Sources
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes