Non-Invariance in Nested Prediction Models under Selective Predictor Availability

summary

The gist

Non-invariance in nested prediction models under selective predictor availability decomposes into components due to omission of the additional predictor, potential residual non-invariance…

In short

The paper formalizes how not measuring a predictor selectively causes prediction models to become non-invariant when applied to different patient groups. It decomposes this non-invariance into components from omission, residual effects, and interaction. The key finding is that the bias introduced by imputing missing data is directly equal to the observable model non-invariance.

Key concepts

Nested Prediction Models
These are prediction models built in layers. A restricted model uses only available data (predictors X), while an extended model includes an extra predictor (Z) that might be selectively measured based on patient information. The paper examines how the performance difference between these two models changes depending on which patients you are looking at.
Non-Invariance
This refers to the fact that a prediction model's performance (its accuracy or bias) is not the same when applied to different populations, even if both populations share some predictors. In this context, it specifically measures how much the restricted model's error changes when comparing selected versus target patient groups.
Selective Predictor Availability
This occurs when a clinical predictor (like ACR) is only measured if certain conditions are met or based on other patient information. This means the availability of a predictor is not random but depends on the specific patient, creating selection bias that affects how models generalize to the broader population.

Terminology used across episodes

This episode discusses

The paper

Non-Invariance in Nested Prediction Models under Selective Predictor Availability · Read on arXiv

School of Life Course & Population Sciences · Department of Population Health Sciences, King’s College London

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Non-Invariance in Nested Prediction Models under Selective Predictor Availability".

Jane: Here's the gist of the paper. Non-invariance in nested prediction models under selective predictor availability decomposes into components due to omission of the additional predictor, potential residual non-invariance, and their interaction <ref:2609.02836#pg18>.

Tom: This framework provides a formal basis for understanding how selective predictor measurement affects the generalisability of clinical prediction models using routinely collected health data <ref:2609.02836#pg15>.

Jane: Formal Framework and Setting The paper introduces nested prediction models under differential predictor availability, defining a restricted model based on available predictors X and an extended model including an additional selectively measured predictor Z <ref:2609.02836#pg16>.

Tom: The setting is formalized using a Directed Acyclic Graph (DAG) (D1) to illustrate the causal structures underlying these models under differential predictor availability <ref:2609.02836#pg18>.

Paper summary: Tom: So, wrapping up this discussion on "Non-Invariance in Nested Prediction Models under Selective Predictor Availability." The main point is that when you select your patients based on whether a predictor Z is available or not, the difference in prediction accuracy between those groups isn't just random noise.

Jane: It’s systematically decomposed into three parts: the omission of Z, any leftover non-invariance even with Z included, and how those factors interact. So it shows that we have a formal way to break down *why* a model might perform differently across populations.

Lu: And the most concrete result is that this total non-invariance is fully transferred into the imputation bias when you impute Z based on data from the selected population <ref:2609.02836#pg18>.

Meng: So, for us in deployment, it means that using standard imputation based on what we have in the selected group carries this non-invariance forward as bias into our extended model. You can't just ignore that transfer when you try to generalize your prediction model.

Tom: The implication is that while you can use these imputation strategies to make the extended model computable, it doesn't fix the underlying identification problem; it just moves the non-invariance around as bias into a new place.

Jane: It’s about understanding that concerns about generalisability aren't automatically reasons to discard a model validated in an external population; it depends entirely on the selection mechanism used for that validation <ref:2609.02836#pg18>.

Lu: The paper provides a principled framework for this tension between what’s available and what’s needed to build models that work across different patient groups.

Meng: We need to pay attention to how we identify the bias directly from the observable data, because the paper shows that it equals the unobservable bias you'd introduce by imputing Z <ref:2609.02836#pg18>.

Tom: It’s a shift from just saying "the model is off" to understanding *why* it’s off and *how much* we have to account for that specific missing data piece in our deployment strategy.

Conclusion: Tom: So we’re wrapping up on this paper about non-invariance in nested prediction models under selective predictor availability, and who wrote it, let's see.

Jane: It’s by Sperrin, Riley, Collins, and Martin. It sounds like a mouthful for a radio show.

Lu: They focus on how omitting a predictor Z selectively changes the model's performance between different patient groups.

Meng: So they’re looking at how we build models when we don't always have all the data points we need for everyone.

Tom: Basically, it breaks down why a model might look different when you only give it some of its inputs, and how that difference comes from missing information or interaction effects.

Jane: The main thing they show is that this non-invariance usually boils down to the bias introduced when you try to guess the missing predictor based on what you know about the selected group.

Lu: They formalize this by showing that if there's no extra info, the non-invariance in your small model actually equals a specific imputation bias.

Meng: So it’s not just a theoretical issue; it means if you use standard imputation methods on routine data, you’re carrying that error forward into your final prediction. That’s a deployment headache.

Tom: Exactly. It links the math—the non-invariance—directly to the practical problem of imputing Z, which is what we actually have to do in clinical settings.

Jane: And they suggest that while you can use those imputation methods to make your extended model work, it doesn't magically fix the underlying identification issue.

Lu: It’s a question of identification, and it shows that observable non-invariance is measurable directly from the data we already have, which is pretty promising for validation.

Tom: Pretty promising, but it also highlights that you need a careful strategy when validating models across different populations; just because a model works in one group doesn't mean it works everywhere.

Jane: It sets up the next big question for us about how we should actually test if those imputation strategies are actually working in the real world.

More episodes

← Home