Missing data and cluster graphs: cluster-level missingness vs variable-level missingness
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Missing data and cluster graphs".
Jane: Here's the gist of the paper. Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences <ref:2605.20943#pg2> Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available <ref:2605.20943#pg2>.
Tom: In this paper, we investigate recoverability from such abstract representations <ref:2605.20943#pg2>.
Jane: The authors introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level <ref:2605.20943#pg2>.
Tom: They formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models and study how this abstraction affects the recoverability of probabilistic and causal queries <ref:2605.20943#pg2>.
Paper summary: Jane: So we've covered the main points of "Missing data and cluster graphs: cluster-level missingness vs variable-level missingness." We looked at how the m-C-DMG, which keeps variable indicators but clusters them causally, compares to the cm-C-DMG, which merges all indicators within a cluster into one vertex.
Tom: The paper shows that these two abstractions are not interchangeable; they encode different amounts of information, and the cm version is generally less informative for recovery. The main takeaway is that when you have missing data and only coarse structural knowledge, you need to be careful about which abstraction you use because it changes what queries you can answer.
Lu: It provides a principled way to analyze settings where we are forced to deal with uncertainty around relationships between variables because we don't know everything about them. It gives us specific conditions for when you can actually recover the joint distribution or a causal query from these abstract representations, which is really useful theory.
Meng: For deploying this in production, it means if your system relies on cluster summaries, you need to rigorously test whether that coarser representation preserves the necessary details for the queries you care about without introducing unacceptable errors at scale.
Jane: In simpler terms, this work helps us understand how much information we lose by simplifying the structure of our data representation when dealing with missing data in science and social studies. It’s a framework for making informed decisions about which level of structural detail is appropriate for a given problem.
Conclusion: Tom: So we're wrapping up this paper about missing data, and the title itself is pretty direct: "Missing data and cluster graphs: cluster-level missingness vs variable-level missingness."
Jane: It’s a really clear way to put it. Basically, they are comparing two different ways to look at missing information. One way looks at each individual variable separately, and the other way groups those variables into clusters first.
Lu: Exactly. They introduce two models, the m-C-DMG and the cm-C-DMG, which represent these different levels of abstraction—variable level versus cluster level.
Meng: But they’re not just drawing diagrams; they're defining how compatible those abstract graphs are with real missing data models. That compatibility stuff is where it gets interesting for deployment.
Tom: Right, so the core finding seems to be that this abstraction matters a lot for what you can actually figure out from the data later on.
Jane: They show that even when you only have a coarse structural idea of your data, like just cluster information, you can still recover certain things if the abstract graph is compatible in a specific way.
Lu: The paper establishes some strict rules for compatibility. If one cluster has variables with different missingness patterns, it’s much more powerful than just looking at the big cluster summary alone.
Meng: And thecm-C-DMG, which merges everything into one vertex per cluster, is generally less informative than the m-C-DMG because it loses those specific variable details.
Tom: So what does this mean practically? If you’re building a system that relies on these cluster summaries to make decisions, you need to know if you’re using the more detailed m version or the coarser cm version.
Jane: It means we have a principled way to choose our structural simplification based on what kind of query—like figuring out a joint distribution or a causal effect—we actually need to answer.
Lu: They give necessary and sufficient conditions for recovering those queries, which involves checking if certain vertices in the m-C-DMG are connected in a specific way.
Meng: From an engineering standpoint, that condition sounds like it’s going to be tricky to implement because you have to check paths through colliders and specific sets of vertices.
Tom: It sounds like they’re giving us the theoretical blueprint for when we can trust our simplified models, which is a big deal for real-world applications.
Jane: It really moves the conversation from just "how do we handle missing data?" to "what level of structure are we allowed to ignore, and what's the cost?" (SOUND of music slightly increasing in volume and then fading out)
Willow Scott, Eugenio Valdano, Charles K. Assaad
Sorbonne University · Inserm
stat.ME
Submitted: 2026-05-20
Updated: 2026-11-03
Comments: Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 71/100
The gist: Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences <ref:2605.20943#pg2> Recoverability from missing data is typically studied…
Key concepts
- m-C-DMG
- This graph retains specific missingness indicators for each individual variable while representing causal relationships between variables at the cluster level. It is a more detailed abstraction that keeps track of how each variable is missing data.
- cm-C-DMG
- This is a coarser abstraction where all missingness indicators within a cluster are merged into a single vertex. It aggregates information at the cluster level, sacrificing variable-specific detail for simplicity in structural representation.
- Compatibility
- This defines how well an abstract graph (like m-C-DMG or cm-C-DMG) aligns with the underlying variable-level missingness models. Compatibility determines what kind of probabilistic or causal queries can be reliably recovered from the graph structure.
- Recoverability
- This refers to the ability to successfully reconstruct specific information, such as joint distributions or causal effects, from an abstract representation of missing data. The paper establishes conditions under which these queries can be recovered using either graph type.
Terminology
Summary
Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences <ref:2605. p. 0 assaad@inserm. fr
Abstract
Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences. fr
Abstract
Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences. Recoverability from missing data
is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available,
for instance when variables are grouped into clusters due to limited knowledge
or interpretability reasons. Recoverability from missing data
is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available,
for instance when variables are grouped into clusters due to limited knowledge
or interpretability reasons. In this paper, we investigate recoverability from such
abstract representations.Where the paper says it
Where the paper says it
Where the paper says it
Introduction to Abstractions
The authors introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level <ref:2605.20943#pg2>. They formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models and study how this abstraction affects the recoverability of probabilistic and causal queries <ref:2605. p. 0 We introduce two classes of cluster-based missingness
graphs: the m-C-DMG, which retains variable-specific missingness indicators, and
the cm-C-DMG, which aggregates missingness mechanisms at the cluster level. We formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models, and study how this abstraction affects the
recoverability of probabilistic and causal queries. While both representations encode missingness
mechanisms at a higher level of abstraction, they differ in the amount of information they preserve,
raising the question of how this impacts recoverability. Our goal is to understand how recoverability of queries – such as joint distributions and causal effects –
depends on the level of abstraction.Where the paper says it
Where the paper says it
Defining the Missingness Graphs
The m-C-DMG retains variable-specific missingness indicators while representing causal relations between variables at the cluster level <ref:2605.20943#pg2>. The cm-C-DMG is a coarser abstraction that merges missingness indicators within each cluster into a single vertex <ref:2605. p. 1 The m-C-DMG retains variable-specific missingness indicators while representing causal relations
between variables at the cluster level. The cm-C-DMG is a coarser abstraction that merges missingness
indicators within each cluster into a single vertex. 3 Missingness in cluster graphs
A central objective of this paper is to investigate how missingness mechanisms can be incorporated
into cluster graphs. To this end, we introduce two abstract missingness representations, which differ
in the level of detail they retain about missingness indicators:
• missingness-C-DMG (m-C-DMG): an abstract graph in which each variable inside a cluster
is associated with its own missingness indicator;
• clustered-missingness-C-DMG (cm-C-DMG): a coarser abstract graph in which a single
missingness vertex represents the missingness mechanism of the whole cluster. 3 Missingness in cluster graphs
A central objective of this paper is to investigate how missingness mechanisms can be incorporated
into cluster graphs. To this end, we introduce two abstract missingness representations, which differ
in the level of detail they retain about missingness indicators:
• missingness-C-DMG (m-C-DMG): an abstract graph in which each variable inside a cluster
is associated with its own missingness indicator;
• clustered-missingness-C-DMG (cm-C-DMG): a coarser abstract graph in which a single
missingness vertex represents the missingness mechanism of the whole cluster. To this end, we introduce two abstract missingness representations, which differ
in the level of detail they retain about missingness indicators:
• missingness-C-DMG (m-C-DMG): an abstract graph in which each variable inside a cluster
is associated with its own missingness indicator;
• clustered-missingness-C-DMG (cm-C-DMG): a coarser abstract graph in which a single
missingness vertex represents the missingness mechanism of the whole cluster. This distinction is crucial because the two abstractions do not encode the same amount of information.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Compatibility and Recoverability Semantics
The authors define compatibility between abstract graphs and variable-level missingness models <ref:2605.20943#pg2>. Proposition 1 states that the inclusion of compatibility classes is strict whenever at least one cluster contains two variables with distinct missingness mechanisms <ref:2605. p. 4 Then compatible(G
C,m) ⊆ compatible(G
C,cm). Moreover,
the inclusion is strict whenever at least one cluster contains two variables with distinct missingness
mechanisms. The converse does not hold in general. The above proposition shows that there may exist a cm-C-DMG G
C,cm and a target query Q such that
Q is recoverable in every G
m ∈ compatible(G
C,cm), while Q is not recoverable from G
C,cm alone. The m-C-DMG preserves variable-specific missingness information within clusters, whereas the
cm-C-DMG merges these distinctions into a single cluster-level missingness vertex. As a consequence,
the cm-C-DMG is generally less informative than the m-C-DMG.Where the paper says it
Where the paper says it
Where the paper says it
Recovery Conditions for Queries
Theorem 1 provides necessary and sufficient conditions for recovering the joint distribution Pr(C) from an m-C-DMG or cm-C-DMG <ref:2605.20943#pg2>. The condition involves the absence of any vertex CX ∈ C m such that CX and its R-vertex are neighbors or connected by a path in which all intermediate vertices are colliders and elements of C m ∪ Co <ref:2605. p. 6 • CX and its R-vertex are neighbors
• CX and its R-vertex are connected by a path in which all intermediate vertices are colliders
and elements of C
m ∪ C
o
. When recoverable, Pr(c) is given by
Pr(c) =
Pr(R = 0, c)
∏i Pr(Ri = 0 ∣ MBo(Ri
,GC,∗m),MBm(Ri
,GC,∗m), RMBm(Ri,GC,∗m))
,
where Cm is the set of clusters in C containing at least one variable from Vm, Co = C ∖ Cm,
MBo
(Ri
,G
C,m) is the Markov Blanket of Ci
in Co, MBm(Ri
,G
C,m) is the Markov Blanket of Ci
in Cm. Theorem 2 (Recoverability of macro causal effects in m-C-DMGs and cm-C-DMGs). A causal
query Q is recoverable from an m-C-DMG or cm-C-DMG if there exists a sequence of applications
of do-calculus, probability manipulations, and Equation 1 that transforms Q into an expression
involving only observable quantities (namely proxy variables, missingness indicators, and fully
observed variables) and containing no do-operators.Where the paper says it
Where the paper says it
Conclusion and Societal Impact
The work contributes to the theoretical understanding of missing data problems under structural abstraction <ref:2605.20943#pg2>. The study provides a principled way to analyze settings where only coarse structural knowledge is available <ref:2605. p. 8 Studying missingness through causal abstractions has several advantages. First,
it provides a principled way to analyze settings where only coarse structural knowledge is available. Because m-C-DMGs and cm-C-DMGs are abstractions, they may
be conservative: a query that is not recoverable from the abstract graph may still be recoverable
in some compatible variable-level graph. Moreover, cm-C-DMGs can hide distinctions between
missingness indicators that are essential for recovery. For non-causal queries, we focused mainly on
recovering joint distributions; it would therefore be interesting to further investigate conditional
distributions. Finally, future work should extend the framework to the recovery of micro causal
effects under causal abstractions, building on recent work on identifiability from abstract causal
graphs [Assaad et al. Societal impacts
This paper presents work whose goal is to advance the field of Machine Learning and Epidemiology,
and the limits of what can be inferred from data. There are many potential societal consequences of
our work, none which we feel must be specifically highlighted here. Acknowledgement
This work was supported by the CIPHOD project (ANR-23-CPJ1-0212-01) as well as by the PPIMPACT project. References
Tara V. Variables are thus often grouped into clusters, either for interpretability
Preprint. arXiv:2605.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
References
Karthika Mohan and Judea Pearl Graphical models for recovering probabilistic and causal queries from missing data In Z. Ghahramani, M. p. 9 Graphical models for recovering probabilistic and causal queries
from missing data. Ghahramani, M. Ghahramani, M. Welling, C. Welling, C. Lawrence, and K. Weinberger,
editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc. Graphical models for processing missing data. Journal of the
American Statistical Association, 116(534):1023–1037, 2021. AAAI
Press, 2023. ISBN 978-1-57735-880-0. AAAI
Press, 2023. ISBN 978-1-57735-880-0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Karthika Mohan and Judea Pearl Graphical models for inference with missing data In C. p. 9 Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data. Ghahramani, M. Welling, C. Weinberger,
editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.Where the paper says it
Where the paper says it
Where the paper says it
Judea Pearl Causality Models Reasoning and Inference Cambridge University Press USA 2nd edition 2009 ISBN 052189560X
Donald B Rubin Inference and missing data Biometrika 63(3) 581–592 12 1976 ISSN 0006-3444 doi:10.1093/biomet/63. p. 9 ISSN 0006-3444. 1093/biomet/63.Where the paper says it
Werner Wothke Longitudinal and multigroup modeling with missing data In Modeling Longitudinal and Multilevel Data Practical Issues Applied Approaches Specific Examples Psychology Press 2000 doi:10.4324/9781410601940
Tara V Anand and George Hripcsak Leveraging cluster causal diagrams for determining causal effects in medicine AMIA Annual Symposium Proceedings 25 PMID: 40417561
Tara V Anand Adele H Ribeiro Jin Tian and Elias Bareinboim Causal effect identification in cluster dags In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence AAAI Press 2023 ISBN 978-1-57735-880-0 doi:10.1609/aaai.v37i10.26435
Charles K Assaad Imad Ez Zejjari and Lei Zan Root cause identification for collective anomalies in time series given an acyclic summary causal graph with loops In Francisco Ruiz Jennifer Dy and Jan-Willem van de Meent editors Proceedings of The 26th International Conference on Artificial Intelligence and Statistics volume 206 of Proceedings of Machine Learning Research pages 8395–8404 PMLR 25–27 Apr 2023
Charles K Assaad Emilie Devijver Eric Gaussier Gregor Goessler and Anouar Meynaoui Identifiability of total effects from abstractions of time series causal graphs In Negar Kiyavash and Joris M Mooij editors Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence volume 244 of Proceedings of Machine Learning Research pages 173–185 PMLR 15–19 Jul 2024
Simon Ferreira and Charles K Assaad Identifying macro conditional independencies and macro total effects in summary causal graphs with latent confounding Proceedings of the AAAI Conference on Artificial Intelligence 39(25) 26787–26795 Apr. 2025 doi:10. p. 0 , 2022, Assaad et al. , 2023,
Anand and Hripcsak, 2025].Where the paper says it
Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Application of the front-door formula Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.000000001477
Jonas Peters Dominik Janzing and Bernhard Schölkopf Causal inference on time series using restricted structural equation models In C.J. Burges L Bottou M Welling Z Ghahramani and K Weinberger editors Advances in Neural Information Processing Systems volume 26 Curran Associates Inc 2013
Karthika Mohan Judea Pearl Graphical models for recovering probabilistic and causal queries from missing data In Z Ghahramani M Welling C Cortes N Lawrence and K Weinberger editors Advances in Neural Information Processing Systems volume 27 Curran Associates Inc 2014
Karthika Mohan Judea Pearl Graphical models for processing missing data Journal of the American Statistical Association 116(534) 2021 doi:10.880/01621459.
Karthika Mohan Judea Pearl Graphical models for inference with missing data In C. p. 9 Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data.Where the paper says it
Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.000001477
Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.00001477
Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Data recovery from coarse structural information allows AI systems to perform valid inference even when full variable-level missingness models are unknown, clarifying
when cluster-level missingness information is sufficient for valid inference.
This enables reliable probabilistic and causal queries in situations with limited knowledge of underlying data-generating mechanisms. -
The introduction of the m-C-DMG and cm-C-DMG abstractions allows AI to model complex systems by retaining
variable specific missingness indicators
or aggregating them into a singlecluster level missingness vertex.
This provides a principled way to analyze settings where only coarse structural knowledge is available. -
AI systems can utilize the do-calculus, which has been shown to be sound for C-DMGs, to recover macro causal effects from abstract graphs. This enables the assessment of effects like
the effect of sexual behavior on the risk of acquiring HIV, even when data is partially observed.
-
The framework allows AI to determine if a target quantity Q is recoverable in an abstract graph by checking specific graphical conditions, such as
the absence of any vertex CX in C m such that CX and its R-vertex are neighbors
for joint distributions. This provides a clear decision boundary between when finer-grained modeling is necessary versus when abstraction suffices. -
AI systems can leverage the results showing that
recoverability from a cm-C-DMG implies recoverability from the corresponding m-C-DMG,
enabling researchers to use coarser, more compact abstractions for inference without losing all recoverability capabilities.
Abstract
Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences. Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available, for instance when variables are grouped into clusters due to limited knowledge or interpretability reasons. In this paper, we investigate recoverability from such abstract representations. We introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level. We formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models, and study how this abstraction affects the recoverability of probabilistic and causal queries. In particular, we give graphical conditions of recovering the joint distribution as well as graphical conditions of recovering a macro causal effect.
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes