Missing data and cluster graphs: cluster-level missingness vs variable-level missingness

arXiv:2605.20943 · stat.ME · Submitted 2026-05-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Missing data and cluster graphs".

Jane: Here's the gist of the paper. Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences <ref:2605.20943#pg2> Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available <ref:2605.20943#pg2>.

Tom: In this paper, we investigate recoverability from such abstract representations <ref:2605.20943#pg2>.

Jane: The authors introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level <ref:2605.20943#pg2>.

Tom: They formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models and study how this abstraction affects the recoverability of probabilistic and causal queries <ref:2605.20943#pg2>.

Paper summary: Jane: So we've covered the main points of "Missing data and cluster graphs: cluster-level missingness vs variable-level missingness." We looked at how the m-C-DMG, which keeps variable indicators but clusters them causally, compares to the cm-C-DMG, which merges all indicators within a cluster into one vertex.

Tom: The paper shows that these two abstractions are not interchangeable; they encode different amounts of information, and the cm version is generally less informative for recovery. The main takeaway is that when you have missing data and only coarse structural knowledge, you need to be careful about which abstraction you use because it changes what queries you can answer.

Lu: It provides a principled way to analyze settings where we are forced to deal with uncertainty around relationships between variables because we don't know everything about them. It gives us specific conditions for when you can actually recover the joint distribution or a causal query from these abstract representations, which is really useful theory.

Meng: For deploying this in production, it means if your system relies on cluster summaries, you need to rigorously test whether that coarser representation preserves the necessary details for the queries you care about without introducing unacceptable errors at scale.

Jane: In simpler terms, this work helps us understand how much information we lose by simplifying the structure of our data representation when dealing with missing data in science and social studies. It’s a framework for making informed decisions about which level of structural detail is appropriate for a given problem.

Conclusion: Tom: So we're wrapping up this paper about missing data, and the title itself is pretty direct: "Missing data and cluster graphs: cluster-level missingness vs variable-level missingness."

Jane: It’s a really clear way to put it. Basically, they are comparing two different ways to look at missing information. One way looks at each individual variable separately, and the other way groups those variables into clusters first.

Lu: Exactly. They introduce two models, the m-C-DMG and the cm-C-DMG, which represent these different levels of abstraction—variable level versus cluster level.

Meng: But they’re not just drawing diagrams; they're defining how compatible those abstract graphs are with real missing data models. That compatibility stuff is where it gets interesting for deployment.

Tom: Right, so the core finding seems to be that this abstraction matters a lot for what you can actually figure out from the data later on.

Jane: They show that even when you only have a coarse structural idea of your data, like just cluster information, you can still recover certain things if the abstract graph is compatible in a specific way.

Lu: The paper establishes some strict rules for compatibility. If one cluster has variables with different missingness patterns, it’s much more powerful than just looking at the big cluster summary alone.

Meng: And thecm-C-DMG, which merges everything into one vertex per cluster, is generally less informative than the m-C-DMG because it loses those specific variable details.

Tom: So what does this mean practically? If you’re building a system that relies on these cluster summaries to make decisions, you need to know if you’re using the more detailed m version or the coarser cm version.

Jane: It means we have a principled way to choose our structural simplification based on what kind of query—like figuring out a joint distribution or a causal effect—we actually need to answer.

Lu: They give necessary and sufficient conditions for recovering those queries, which involves checking if certain vertices in the m-C-DMG are connected in a specific way.

Meng: From an engineering standpoint, that condition sounds like it’s going to be tricky to implement because you have to check paths through colliders and specific sets of vertices.

Tom: It sounds like they’re giving us the theoretical blueprint for when we can trust our simplified models, which is a big deal for real-world applications.

Jane: It really moves the conversation from just "how do we handle missing data?" to "what level of structure are we allowed to ignore, and what's the cost?" (SOUND of music slightly increasing in volume and then fading out)

Willow Scott, Eugenio Valdano, Charles K. Assaad

Sorbonne University · Inserm

stat.ME

Submitted: 2026-05-20

Updated: 2026-11-03

Comments: Accepted at NeurIPS 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 71/100

The gist: Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences <ref:2605.20943#pg2> Recoverability from missing data is typically studied…

Key concepts

m-C-DMG
This graph retains specific missingness indicators for each individual variable while representing causal relationships between variables at the cluster level. It is a more detailed abstraction that keeps track of how each variable is missing data.
cm-C-DMG
This is a coarser abstraction where all missingness indicators within a cluster are merged into a single vertex. It aggregates information at the cluster level, sacrificing variable-specific detail for simplicity in structural representation.
Compatibility
This defines how well an abstract graph (like m-C-DMG or cm-C-DMG) aligns with the underlying variable-level missingness models. Compatibility determines what kind of probabilistic or causal queries can be reliably recovered from the graph structure.
Recoverability
This refers to the ability to successfully reconstruct specific information, such as joint distributions or causal effects, from an abstract representation of missing data. The paper establishes conditions under which these queries can be recovered using either graph type.

Terminology

Summary

Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences <ref:2605. p. 0

Where the paper says it

assaad@inserm. fr Abstract Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences.

20943#pg2> Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available <ref:2605. p. 0
Where the paper says it

fr Abstract Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences. Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available, for instance when variables are grouped into clusters due to limited knowledge or interpretability reasons.

20943#pg2>. In this paper, we investigate recoverability from such abstract representations <ref:2605. p. 0
Where the paper says it

Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available, for instance when variables are grouped into clusters due to limited knowledge or interpretability reasons. In this paper, we investigate recoverability from such abstract representations.

20943#pg2>.

Introduction to Abstractions

The authors introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level <ref:2605.20943#pg2>. They formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models and study how this abstraction affects the recoverability of probabilistic and causal queries <ref:2605. p. 0

Where the paper says it

We introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level. We formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models, and study how this abstraction affects the recoverability of probabilistic and causal queries.

20943#pg2>. The goal is to understand how recoverability of queries – such as joint distributions and causal effects – depends on the level of abstraction <ref:2605. p. 1
Where the paper says it

While both representations encode missingness mechanisms at a higher level of abstraction, they differ in the amount of information they preserve, raising the question of how this impacts recoverability. Our goal is to understand how recoverability of queries – such as joint distributions and causal effects – depends on the level of abstraction.

20943#pg2>.

Defining the Missingness Graphs

The m-C-DMG retains variable-specific missingness indicators while representing causal relations between variables at the cluster level <ref:2605.20943#pg2>. The cm-C-DMG is a coarser abstraction that merges missingness indicators within each cluster into a single vertex <ref:2605. p. 1

Where the paper says it

The m-C-DMG retains variable-specific missingness indicators while representing causal relations between variables at the cluster level. The cm-C-DMG is a coarser abstraction that merges missingness indicators within each cluster into a single vertex.

20943#pg2>. Definition 2 describes the m-C-DMG, where each missingness indicator represents the missingness mechanism of the single variable X <ref:2605. p. 3
Where the paper says it

3 Missingness in cluster graphs A central objective of this paper is to investigate how missingness mechanisms can be incorporated into cluster graphs. To this end, we introduce two abstract missingness representations, which differ in the level of detail they retain about missingness indicators: • missingness-C-DMG (m-C-DMG): an abstract graph in which each variable inside a cluster is associated with its own missingness indicator; • clustered-missingness-C-DMG (cm-C-DMG): a coarser abstract graph in which a single missingness vertex represents the missingness mechanism of the whole cluster.

20943#pg2>. Definition 3 describes the cm-C-DMG, where a single missingness vertex represents a cluster <ref:2605. p. 3
Where the paper says it

3 Missingness in cluster graphs A central objective of this paper is to investigate how missingness mechanisms can be incorporated into cluster graphs. To this end, we introduce two abstract missingness representations, which differ in the level of detail they retain about missingness indicators: • missingness-C-DMG (m-C-DMG): an abstract graph in which each variable inside a cluster is associated with its own missingness indicator; • clustered-missingness-C-DMG (cm-C-DMG): a coarser abstract graph in which a single missingness vertex represents the missingness mechanism of the whole cluster.

20943#pg2>. The distinction is crucial because the two abstractions do not encode the same amount of information <ref:2605. p. 3
Where the paper says it

To this end, we introduce two abstract missingness representations, which differ in the level of detail they retain about missingness indicators: • missingness-C-DMG (m-C-DMG): an abstract graph in which each variable inside a cluster is associated with its own missingness indicator; • clustered-missingness-C-DMG (cm-C-DMG): a coarser abstract graph in which a single missingness vertex represents the missingness mechanism of the whole cluster. This distinction is crucial because the two abstractions do not encode the same amount of information.

20943#pg2>.

Compatibility and Recoverability Semantics

The authors define compatibility between abstract graphs and variable-level missingness models <ref:2605.20943#pg2>. Proposition 1 states that the inclusion of compatibility classes is strict whenever at least one cluster contains two variables with distinct missingness mechanisms <ref:2605. p. 4

Where the paper says it

Then compatible(G C,m) ⊆ compatible(G C,cm). Moreover, the inclusion is strict whenever at least one cluster contains two variables with distinct missingness mechanisms.

20943#pg2>. Proposition 2 shows that for any target query Q, Q is recoverable in G C,cm implies Q is recoverable in G m C <ref:2605. p. 5
Where the paper says it

The converse does not hold in general. The above proposition shows that there may exist a cm-C-DMG G C,cm and a target query Q such that Q is recoverable in every G m ∈ compatible(G C,cm), while Q is not recoverable from G C,cm alone.

20943#pg2>. This suggests that the cm-C-DMG is generally less informative than the m-C-DMG <ref:2605. p. 3
Where the paper says it

The m-C-DMG preserves variable-specific missingness information within clusters, whereas the cm-C-DMG merges these distinctions into a single cluster-level missingness vertex. As a consequence, the cm-C-DMG is generally less informative than the m-C-DMG.

20943#pg2>.

Recovery Conditions for Queries

Theorem 1 provides necessary and sufficient conditions for recovering the joint distribution Pr(C) from an m-C-DMG or cm-C-DMG <ref:2605.20943#pg2>. The condition involves the absence of any vertex CX ∈ C m such that CX and its R-vertex are neighbors or connected by a path in which all intermediate vertices are colliders and elements of C m ∪ Co <ref:2605. p. 6

Where the paper says it

• CX and its R-vertex are neighbors • CX and its R-vertex are connected by a path in which all intermediate vertices are colliders and elements of C m ∪ C o . When recoverable, Pr(c) is given by Pr(c) = Pr(R = 0, c) ∏i Pr(Ri = 0 ∣ MBo(Ri ,GC,∗m),MBm(Ri ,GC,∗m), RMBm(Ri,GC,∗m)) , where Cm is the set of clusters in C containing at least one variable from Vm, Co = C ∖ Cm, MBo (Ri ,G C,m) is the Markov Blanket of Ci in Co, MBm(Ri ,G C,m) is the Markov Blanket of Ci in Cm.

20943#pg2>. Theorem 2 states that a causal query Q is recoverable from an m-C-DMG or cm-C-DMG if there exists a sequence of applications of do-calculus, probability manipulations, and Equation 1 that transforms Q into an expression involving only observable quantities <ref:2605. p. 7
Where the paper says it

Theorem 2 (Recoverability of macro causal effects in m-C-DMGs and cm-C-DMGs). A causal query Q is recoverable from an m-C-DMG or cm-C-DMG if there exists a sequence of applications of do-calculus, probability manipulations, and Equation 1 that transforms Q into an expression involving only observable quantities (namely proxy variables, missingness indicators, and fully observed variables) and containing no do-operators.

20943#pg2>.

Conclusion and Societal Impact

The work contributes to the theoretical understanding of missing data problems under structural abstraction <ref:2605.20943#pg2>. The study provides a principled way to analyze settings where only coarse structural knowledge is available <ref:2605. p. 8

Where the paper says it

Studying missingness through causal abstractions has several advantages. First, it provides a principled way to analyze settings where only coarse structural knowledge is available.

20943#pg2>. Limitations include the possibility that m-C-DMGs and cm-C-DMGs may be conservative, and cm-C-DMGs can hide distinctions between missingness indicators that are essential for recovery <ref:2605. p. 8
Where the paper says it

Because m-C-DMGs and cm-C-DMGs are abstractions, they may be conservative: a query that is not recoverable from the abstract graph may still be recoverable in some compatible variable-level graph. Moreover, cm-C-DMGs can hide distinctions between missingness indicators that are essential for recovery.

20943#pg2>. Future work should extend the framework to the recovery of micro causal effects under causal abstractions <ref:2605. p. 8
Where the paper says it

For non-causal queries, we focused mainly on recovering joint distributions; it would therefore be interesting to further investigate conditional distributions. Finally, future work should extend the framework to the recovery of micro causal effects under causal abstractions, building on recent work on identifiability from abstract causal graphs [Assaad et al.

20943#pg2>. The paper presents work whose goal is to advance the field of Machine Learning and Epidemiology and has several potential societal consequences <ref:2605. p. 8
Where the paper says it

Societal impacts This paper presents work whose goal is to advance the field of Machine Learning and Epidemiology, and the limits of what can be inferred from data. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

20943#pg2>. The work was supported by the CIPHOD project (ANR-23-CPJ1-0212-01) as well as by the PPIMPACT project <ref:2605. p. 9
Where the paper says it

Acknowledgement This work was supported by the CIPHOD project (ANR-23-CPJ1-0212-01) as well as by the PPIMPACT project. References Tara V.

20943#pg2>. The paper is 2605. p. 0
Where the paper says it

Variables are thus often grouped into clusters, either for interpretability Preprint. arXiv:2605.

20943.

References

Karthika Mohan and Judea Pearl Graphical models for recovering probabilistic and causal queries from missing data In Z. Ghahramani, M. p. 9

Where the paper says it

Graphical models for recovering probabilistic and causal queries from missing data. Ghahramani, M.

Welling, C. p. 9
Where the paper says it

Ghahramani, M. Welling, C.

Cortes, N. Lawrence, and K. p. 9
Where the paper says it

Welling, C. Lawrence, and K.

Weinberger editors Advances in Neural Information Processing Systems volume 27 Curran Associates Inc 2014 doi:10. p. 9
Where the paper says it

Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.

1609/aaai.v37i10.26435 Mohan et al Graphical models for processing missing data Journal of the American Statistical Association 116(534) 2021 doi:10. p. 9
Where the paper says it

Graphical models for processing missing data. Journal of the American Statistical Association, 116(534):1023–1037, 2021.

880/01621459. p. 9
Where the paper says it

AAAI Press, 2023. ISBN 978-1-57735-880-0.

p. 9
Where the paper says it

AAAI Press, 2023. ISBN 978-1-57735-880-0.

Karthika Mohan and Judea Pearl Graphical models for inference with missing data In C. p. 9

Where the paper says it

Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data.

J. Burges, L. Bottou, M. Welling, Z. p. 9
Where the paper says it

Ghahramani, M. Welling, C.

Ghahramani, and K Weinberger editors Advances in Neural Information Processing Systems volume 26 Curran Associates Inc 2013 doi:10. p. 9
Where the paper says it

Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.

1609/aaai.v37i10.26435

Judea Pearl Causality Models Reasoning and Inference Cambridge University Press USA 2nd edition 2009 ISBN 052189560X

Donald B Rubin Inference and missing data Biometrika 63(3) 581–592 12 1976 ISSN 0006-3444 doi:10.1093/biomet/63. p. 9

Where the paper says it

ISSN 0006-3444. 1093/biomet/63.

3.581

Werner Wothke Longitudinal and multigroup modeling with missing data In Modeling Longitudinal and Multilevel Data Practical Issues Applied Approaches Specific Examples Psychology Press 2000 doi:10.4324/9781410601940

Tara V Anand and George Hripcsak Leveraging cluster causal diagrams for determining causal effects in medicine AMIA Annual Symposium Proceedings 25 PMID: 40417561

Tara V Anand Adele H Ribeiro Jin Tian and Elias Bareinboim Causal effect identification in cluster dags In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence AAAI Press 2023 ISBN 978-1-57735-880-0 doi:10.1609/aaai.v37i10.26435

Charles K Assaad Imad Ez Zejjari and Lei Zan Root cause identification for collective anomalies in time series given an acyclic summary causal graph with loops In Francisco Ruiz Jennifer Dy and Jan-Willem van de Meent editors Proceedings of The 26th International Conference on Artificial Intelligence and Statistics volume 206 of Proceedings of Machine Learning Research pages 8395–8404 PMLR 25–27 Apr 2023

Charles K Assaad Emilie Devijver Eric Gaussier Gregor Goessler and Anouar Meynaoui Identifiability of total effects from abstractions of time series causal graphs In Negar Kiyavash and Joris M Mooij editors Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence volume 244 of Proceedings of Machine Learning Research pages 173–185 PMLR 15–19 Jul 2024

Simon Ferreira and Charles K Assaad Identifying macro conditional independencies and macro total effects in summary causal graphs with latent confounding Proceedings of the AAAI Conference on Artificial Intelligence 39(25) 26787–26795 Apr. 2025 doi:10. p. 0

Where the paper says it

, 2022, Assaad et al. , 2023, Anand and Hripcsak, 2025].

1609/aaai.v39i25.3488

Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Application of the front-door formula Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.000000001477

Jonas Peters Dominik Janzing and Bernhard Schölkopf Causal inference on time series using restricted structural equation models In C.J. Burges L Bottou M Welling Z Ghahramani and K Weinberger editors Advances in Neural Information Processing Systems volume 26 Curran Associates Inc 2013

Karthika Mohan Judea Pearl Graphical models for recovering probabilistic and causal queries from missing data In Z Ghahramani M Welling C Cortes N Lawrence and K Weinberger editors Advances in Neural Information Processing Systems volume 27 Curran Associates Inc 2014

Karthika Mohan Judea Pearl Graphical models for processing missing data Journal of the American Statistical Association 116(534) 2021 doi:10.880/01621459.

Karthika Mohan Judea Pearl Graphical models for inference with missing data In C. p. 9

Where the paper says it

Karthika Mohan, Judea Pearl, and Jin Tian. Graphical models for inference with missing data.

J. Burges L Bottou M Welling Z Ghahramani and K Weinberger editors Advances in Neural Information Processing Systems volume 26 Curran Associates Inc 2013

Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.000001477

Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.00001477

Keisuke Inoue Beate Ritz and Onyebuchi A Arah Causal effect of chronic pain on mortality through opioid prescriptions Epidemiology 33(4) 544–552 2022 doi:10.1097/EDE.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Data recovery from coarse structural information allows AI systems to perform valid inference even when full variable-level missingness models are unknown, clarifying when cluster-level missingness information is sufficient for valid inference. This enables reliable probabilistic and causal queries in situations with limited knowledge of underlying data-generating mechanisms.

  2. The introduction of the m-C-DMG and cm-C-DMG abstractions allows AI to model complex systems by retaining variable specific missingness indicators or aggregating them into a single cluster level missingness vertex. This provides a principled way to analyze settings where only coarse structural knowledge is available.

  3. AI systems can utilize the do-calculus, which has been shown to be sound for C-DMGs, to recover macro causal effects from abstract graphs. This enables the assessment of effects like the effect of sexual behavior on the risk of acquiring HIV, even when data is partially observed.

  4. The framework allows AI to determine if a target quantity Q is recoverable in an abstract graph by checking specific graphical conditions, such as the absence of any vertex CX in C m such that CX and its R-vertex are neighbors for joint distributions. This provides a clear decision boundary between when finer-grained modeling is necessary versus when abstraction suffices.

  5. AI systems can leverage the results showing that recoverability from a cm-C-DMG implies recoverability from the corresponding m-C-DMG, enabling researchers to use coarser, more compact abstractions for inference without losing all recoverability capabilities.

Abstract

Missing data is pervasive in many scientific domains such as public health, environmental science, and the social sciences. Recoverability from missing data is typically studied using fully specified variable-level missingness models despite that, in many applications, only coarse structural information is available, for instance when variables are grouped into clusters due to limited knowledge or interpretability reasons. In this paper, we investigate recoverability from such abstract representations. We introduce two classes of cluster-based missingness graphs: the m-C-DMG, which retains variable-specific missingness indicators, and the cm-C-DMG, which aggregates missingness mechanisms at the cluster level. We formalize the notion of compatibility between these abstract graphs and underlying variable-level missingness models, and study how this abstraction affects the recoverability of probabilistic and causal queries. In particular, we give graphical conditions of recovering the joint distribution as well as graphical conditions of recovering a macro causal effect.

Related papers