Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data
summary
The gist
Multiple imputation and test-wise deletion are investigated as alternative solutions for causal discovery with incomplete cohort data The paper establishes necessary and sufficient conditions for the…
In short
The study compares test-wise deletion and multiple imputation for causal discovery when cohort data is incomplete. Test-wise deletion requires strict conditions for graph recoverability, while multiple imputation shows advantages with certain variable types but is less universally superior in complex settings.
Key concepts
- Test-wise Deletion
- This method estimates causal graphs by performing conditional independence tests on subsets of data that have complete information for the variables involved in each test. Its success depends on specific conditions like the admissible separator condition to ensure correct graph identification.
- Multiple Imputation
- This technique handles missing data by creating several complete datasets through imputation models, such as MICE. It is particularly useful when dealing with small numbers of Gaussian or discrete variables but struggles when the dataset contains a mix of measurement scales.
- Causal Discovery
- This is the process of estimating a causal graph—showing how variables influence each other—using observational data. The paper investigates how missing data impacts this estimation by comparing deletion and imputation strategies.
- Admissible Separator Condition
- A necessary and sufficient condition for test-wise deletion to successfully identify conditional dependencies in a causal graph. It requires that for any pair of non-adjacent nodes, there is a set of variables that completely separates them under the observed data.
Terminology used across episodes
This episode discusses
- Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data · Paper Radio
- Causal Discovery in the Presence of Missing Data
The paper
Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data · Read on arXiv
Leibniz Institute for Prevention Research and Epidemiology—BIPS · University of Bremen
Causal discovery algorithms estimate causal graphs from observational data. This can provide a valuable complement to analyses focussing on the causal relation between individual treatment-outcome pairs. Constraint-based causal discovery algorithms rely on conditional independence testing when building the graph. Until recently, these algorithms have been unable to handle missing values. In this paper, we investigate two alternative solutions: Test-wise deletion and multiple imputation. We establish necessary and sufficient conditions for the recoverability of causal structures under test-wise deletion, and argue that multiple imputation is more challenging in the context of causal discovery than for estimation. We conduct an extensive comparison by simulating from benchmark causal graphs: As one might expect, we find that test-wise deletion and multiple imputation both clearly outperform list-wise deletion and single imputation. Crucially, our results further suggest that multiple imputation is especially useful in settings with a small number of either Gaussian or discrete variables, but when the dataset contains a mix of both neither method is uniformly best. The methods we compare include random forest imputation and a hybrid procedure combining test-wise deletion and multiple imputation. An application to data from the IDEFICS cohort study on diet- and lifestyle-related diseases in European children serves as an illustrating example.
DOI: 10.1002/sim.9535
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data".
Jane: Here's the gist of the paper. Multiple imputation and test-wise deletion are investigated as alternative solutions for causal discovery with incomplete cohort data The paper establishes necessary and sufficient conditions for the recoverability of causal structures under test-wise deletion and argues that multiple imputation is more challenging in the context of causal discovery than for estimation <ref:2108.13331#pg5> Pruning redundant reasoning tokens Causal discovery algorithms estimate causal graphs from observational data, which can provide a valuable complement to analyses focusing on individual treatment-outcome pairs <ref:2108.13331#pg6> Constraint-based causal discovery algorithms rely on conditional independence testing when building the graph <ref:2108.13331#pg6> The paper investigates two alternative solutions: test-wise deletion and multiple imputation <ref:2108.13331#pg6> Test-wise deletion involves performing each conditional independence test using the subset of records containing complete data for all variables involved in that particular test <ref:2108.13331#pg6> Multiple imputation is shown to be especially useful in settings with a small number of either Gaussian or discrete variables, but neither method is uniformly best when the dataset contains a mix of both <ref:2108.13331#pg6> How the training signal is built The paper formally investigates and compares test-wise deletion and multiple imputation in the context of causal discovery <ref:2108.13331#pg7> Necessary and sufficient conditions for the recoverability of causal graphs under test-wise deletion are established building on Tu et al.
Tom: (2019) <ref:2108.13331#pg10> Multiple imputation is extended to discrete and mixed variables, characterizing situations where it outperforms test-wise deletion <ref:2108.13331#pg10> The comparison includes random forest imputation and a hybrid procedure combining test-wise deletion and multiple imputation <ref:2108.13331#pg6> Where the gains come from The performance comparison on simulated and real data shows that both test-wise deletion and multiple imputation clearly outperform list-wise deletion and single imputation <ref:2108.13331#pg6> Multiple imputation is expected to benefit from graphs with strong associations between variables, as the observed values then contain more information about the missing ones <ref:2108.13331#pg6> Test-wise deletion is expected to yield a denser graph than list-wise deletion due to the larger power of some tests <ref:2108.13331#pg10> The hybrid method proposed involves estimating a preliminary graph skeleton using test-wise deletion and then performing MICE imputation where the imputation model for variable V contains only V’s neighbours and the neighbours of the neighbours <ref:2108.13331#pg5> Detailed comparison on synthetic data The simulation study compared test-wise deletion and multiple imputation with different imputation models to guide practical choice <ref:2108.13331#pg6> For Gaussian variables, multiple imputation outperformed test-wise deletion in almost all ‘MCAR’ and ‘MAR’ scenarios, but not in the ‘MNAR’ scenarios <ref:2108.13331#pg6> For mixed variables, test-wise deletion outperformed multiple imputation in all simulation scenarios <ref:2108.13331#pg6> The results suggest that multiple imputation is particularly effective when missing values occur in the conditioning variable(s) <ref:2108.13331#pg6> Data application An application to data from the IDEFICS cohort study on diet- and lifestylerelated diseases in European children serves as an illustrating example <ref:2108.13331#pg7> The analysis used a specified partial node ordering and set sex and age as exogenous variables <ref:2108.13331#pg25> The graphs estimated were sparser than expected based on expert knowledge, possibly due to the small sample size or deviations from the CG assumption <ref:2108.13331#pg25> Test-wise deletion performed well overall in this application <ref:2108.13331#pg25> The multiple imputation methods tended to discover more edges adjoining ‘critical’ nodes than test-wide deletion <ref:2108.13331#pg26> Test-wise deletion relies on faithful observability and the admissible separator condition, whereas multiple imputation requires the missing values to be MAR <ref:2108.13331#pg8> Both assumptions are implied by the stronger MCAR but are otherwise difficult to justify in practice <ref:2108.13331#pg8> The paper concludes that while multiple imputation outperforms test-wise deletion in small graphs and Gaussian variables, there is no overall best approach in realistically complex settings with a larger number of variables especially when these are a mix of continuous and discrete measurements <ref:2108.13331#pg8> Random forest imputation or the hybrid method might be useful especially in settings with 50 or more variables <ref:2108.13331#pg8> Future research should address model selection of the imputation models in MICE <ref:2108.13331#pg8> Reliable learning of causal graphs from data with mixed measurement scales remains a challenge especially with the additional complication of missing values <ref:2108.13331#pg8> A Conditional independence testing Fisher’s z-test tests for a zero conditional correlation assuming that the variables in the test follow a multivariate Gaussian distribution <ref:2108.13331#pg7> The G2-test is a non-parametric conditional independence test for contingency tables <ref:2108.13331#pg7> The CG-test is a likelihood ratio test for conditional independence between CG-distributed variables <ref:2108.13331#pg7> Rules for combining likelihood ratio statistics have been suggested by Meng and Rubin (1992) <ref:2108.13331#pg30> The pooled test statistic D3 is used to combine the likelihood ratio statistics from multiple imputed datasets <ref:2108.13331#pg30> Identifiability of conditional (in)dependencies under test-wise deletion Under faithfulness and faithful observability, conditional dependencies are always identified under test-wise deletion <ref:2108.13331#pg6> A necessary and sufficient criterion for this type of identification is the admissible separator condition <ref:2108.13331#pg10> The admissible separator condition requires that for all pairs (X, Y) of non-adjacent nodes in V, there exists a set Z such that X ⊥⊥ Y (Z, RXY Z = 1) <ref:2108.13331#pg10> This condition is not empirically verifiable and arguably difficult to assess in practice <ref:2108.13331#pg10> References The paper cites numerous foundational works across causal inference, missing data methods, and graphical models <ref:2108.13331#pg7> Key references include Pearl (1988) for probabilistic reasoning and Rubin (2076) for inference and missing data <ref:2108.13331#pg6>, Meng and Rubin (1992) for combining likelihood ratio statistics with multiply imputed data sets <ref:2108.13331#pg30>, and Foraita et al.
Jane: (2020) for causal discovery of gene regulation with incomplete data <ref:2108.13331#pg7> Acknowledgements The authors gratefully acknowledge financial support by the German Research Foundation (DFG—Project DI 2372/1-1) <ref:2108.13331#pg5> The paper was supported by the German Research Foundation (DFG—Project DI 2372/1-1) <ref:2108.
Paper summary: Tom: So, we're looking at this paper, "Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data." It's about how to find the causal structure of a system when your data has missing pieces.
Jane: Right. Basically, the paper investigates two main ways to handle those missing values in causal discovery: test-wise deletion and multiple imputation. The core idea is they want to see which one works better when you don't have all your data points.
Lu: The authors set up a comparison by looking at simulated benchmark causal graphs. They establish necessary and sufficient conditions for when test-wise deletion can successfully recover the true causal structure, and they argue that multiple imputation is actually more difficult in this specific context than it is for just estimating a single effect > Page one of that work reads: "Causal discovery algorithms estimate causal graphs from observational data <ref:2108.13331#pg1,Causal discovery algorithms estimate causal graphs from observational data>. This can provide a valuable complement to analyses focussing on the causal relation between individual treatment-outcome pairs."
Meng: So, they are testing if test-wise deletion and multiple imputation are viable alternatives when standard constraint-based methods fail because of missing data > Page two of that work reads: "Constraint-based causal discovery algorithms rely on conditional independence testing when building the graph <ref:2108.13331#pg1,Constraint-based causal discovery algorithms rely on conditional independence testing when building>." I'm curious how much cleaner the recovery is with this approach.
Lalam: The paper claims test-wise deletion relies on faithfulness and faithful observability, which are conditions that guarantee you can always identify conditional dependencies under that method > Page six of that work reads: "Under faithfulness and faithful observability, conditional dependencies are always identified under test-wise deletion." It also points to the admissible separator condition as the necessary and sufficient criterion for this identification > Page ten of that work reads: "A necessary and sufficient criterion for this type of identification is the admissible separator condition <ref:2108.13331#pg3>."
Tom: That sounds like a big assumption there. If test-wise deletion works under those conditions, it's a solid way to build a graph without needing to fill in all the gaps > Page ten of that work reads: "The admissible separator condition requires that for all pairs (X, Y) of non-adjacent nodes in V, there exists a set Z such that X is not independent of Y given Z and RXY Z is one <ref:2108.13331#pg3>."
Paper summary: Jane: And then they compare this to multiple imputation. The paper suggests multiple imputation is especially useful when you have a small number of variables, like Gaussian or discrete ones > Page six of that work reads: "Multiple imputation is shown to be especially useful in settings with a small number of either Gaussian or discrete variables, but neither method is uniformly best when the dataset contains a mix of both."
Lu: But it doesn't mean one is universally better. They found that for Gaussian variables, multiple imputation actually outperformed test-wise deletion in almost all scenarios except when the data has missing values that are not simple missing at random > Page six of that work reads: "For Gaussian variables, multiple imputation outperformed test-wise deletion in almost all ‘MCAR’ and ‘MAR’ scenarios, but not in the ‘MNAR’ scenarios."
Meng: So, the paper shows a split result based on the type of data. When you have mixed measurements, test-wise deletion actually performed better than multiple imputation across all their simulation scenarios > Page six of that work reads: "For mixed variables, test-wise deletion outperformed multiple imputation in all simulation scenarios." That’s a key detail for deployment because we often deal with messy, real-world data.
Lalam: The paper also mentions the hybrid method they proposed. It involves first building a graph skeleton using test-wise deletion and then using multiple imputation, where the imputation model only looks at the neighbors of your target variable and their neighbors > Page five of that work reads: "The hybrid method proposed involves estimating a preliminary graph skeleton using test-wise deletion and then performing MICE imputation where the imputation model for variable V contains only V’s neighbours and the neighbours of the neighbours <ref:2108.13331#pg3>."
Tom: That sounds like a practical way to get the best of both worlds. It uses test-wise deletion to get a starting structure, and then imputation to refine it > Page five of that work reads: "The hybrid method proposed involves estimating a preliminary graph skeleton using test-wise deletion and then performing MICE imputation where the imputation model for variable V contains only V’s neighbours and the neighbours of the neighbours <ref:2108.13331#pg3>."
Jane: And when they looked at real data, test-wise deletion and multiple imputation both showed better performance compared to list-wise deletion or single imputation > Page six of that work reads: "The performance comparison on simulated and real data shows that both test-wise deletion and multiple imputation clearly outperform list-wise deletion and single imputation."
Paper summary: Lu: The application to the IDEFICS cohort study, which included diet and lifestyle factors in children, showed test-wise deletion performed quite well overall > Page twenty-five of that work reads: "Test-wise deletion performed well overall in this application." However, multiple imputation tended to discover more edges adjoining critical nodes than test-wise deletion > Page twenty-six of that work reads: "The multiple imputation methods tended to discover more edges adjoining ‘critical’ nodes than test-wise deletion."
Meng: So, the results are mixed depending on what you're trying to achieve. If you need a fast, reliable structure and you have mixed data types, test-wise deletion seems safer for establishing the initial graph > Page eight of that work reads: "Test-wise deletion relies on faithful observability and the admissible separator condition."
Lalam: But multiple imputation still has an edge when missing values happen in the conditioning variable itself, which is something they found important > Page six of that work reads: "Multiple imputation is expected to benefit from graphs with strong associations between variables, as the observed values then contain more information about the missing ones."
Tom: So for someone listening who just wants a high-level view, this paper boils down to this: there isn't one single best method for causal discovery when data is incomplete > Page eight of that work reads: "The paper concludes that while multiple imputation outperforms test-wise deletion in small graphs and Gaussian variables, there is no overall best approach in realistically complex settings with a larger number of variables especially when these are a mix of continuous and discrete measurements."
Jane: That makes sense. It suggests the choice depends heavily on the specific data you're working with, like whether it's all continuous or if it has those mixed measurements > Page eight of that work reads: "multiple imputation outperforms test-wise deletion in small graphs and Gaussian variables, but there is no overall best approach in realistically complex settings with a larger number of variables especially when these are a mix of continuous and discrete measurements."
Lu: And for more complex scenarios, like when you have fifty or more variables that are a mix, they suggest trying methods like random forest imputation or that hybrid procedure we mentioned earlier > Page eight of that work reads: "Random forest imputation or the hybrid method might be useful especially in settings with fifty or more variables."
Meng: From an engineering standpoint, I'd watch the cost of running those complex models. If you're going to use random forest imputation for fifty variables, you need to make sure it doesn't take too long to train every time > Page eight of that work reads: "Random forest imputation or the hybrid method might be useful especially in settings with fifty or more variables."
Paper summary: Lalam: The authors also flag a limitation—reliable learning of causal graphs from data with mixed measurement scales is still a challenge, and missing values add even more complexity > Page eight of that work reads: "Reliable learning of causal graphs from data with mixed measurement scales remains a challenge especially with the additional complication of missing values."
Tom: So, to sum up this paper on "Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data," it's showing us that neither method is perfect on its own > Page one of that work reads: "Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data August thirty-one two thousand twenty-one Janine Witte, Ronja Foraita, Vanessa Didelez at Leibniz Institute for Prevention Research and Epidemiology—BIPS two University of Bremen <ref:2108.13331#pg1,Multiple imputation and test-wise deletion for causal discovery with incomplete cohort>."
Jane: It's guiding us toward a more nuanced approach where we choose our technique based on the specific data characteristics, like the number of variables and whether they are continuous or discrete > Page eight of that work reads: "multiple imputation outperforms test-wise deletion in small graphs and Gaussian variables, but there is no overall best approach in realistically complex settings with a larger number of variables especially when these are a mix of continuous and discrete measurements."
Lu: The implications for causal discovery are that we can't just pick one fix; we need to consider the assumptions about missing data, like whether it's simple or not > Page eight of that work reads: "Test-wise deletion relies on faithful observability and the admissible separator condition, whereas multiple imputation requires the missing values to be MAR."
Meng: So for production systems, this means if you're dealing with a large set of mixed variables, you might need a hybrid method to balance initial structure discovery with robust filling in of those gaps > Page eight of that work reads: "The hybrid method proposed involves estimating a preliminary graph skeleton using test-wise deletion and then performing MICE imputation where the imputation model for variable V contains only V’s neighbours and the neighbours of the neighbours."
Lalam: The overall message is that learning causal graphs from messy, incomplete data is hard, and we need to keep testing those conditions like faithfulness or missing at random as we move forward > Page eight of that work reads: "Reliable learning of causal graphs from data with mixed measurement scales remains a challenge especially with the additional complication of missing values."
Conclusion: Tom: So we’re wrapping up this paper on "Multiple imputation and test-wise deletion for causal discovery with incomplete cohort data." It basically looks at two different ways to figure out cause and effect when your dataset has missing values.
Jane: Right, so it compares test-wise deletion, which just skips the bad data points for each test, against multiple imputation, where you fill in the gaps using statistical models. What are the main takeaways from this comparison?
Lu: The paper establishes specific conditions for when test-wise deletion can actually find a correct causal graph under faithfulness and observational assumptions. It also shows that multiple imputation has its own strengths, especially with small sets of variables, but it's not always better.
Meng: I'm hearing that test-wise deletion is more reliable when you have mixed data types, like continuous and discrete measurements. That’s important because real-world data isn't usually perfectly clean.
Tom: Exactly. It suggests that test-wise deletion relies on some strict rules about how the variables relate to each other, like the admissible separator condition. If those rules hold, you can reliably identify those conditional dependencies without needing to fill in every single missing piece of information with a complex model.
Jane: But then we have multiple imputation which works better for Gaussian data under simple missing data assumptions, but it struggles when the missingness is more complex than just being random. The authors point out that if the missing values aren't simple random, things get tricky again.
Lu: And they admit that there isn't one single best approach for all situations. For very complex settings with fifty or more variables and a mix of measurements, both methods have their limits.
Meng: So what does this mean for us when we’re building production systems? We can't just pick one method blindly; the choice depends entirely on the structure of your data and how much you can afford to run those imputation models.
Tom: Right. It points toward using a hybrid approach, where you get a preliminary graph skeleton from test-wise deletion and then use multiple imputation just for refining the details around critical nodes. That seems like a practical path forward.
Jane: And that’s what we’re hearing: the paper doesn't give us one magic fix, but it tells us which tool fits which job better depending on the data structure you’re dealing with. Next up, we look at how those underlying conditional independence tests actually work to test for those relationships.
More episodes
- Daily Summary for 2026-11-19
- 2310.07973-Statistical Performance Guarantee for Subgroup Identification with Generic Machine Learning
- 2408.09155-Learning Robust Treatment Rules for Censored Data
- Daily Summary for 2026-11-19
- 2603.23397-Kinetic Langevin Splitting Schemes for Constrained Sampling
- 2606.09906-An information-geometric framework for mapping maximum potential biodiversity
- 2603.27718-Induced replication and the assessment of models
- 1307.0898-Entropy of a Zipfian Distributed Lexicon
- 2406.19974-Generalizing self-normalized importance sampling with couplings
- 2412.10213-Collaborative Optimal Design of Controlled Experiments with Subject Covariates