Position: Causal Machine Learning Requires Rigorous Synthetic Experiments for Broader Adoption
cs.LG, cs.AI, stat.ME, stat.ML
Submitted: 2025-08-12
Updated: 2025-08-12
Comments: Accepted at ICML 2025
Journal ref: Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:81995-82015, 2025
Code: https://github.com/panispani/causalml-needs-synth-eval
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Causal machine learning has the potential to revolutionize decision-making by combining the predictive power of machine learning algorithms with the theory of causal inference.
Terminology
Abstract
Causal machine learning has the potential to revolutionize decision-making by combining the predictive power of machine learning algorithms with the theory of causal inference. However, these methods remain underutilized by the broader machine learning community, in part because current empirical evaluations do not permit assessment of their reliability and robustness, undermining their practical utility. Specifically, one of the principal criticisms made by the community is the extensive use of synthetic experiments. We argue, on the contrary, that synthetic experiments are essential and necessary to precisely assess and understand the capabilities of causal machine learning methods. To substantiate our position, we critically review the current evaluation practices, spotlight their shortcomings, and propose a set of principles for conducting rigorous empirical analyses with synthetic data. Adopting the proposed principles will enable comprehensive evaluations that build trust in causal machine learning methods, driving their broader adoption and impactful real-world use.
Sources
- CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data
- The Benchmark Lottery
- Learned Causal Method Prediction
- Counterfactual (Non-)identifiability of Learned Structural Causal Models
- RealCause: Realistic Causal Inference Benchmarking
- Are you doing better than random guessing? A call for using negative controls when evaluating causal discovery algorithms
- Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis
- Hierarchical Bias-Driven Stratification for Interpretable Causal Effect Estimation
- Doubly robust inference via calibration
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis