Beyond Distributional Fidelity: Causal-Penalized Diffusion for Synthetic Tabular Data

arXiv:2610.11407 · stat.ML, cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Beyond Distributional Fidelity".

Jane: The gist The authors propose a causal-fidelity-aware training framework that adds a causal discrepancy penalty to generative objectives for synthetic tabular data,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We've just talked about how synthetic data generators are often optimized for looking statistically similar to real data, but this paper argues that statistical similarity alone isn't enough because causal effects might get distorted in the process >

Jane: That’s right. The core thesis of "Beyond Distributional Fidelity: Causal-Penalized Diffusion for Synthetic Tabular Data" is that we need a way to study whether causal fidelity can be improved directly within a fully generative tabular model >

Lu: They define Causal Fidelity as the discrepancy between inferential distributions obtained from real and synthetic data with respect to a target estimand, and they show this doesn't happen just because of high statistical fidelity >

Meng: Essentially, they found that high distributional fidelity doesn't generally imply high causal fidelity when dealing with tabular data, which is why they want to incorporate causal fidelity directly into the generative objective >

Tom: So the paper claims that instead of using separate evaluation methods, we can add a specific causal discrepancy penalty to the generative objective function itself >

Jane: They propose a causal-fidelity-aware training framework that augments the standard generative objective with this specific causal discrepancy penalty >

Lu: The framework is instantiated with a causal-penalized TabDDPM and optimized using an on-policy score-function estimator, which is smart because it doesn't require end-to-end differentiability >

Meng: That means they are able to incorporate a broad class of causal inference operators without needing the whole model to be perfectly differentiable for this part >

Tom: They formalize the loss as Lcausal(theta) equals the expected discrepancy across synthetic datasets from real and synthetic data with respect to a target estimand >

Jane: And they show that this framework improves causal fidelity while maintaining competitive distributional fidelity when tested on diverse treatment-effect simulations and benchmark datasets >

Lu: The key finding is that for diffusion-based models like TabDDPM, even with this penalty, there's still a systematic shift from the reference posterior, which suggests causal information isn't perfectly preserved by standard tabular generators >

Meng: It’s a practical finding because it tells us that standard tools might be missing something fundamental about how these models learn causal relationships in the first place >

Tom: So, this paper is essentially showing a path to improve the quality of synthetic data specifically for causal inference by directly modifying the training process >

Jane: It opens up possibilities for using these generated datasets more reliably for downstream treatment-effect analysis, which is a big deal in many fields >

Conclusion: Tom: So, wrapping up this look at "Beyond Distributional Fidelity: Causal-Penalized Diffusion for Synthetic Tabular Data," the authors are showing us that we can move beyond just making data look right statistically >

Jane: They're demonstrating a method where you can directly build causal fidelity into the training process using a penalty term to guide the generator toward better causal preservation >

Lu: The implication is that for tabular synthesis, we don't have to rely on external evaluation methods or just hope that statistical similarity translates into good causal estimates >

Meng: It means we can create synthetic data that is more trustworthy for applications where treatment effects are crucial, because the authors show it actually works across different model families >

Tom: The paper confirms that this framework improves causal fidelity while largely preserving distributional fidelity, which addresses the main tension in synthetic data generation >

Jane: So, in simple terms, they've shown how to train a generative model to be both statistically accurate and causally informed simultaneously >

Lan Tao, Yongxian He, Shirong Xu, Yidong Ouyang, Guang Cheng

Department of Statistics & Data Science, University of California, Los Angeles · Herbert Wertheim School of Public Health and Human Longevity Science, University of California, San Diego · Wang Yanan Institute for Studies in Economics, Xiamen University

stat.ML, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist The authors propose a causal-fidelity-aware training framework that adds a causal discrepancy penalty to generative objectives for synthetic tabular data, showing that this regularization

Key concepts

Causal Fidelity: This measures how well the inferential distributions derived from synthetic data match those from real data when estimating a specific causal effect. High statistical similarity does not mean high causal fidelity; this concept specifically targets the preservation of true causal relationships.
Causal Discrepancy Penalty: This is an added term to the generative training objective that penalizes the model if the synthetic data's distributions significantly differ from real data concerning a target causal estimand. It acts as a guide, encouraging the model to generate samples that respect known causal structures.
On-Policy Score-Function Estimator: Since calculating the exact gradient for this causal penalty is often difficult, the authors use an on-policy score-function estimator. This technique allows them to estimate the necessary gradients using sampled trajectories from the model's generation process, enabling training without needing a fully differentiable objective.
TabDDPM: This refers to a specific type of generative model—a diffusion model adapted for tabular data. The authors use this architecture as the foundation for their framework, modifying it by incorporating the causal discrepancy penalty into its overall training process.

Terminology

Fidelity — 90× in this paper · explained in 2 episodes
Causal Inference — 20× in this paper · explained in 4 episodes
Regularization — 12× in this paper · explained in 3 episodes
Generative Model — 8× in this paper · explained in 2 episodes
Benchmark — 7× in this paper · explained in 2 episodes
Generative Models — 7× in this paper · explained in 3 episodes
Wasserstein Distance — 6× in this paper · explained in 5 episodes
International Stroke Trial — 4× in this paper
Average Treatment Effect — 3× in this paper
Consistency — 3× in this paper · explained in 3 episodes
Diffusion Model — 3× in this paper · explained in 10 episodes
Embeddings — 2× in this paper · explained in 2 episodes
Foundation Model — 2× in this paper · explained in 3 episodes
Foundation Models — 2× in this paper · explained in 7 episodes
Latent Space — 2× in this paper · explained in 7 episodes
Model-Agnostic — 2× in this paper · explained in 2 episodes
Alignment — 1× in this paper
Benchmarking — 1× in this paper · explained in 4 episodes
Conditional Average Treatment Effect — 1× in this paper
Diffusion Models — 1× in this paper · explained in 9 episodes
Generalization — 1× in this paper · explained in 13 episodes
In-Context Learning — 1× in this paper · explained in 7 episodes
Infant Health and Development Program — 1× in this paper
Latent Representation — 1× in this paper · explained in 2 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Robustness — 1× in this paper · explained in 5 episodes

Summary

The gist The authors propose a causal-fidelity-aware training framework that adds a causal discrepancy penalty to generative objectives for synthetic tabular data, showing that this regularization improves causal fidelity while maintaining competitive distributional fidelity

Problem and Motivation

Synthetic tabular generators are often optimized for distributional fidelity, but statistical similarity alone does not guarantee preservation of causal effects. Causal Fidelity is defined as the discrepancy between inferential distributions obtained from real and synthetic data with respect to a target estimand, where high statistical fidelity does not generally imply high causal fidelity. Preliminary comparisons of tabular generators show that their ability to preserve the average treatment effect (ATE) posterior varies substantially across model families, suggesting that causal information is not reliably preserved by standard tabular generators. This motivates incorporating causal fidelity directly into the generative objective.

Proposed Framework

The authors formalize causal fidelity as the discrepancy between inferential distributions obtained from real and synthetic data with respect to a target causal estimand. They propose a causal-fidelity-aware training framework that augments the standard generative objective with a causal discrepancy penalty. Because this discrepancy need not be differentiable with respect to the generator parameters, they optimize it using an on-policy score-function estimator. This allows for the incorporation of a broad class of causal inference operators without requiring end-to-end differentiability.

Mathematical Formulation

The overall causal fidelity loss is quantified by the expected discrepancy across synthetic datasets: Lcausal(θ) = EDsyn∼Pbnsynθ[d(pbreal(τ), pbsyn(τ))]. The causal discrepancy of a synthetic dataset is defined as Dτ (Dreal, Dsyn) = d(pbreal(τ), pbsyn(τ)) where d is a distributional discrepancy such as the Wasserstein distance. The training objective becomes min θ Lgen(θ) + λLcausal(θ), where λ ≥ 0 controls the tradeoff between conventional generative fidelity and causal fidelity. The causal discrepancy acts as a dataset-level weight on the generator score, encouraging probability mass to shift away from regions associated with poor causal fidelity.

Training Procedure

The framework is instantiated with a causal-penalized TabDDPM optimized using an on-policy score-function estimator. For diffusion-based generators, the expected causal loss can be written as Lcausal(θ) = EZ∼Pθ Dτ (Dreal, h(Z)) where Z is the collection of stochastic reverse trajectories generated by the model. The causal gradient is estimated as a discrepancy-weighted average of the trajectory scores: gcausal = λ/m Xm j=1 StopGrad(Dj − b)sj where b is a moving-average baseline used to reduce variance.

Experimental Results

Simulation studies across three scenarios—linear outcome models, nonlinear outcome models, and heterogeneous treatment-effect models—show that the causal-penalized TabDDPM consistently improves causal fidelity across all settings compared with vanilla TabDDPM. In the real-world application on IHDP and IST datasets, causal regularization reduces the ATE posterior Wasserstein distance and posterior mean gap by 27.3% and 30.4% on IHDP, and by 22.7% and 24.5% on IST. These improvements are accompanied by relatively small changes in conventional distributional fidelity, demonstrating that causal regularization improves causal fidelity while maintaining competitive distributional fidelity.

Theoretical Bounds

Theorem 1 demonstrates that joint distributional proximity alone does not uniformly control ATE error, as the ATE discrepancy can remain fixed even as the joint TV distance approaches zero under pointwise overlap. Theorem 2 establishes a comparator bound, showing that for any comparator with Lcausal(¯θ) < Lcausal(θgen), Equation (11) provides a sufficient condition for strict improvement. The population ATE discrepancy is bounded by τeθ − τ ≤ esyn(θ) + Lcausal(θ) + ereal. The Gaussian example shows that every λ > 0 strictly reduces the ATE error relative to the unpenalized solution when β0 ≠ 0.

Conclusion

The authors conclude that they have formalized causal fidelity, introduced a causal-fidelity-aware training framework instantiated as a causal-penalized TabDDPM, and shown that this method improves causal fidelity while largely preserving distributional fidelity across diverse settings. Future work may extend the framework to broader causal estimands and other classes of tabular generative models. The AI Use Statement indicates that OpenAI Codex was used to assist with code development and deployment, and ChatGPT was used for manuscript editing, language polishing, and refinement of parts of the theoretical analysis. All AI-assisted outputs were reviewed and verified by the authors. The research ideas, methodology, experimental design, and conclusions were developed by the authors.

References

Akrami H., Aydore S., Leahy R. M., and Joshi A. A. (2020). Robust variational autoencoder for tabular data with beta divergence. arXiv preprint arXiv:2006.08204

Amad H., Qian Z., Frauen D., Piskorz J., Feuerriegel S., and van der Schaar M. (2026). Improving the generation and evaluation of synthetic data for downstream medical causal inference. Advances in Neural Information Processing Systems, 38:30435–30478

Asiaee A., Liang Z. J., and Yan C. (2026). Causalwrap: Model-agnostic causal constraint wrappers for tabular synthetic data. arXiv preprint arXiv:2603.02015

Balazadeh V., Kamkari H., Thomas V., Li B., Ma J., Cresswell J. C., and Krishnan R. G. (2025). Causalpfn: Amortized causal effect estimation via in-context learning. arXiv preprint arXiv:2506.07918

Borisov V., Seßler K., Leemann T., Pawelczyk M., and Kasneci G. (2022). Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280

Gruber S., Phillips R. V., Lee H., Ho M., Concato J., and van der Laan M. J. (2024). Targeted learning: Toward a future informed by real-world evidence. Statistics in Biopharmaceutical Research, 16(1):11–25

Hahn P. R., Murray J. S., and Carvalho C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3):965–1056

Hernandez M., Osorio-Marulanda P. A., Catalina M., Loinaz L., Epelde G., and Aginako N. (2025). Comprehensive evaluation framework for synthetic tabular data in health: fidelity, utility and privacy analysis of generative models with and without privacy guarantees. Frontiers in Digital Health, 7:1576290

Herurkar D., Ali A., and Dengel A. (2025). Evaluating generative models for tabular data: Novel metrics and benchmarking. arXiv preprint arXiv:2504.20900

Hill J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217–240

Horvitz D. G. and Thompson D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663–685

Imbens G. W. and Rubin D. B. (2015). Causal inference in statistics, social, and biomedical sciences. New York, 517

International Stroke Trial Collaborative Group (1997). The international stroke trial (ist): A randomised trial of aspirin, subcutaneous heparin, both, or neither among 19,435 patients with acute ischaemic stroke. The Lancet, 349(9065):1569–1581

Jordon J., Szpruch L., Houssiau F., Bottarelli M., Cherubin G., Maple C., Cohen S. N., and Weller A. (2022). Synthetic data–what, why and how? arXiv preprint arXiv:2205.03257

Kotelnikov A., Baranchuk D., Rubachev I., and Babenko A. (2023). Tabddpm: Modelling tabular data with diffusion models.

Improvements for AI systems

Bold header: Causal-Fidelity-Aware Training Framework

This framework augments standard generative training with a causal discrepancy penalty to explicitly encourage treatment effect preservation, as formalized by minimizing Lgen(θ) + λLcausal(θ). This allows the model to optimize for both distributional fidelity and causal fidelity simultaneously.

Bold header: On-Policy Causal Gradient Estimation

The framework utilizes an on-policy score-function estimator, specifically estimating the causal gradient as gcausal = λ/m Σj=1 StopGrad(Dj − b)s j, which allows for the incorporation of non-differentiable causal discrepancy measures. This enables the training of models like TabDDPM where gradients through inference operators are otherwise unavailable.

Bold header: Improved ATE Estimation Bounds

The theoretical analysis provides a bound on the expected absolute estimation error: E[τbsyn − τ] ≤ ereal + Lcausal(θ), and further bounds the population ATE discrepancy as τeθ − τ ≤ esyn(θ) + Lcausal(θ) + ereal. This allows researchers to quantify the trade-off between causal fidelity and estimation error.

Bold header: Robust Causal Inference on Complex Mechanisms

The method's performance is shown to improve in nonlinear and heterogeneous settings where vanilla TabDDPM fails, suggesting the system can reliably preserve treatment effects across increasingly complex causal mechanisms.

Bold header: Competitive Benchmark Performance

When applied to real-world datasets like IHDP and IST, Causal-TabDDPM achieves improvements of 27.3% and 30.4% on IHDP, and by 22.7% and 24.5% on IST in ATE posterior Wasserstein distance while maintaining competitive distributional fidelity metrics across both datasets.

Abstract

Synthetic tabular generators are commonly optimized for distributional fidelity, but statistical similarity alone does not guarantee preservation of causal effects. In this paper, we study whether causal fidelity can be improved directly within a fully generative tabular model. Causal Fidelity is defined with respect to a target estimand as the discrepancy between inferential distributions obtained from real and synthetic data, and theoretical results show that high statistical fidelity does not generally imply high causal fidelity. We then propose a causal-fidelity-aware training framework which adds a causal discrepancy penalty to the generative objective. The framework is instantiated with a causal-penalized TabDDPM and optimized using an on-policy score-function estimator. We further establish conditions under which causal regularization improves expected causal fidelity. Experiments across diverse treatment-effect simulations and two benchmark datasets evaluate the ability of our method to improve causal fidelity while preserving competitive statistical fidelity.