Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models
Duong Bach, Hai Nguyen Hong, Cuong Do
cs.LG, cs.AI, stat.ML
Submitted: 2026-08-05
Code: https://github.com/DngBack/CS-WAE
License: http://creativecommons.org/licenses/by/4.0/
The gist: Factorized generative models commonly regularize a latent style variable z s by matching its marginal distribution to a fixed Gaussian prior and interpret this as evidence that the style
Terminology
Abstract
Factorized generative models commonly regularize a latent style variable z s by matching its marginal distribution to a fixed Gaussian prior and interpret this as evidence that the style representation is independent of class information. We show that this interpretation is incorrect. Matching only the marginal distribution places no constraint on the class-conditional distributions, allowing the latent style to remain highly predictive of the label despite appearing perfectly Gaussian in aggregate. We derive an exact decomposition showing that this mismatch is one of four conditions required for factorized sampling, and demonstrate that eliminating it is necessary but not sufficient to obtain the intended factorization. Empirically, our case-study model and four representative latent baselines achieve near-zero global MMD while still allowing a linear probe to recover class labels with 74%--100% accuracy (10% chance level). Our model reaches 99.15% clustering accuracy, whereas externally evaluated class-conditional generation succeeds only 16% of the time. This leakage remains under six independent perturbations involving model capacity, curriculum, prior geometry, and supervision across two datasets. Four mitigation strategies reduce probe accuracy to 21%--46%, although they leave within-class dependence largely unchanged. A post-hoc conditional prior improves externally evaluated class generation to 0.97 on MNIST without retraining but reaches only 0.41 on CIFAR-10, while an empirical style bank achieves 0.88 on CIFAR-10. These results demonstrate that no divergence computed solely on the marginal distribution of the style latent can certify independence from class labels, and that reporting marginal statistics alone does not verify the property commonly claimed in factorized generative models.
Sources
- Isolating Sources of Disentanglement in Variational Autoencoders
- Hyperspherical Variational Auto-Encoders
- Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
- Disentangling by Factorising
- Auto-Encoding Variational Bayes
- Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
- The Variational Fair Autoencoder
- Open-Ended Content-Style Recombination Via Leakage Filtering
- Spherical Cauchy Variational Autoencoders: Heavy Angular Tails and Exact KL Evaluation
- Wasserstein Auto-Encoders
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Unsupervised Deep Embedding for Clustering Analysis
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks