Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias
cs.LG, cs.AI
Submitted: 2026-09-25
Updated: 2026-09-25
Comments: An earlier version is available as a non-peer reviewed preprint, cited in the paper. This submission supersedes it with consent of all authors; all experiments are new
License: http://creativecommons.org/licenses/by/4.0/
The gist: Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal
Terminology
Abstract
Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.
Sources
- Invariant Risk Minimization
- Fairness Under Group-Conditional Prior Probability Shift: Invariance, Drift, and Target-Aware Post-Processing
- 10 Years of Fair Representations: Challenges and Opportunities
- Path-Specific Counterfactual Fairness
- Rethinking Distributional Matching Based Domain Adaptation
- The Variational Fair Autoencoder
- Learning Adversarially Fair and Transferable Representations
- Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation
- Are demographically invariant models and representations in medical imaging fair?
- On (assessing) the fairness of risk score models
- Stride-Net: Fairness-Aware Disentangled Representation Learning for Chest X-Ray Diagnosis
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks