Conformalized Credal Regions for Classification with Ambiguous Ground Truth
stat.ML, cs.LG
Submitted: 2024-11-07
Updated: 2026-10-29
Journal ref: TMLR 2025
License: http://creativecommons.org/licenses/by/4.0/
The gist: An open question in Imprecise Probabilistic Machine Learning is how to empirically derive a credal region (i.e…
Terminology
Abstract
An open question in Imprecise Probabilistic Machine Learning is how to empirically derive a credal region (i.e., a closed and convex family of probabilities on the output space) from the available data, without any prior knowledge or assumption. In classification problems, credal regions are a tool that is able to provide provable guarantees under realistic assumptions by characterizing the uncertainty about the distribution of the labels. Building on previous work, we show that credal regions can be directly constructed using conformal methods. This allows us to provide a novel extension of classical conformal prediction to problems with ambiguous ground truth, that is, when the exact labels for given inputs are not exactly known. The resulting construction enjoys desirable practical and theoretical properties: (i) conformal coverage guarantees, (ii) smaller prediction sets (compared to classical conformal prediction regions) and (iii) disentanglement of uncertainty sources (epistemic, aleatoric). We empirically verify our findings on both synthetic and real datasets.
Sources
- Uncertainty in Natural Language Generation: From Theory to Applications
- Imprecise Markov Semigroups and their Ergodicity
- Credal Two-Sample Tests of Epistemic Uncertainty
- Distributionally Robust Statistical Verification with Imprecise Neural Networks
- A Comprehensive Survey on Evidential Deep Learning and Its Applications
- Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' View
- Deep Residual Learning for Image Recognition
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?
- Conformal Credal Self-Supervised Learning
- On the Robustness of Kernel Goodness-of-Fit Tests
- IBCL: Zero-shot Model Generation under Stability-Plasticity Trade-offs
- Valid and efficient imprecise-probabilistic inference with partial priors, II. General framework
- Conformal prediction under ambiguous ground truth
Related papers
- Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
- Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
- Quantifying Error Propagation and Model Collapse in Diffusion Models
- Behavior of prediction performance metrics with rare events
- Statistical Inference and Learning for Shapley Additive Explanations (SHAP)
- Control Variate Design for Score-Function Gradient Estimators in Importance-Weighted Variational Inference