Full Conformal Prediction under Stochastic Non-Conformity Measure
Thanawat Sornwanee
math.ST, stat.ML
Submitted: 2026-06-27
Comments: ICML 2026 Hypothesis Testing Workshop
License: http://creativecommons.org/licenses/by/4.0/
The gist: The theory of full conformal prediction uses deterministic non-conformity measure, but modern usage of full conformal prediction often relies on machine learning training, making stochasticity
Terminology
Abstract
The theory of full conformal prediction uses deterministic non-conformity measure, but modern usage of full conformal prediction often relies on machine learning training, making stochasticity inevitable. A simple sufficient condition of almost sure permutation invariance of the non-conformity measure can be too restrictive, so many have suggested the relaxation to permutation in distribution as a condition for full conformal prediction validity. We, however, show that this commonly known condition is actually insufficient. We then provide a correct sufficient condition: Conditional Independence & Permutation Invariance in Distribution, which encompasses several stochastic settings that may be used in machine learning.
Sources
- Optimized Conformal Selection: Powerful Selective Inference After Data-Dependent Score Construction
- conformalClassification: A Conformal Prediction R Package for Classification
- Full-conformal novelty detection
- Distribution-free inference with hierarchical data
Related papers
- Conformal Prediction for Dyadic Regression Under Complex Missingness
- Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel
- Bentkus-type asymptotic e-values
- High-Dimensional Asymptotics of Differentially Private PCA
- A proximal gradient algorithm for composite log-concave sampling
- KL Convergence Guarantees for Score diffusion models under minimal data assumptions