Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry

The gist: This work proposes Maximum Mean Imprecision (MMI-CP) to quantify epistemic predictive uncertainty in conformal prediction by measuring conflicting information within the induced credal set, offering a more informative assessment than relying on prediction set size alone.

Theoretical Framework

The paper builds on the connection between conformal prediction and imprecise probabilities (IP), showing that any full CP procedure induces a credal set of predictive distributions under mild assumptions. This structural result is extended to split CP, which provides the mathematical foundation for quantifying EPU by measuring the degree of “conflicting information” encoded in the implicit predictive credal set. The authors introduce MMI-CP as an adaptation of the Maximum Mean Imprecision (MMI) functional to this conformal setting.

Quantifying Epistemic Predictive Uncertainty (EPU)

The paper distinguishes epistemic predictive uncertainty from aleatoric uncertainty by focusing on ambiguity at the level of the predictive model itself. This is contrasted with decision-based uncertainty, which concerns the reliability of a particular prediction or prediction set. The authors argue that epistemic predictive uncertainty can be quantified by measuring how strongly plausible predictive models disagree about future outcomes.

MMI-CP Quantification

The MMI functional is adapted to plausibility measures, denoted as Px(A) = supy∈A πx(y), which has a maxitive structure. Under consonance, this leads to a significant simplification where MMIHTV(Px) equals πσ(2), which denotes the second largest conformal p-value. This measure aggregates, across all confidence levels, the largest conformal p-values among labels excluded from the CPR.

Performance and Comparison

Experiments on active learning and selective classification demonstrate that MMI-based approaches provide substantially more informative and fine-grained uncertainty assessments than reliance on CPR size alone. In both settings, MMI-based methods consistently achieve higher performance on average and in many cases, statistically significantly outperform the set-size-based methods.

Limitations and Future Directions

The paper identifies a fundamental difference between EPU in classification and regression: classification admits instance-specific conformal p-value distributions due to label-space discreteness, whereas in regression, continuity and monotonicity of the score function eliminate test-specific variability. Furthermore, enforcing consonance precludes conformal prediction from producing empty prediction sets. The authors conclude that the negative results in regression reflect intrinsic properties of standard conformal regression rather than shortcomings of their approach. This work highlights the potential of CP serving as a principled basis for decision-making under epistemic uncertainty. The authors plan to extend their framework to other conformal settings such as conformal decision-making or time series problems in future work. The final results show that the optimal choice of α is problem-dependent. The paper concludes by emphasizing that MMI-based methods exhibit robust performance across different evaluation settings.

--- Page 40 ---

The gist: This work proposes Maximum Mean Imprecision (MMI-CP) to quantify epistemic predictive uncertainty in conformal prediction by measuring conflicting information within the induced credal set, offering a more informative assessment than relying on prediction set size alone.

How it works

1. The authors leverage recent results showing that under a mild assumption, any full CP procedure induces a credal set of predictive distributions.

2. They propose MMI-CP to quantify EPU by measuring the degree of “conflicting information” within this induced credal set.

3. For classification, MMIHTV(Px) is computed efficiently in O(K) time. The formula for EPU in classification is given by the discrete “gradients” of the p-value profile.

4. For regression, MMI exhibits a different phenomenon where the quantified EPU is constant across any prediction instances x under certain score function conditions.

Key Results

** The MMI-based approaches outperform set-size-based methods in both active learning and selective classification experiments. The performance gap between set-size-based methods and MMI-based methods narrows in higher-class problems. The authors also show that MMI is robust to the choice of nonconformity score. The paper demonstrates that enforcing consonance yields a stronger statistical guarantee known as Type II validity.

1. Establish a principled basis for decision-making under epistemic uncertainty by moving beyond reliance on prediction set sizes, as the CPR size at a fixed level α characterises uncertainty about the predicted region itself, rather than the uncertainty encountered when producing that prediction. The improved AI system can perform more reliable active learning and selective classification by using MMI-based measures to quantify epistemic predictive uncertainty, which provides substantially more informative and fine-grained uncertainty assessments than reliance on CPR size alone.

2. Implement a robust mechanism for quantifying model ambiguity by utilizing the Maximum Mean Imprecision (MMI) functional, which is shown to be a computationally efficient and analytically tractable uncertainty measure. This allows the system to measure the degree of conflicting information within the induced credal set, offering a principled separation from aleatoric uncertainty.

3. Enhance classification performance by leveraging the structure of conformal p-value profiles through the MMI-CPTV method, which is shown to yield superior results in selective classification experiments, outperforming set-size based approaches across various datasets and class sizes.

4. Improve uncertainty estimation for regression tasks by utilizing MMI-CP for score functions satisfying infy̸∈Cα(x) s(x, y) = qˆ1−α, which ensures that the quantified EPU is constant across any prediction instances x, providing a stable measure where traditional scores fail to distinguish test instances.

5. Strengthen the statistical guarantees of conformal procedures by enforcing the consonance assumption, as Consonance conformal transducers yield a conformal procedure that is uniformly valid, that is P(πXn+1 (Yn+1) ≤ α) ≤ α, upgrading the guarantee from marginal coverage to Type II validity.

Prediction-set size offers a coarse summary of conformal uncertainty. We develop a richer perspective through the geometry of credal sets: families of probability distributions compatible with conformal prediction (CP) regions. We establish this representation for split CP, extending a correspondence recently identified for full CP. Building on this representation, we derive analytical characterisations and exact algorithms for several widely used credal uncertainty measures, reducing potentially exponential computations to linear time after sorting the conformal p-values. We further unify several of these measures with classical conformal summaries by integrating set-size penalties based on Tsallis entropies across error levels. We also prove an impossibility theorem: for common split-conformal regression constructions, uncertainty summaries of the induced credal sets cannot distinguish between inputs. Classification experiments on selective prediction and distribution-shift detection show that these summaries often recover useful ranking information lost by prediction-set size at a fixed error level, enabling finer-grained uncertainty assessment. These results establish credal geometry as a principled framework for quantifying conformal uncertainty and understanding its limitations.