Uncertainty Quantification for Regression: A Unified Framework based on kernel scores

Regression tasks, notably in safety-critical domains, require reliable uncertainty quantification, yet the literature remains largely classificationfocused The framework provides a principled recipe for designing new uncertainty measures whose behavior, such as tail sensitivity or out-of-distribution responsiveness, is governed by the choice of the underlying kernel Extensive experiments across structured regression tasks, including spatial and functional domains, demonstrate effectiveness on downstream tasks such as out-of-distribution detection and active learning

Uncertainty Decomposition Framework

The paper introduces a family of measures for total, aleatoric, and epistemic uncertainty in multivariate regression based on strictly proper kernel scores The framework provides a principled recipe for designing new uncertainty measures whose behavior, such as tail sensitivity or out-of-distribution responsiveness, is governed by the choice of the underlying kernel This approach addresses the gap where existing research has focused either on uncertainty quantification in classification or on parametric univariate regression tasks, neglecting structured domains where generative models show great performance The authors transfer the score-divergence formulation of total, aleatoric, and epistemic uncertainty from classification to the multivariate regression setting

Kernel Scores as Design Lever

The key insight is that kernel scores inherit all the structural properties required for the decomposition in (3)–(4), while additionally providing closed-form expressions for a broad class of distributions, unbiased nonparametric estimators, and applicability to structured domains such as graphs or functional data The choice of kernel then acts as a design lever: we prove explicit connections between properties of the kernel and desirable behavior of the associated uncertainty measure, with regards to the assessment of uncertainties, translation invariance, and robustness Kernel scores are noted for inheriting a metric structure and coming with an unbiased, sample-based estimator

Properties of Kernel Scores

Kernel scores satisfy several crucial properties that translate directly to desirable uncertainty measures The properties of kernel scores described above carry over directly to the induced uncertainty measures Specifically, Proposition 5.1 states that for any proper scoring rule S, for which the map P -> S(P, Q) is convex for fixed Q, it holds that EU(Q) = 0 if and only if Q = δP Furthermore, Proposition 5.2 shows that any kernel score Sk with a translation invariant kernel k(x, x') that is convex in one of its arguments fulfills AU(δP1) ≤ AU(δP2), where P1 ≤cx P2

Instantiations and Performance

The paper proposes several specific instantiations of kernel scores for regression, including the Energy score (SES), the Gaussian kernel score (Skγ), and the Squared-error (SSE) The Energy score is highlighted as being strictly proper, translation invariant, and homogeneous, satisfying both Propositions 5.1 and 5.2 Empirical results across four experimental protocols demonstrate that Skγ achieves the best overall rank in selective prediction and active learning Specifically, Skγ remains stable under ensemble corruption whereas other measures degrade by orders of magnitude

Empirical Validation

The proposed measures are validated across various complex structured regression tasks, including depth estimation and the prediction of dynamical systems In robustness analysis, the Gaussian kernel score Skγ is noted as being robust against outlier members when k is bounded, while other measures degrade by orders of magnitude In out-of-distribution detection, the marginal score SCRPS shows the best performance across all datasets and uncertainty representation methods Overall, our proposed measures consistently outperform the baselines: on every task, at least one kernel-based measure outperforms both baselines The framework provides a general way to construct uncertainty measures that can be used with many different instantiations, leading to different properties

Conclusion

We propose a general framework for uncertainty quantification in regression, based on strictly proper kernel scores, which encompasses different uncertainty measures in a single principled construction In particular, it turns the design of uncertainty measures into the selection of an underlying kernel k, providing a systematic way for domain or task-specific constructions Our analysis shows how structural properties of kernels, such as strictly properness or boundedness, directly translate into distinct characteristics of the induced uncertainty measures Beyond theoretical contributions, our empirical results demonstrate the validity of the proposed measures, yielding consistently strong performance across diverse data modalities, uncertainty representations, and benchmark tasks The kernel-based (strictly proper) scoring rules demonstrate consistently strong performance across all evaluation protocols Skγ achieves the best overall rank in selective prediction and active learning, while SCRPS and SES lead in out-of-distribution detection The strictly proper kernel scores particularly outperform Slog and SSE for expressive predictive distributions beyond unimodal Gaussians, where sensitivity to distributional shape beyond the first two moments is critical

How it works

The paper defines uncertainty measures based on proper scoring rules, leading to TUB(Q), EUB(Q), and AUB(Q) Alternatively, pairwise estimators TUP(Q) and EUP(Q) are proposed, which satisfy the additive decomposition TU = EU + AU Kernel scores are introduced as the central tool of the framework, yielding tractable uncertainty measures whose closed-form expressions for Gaussian and mixture distributions are derived in Appendix B The divergence Dk is essentially the squared distance between the kernel mean embeddings of the probability distributions into some Hilbert space and is closely related to the Maximum Mean Discrepancy (MMD2)

Key Findings

The analysis proves that epistemic uncertainty vanishes if and only if all models agree, while aleatoric uncertainty increases with the variability of the predictive distribution Furthermore, neither measure can be destabilized by outlying ensemble members when k is bounded The performance comparison across datasets shows that Skγ achieves the best aggregated performance across all tasks, with SES and SCRPS obtaining slightly worse but comparable ranks The active learning task shows that Skγ obtains a significantly lower rank than the comparison methods

Limitations and Future Work

While our framework provides a principled foundation, it is not unique; alternative measures may satisfy the same properties Developing principled selection procedures for choosing or learning suitable kernels remains an important direction for future research Exploring alternative score constructions, such as weighted kernel scores [Allen et al., 2023], different kernel families (e.g., Laplace or inverse-multiquadratic), or learnable kernels could enable more targeted sensitivity to specific phenomena, such as extreme events Improving the bandwidth selection procedure within the kernel, for example, by mixing different scales, could further improve empirical performance Exploring other data domains, such as graph-structured data, where the usage of kernel scores could open up new possibilities of uncertainty quantification in molecule design Empirically, extending the evaluation to a broader range of generative models, including diffusion and flow-based architectures, may provide further insight into the practical behavior of kernel-based scores Further theoretical investigation of the relationship between kernel scores and maximum mean discrepancy could yield new insights into their geometric and statistical properties

References

Ben Adlam, Neha Gupta, Zelda Mariet, and Jamie Smith. Understanding the bias-variance tradeoff of Bregman divergences, 2022 Ferran Alet, Ilan Price, Andrew El-Kadi, Dominic Masters, Stratis Markou, Tom R. Andersson, Jacklynn Stott, Remi Lam, Matthew Willson, Alvaro Sanchez-Gonzalez., and Peter Battaglia. Skillful joint probabilistic weather forecasting from marginals Sam Allen, David Ginsbourger., and Johanna Ziegel. Evaluating forecasts for high-impact events using transformed kernel scores Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression Lucas Berry and David Meger. Normalizing Flow Ensembles for Rich Aleatoric and Epistemic Uncertainty Modeling Christopher M. Bishop. Mixture density networks Stephen Boyd and Lieven Vandenberghe. Convex Optimization Christopher Bülte, Philipp Scholl, and Gitta Kutyniok. Probabilistic neural operators for functional uncertainty quantification Christopher Bülte, Yusuf Sale, Timo Löhr, Paul Hofman, Gitta Kutyniok., and Eyke Hüllermeier. An Axiomatic Assessment of Entropy- and Variance-based Uncertainty Quantification in Regression Jieyu Chen, Tim Janke, Florian Steinke., and Sebastian Lerch. Generative machine learning methods for multivariate ensemble postprocessing A. P. Dawid. The geometry of proper scoring rules A. D´Isanto and K. L. Polsterer., Photometric redshift estimation via deep learning - generalized and pre-classificationless, image based, fully probabilistic redshifts Vineet Edupuganti, Morteza Mardani., and Shreyas Vasanawala. Uncertainty Quantification in Deep MRI Reconstruction Alexander Fishkov, Kajetan Schweighofer, Mykyta Ielanskyi., Nikita Kotelevskii., Mohsen Guizani., and Maxim Panov.

1. Bold header: Kernel-based Uncertainty Decomposition for Regression

The framework allows for the design of uncertainty measures into the selection of an underlying kernel k, which provides a principled recipe for creating measures whose behavior, such as tail sensitivity or out-of-distribution responsiveness, is governed by the choice of the underlying kernel. This enables practitioners to select kernels based on specific requirements like robustness and translation invariance.

2. Bold header: Robustness against Outlier Perturbations

The analysis shows that for a bounded kernel, the influence function is bounded, meaning any outlier in Q can only have a finite impact on the estimation of the uncertainty measure M. This guarantees robustness, as demonstrated by the result that for the Gaussian kernel score, Skγ remains stable across all distortion levels for both aleatoric and epistemic uncertainty.

3. Bold header: Task-Specific Uncertainty Measure Selection

The paper provides concrete design guidelines by proving explicit correspondences between properties of the kernel and desirable behavior of the associated uncertainty measure, yielding guidance for task-specific selection. For example, a practitioner can choose the Energy score (SES) if they require a measure that is strictly proper, translation invariant, and homogeneous, or the Gaussian kernel score (Skγ) if they prioritize robustness against ensemble corruption.

4. Bold header: Enhanced Out-of-Distribution Detection

The framework demonstrates superior performance in OOD detection, as evidenced by the result that for out-of-distribution detection, the marginal score SCRPS shows the best performance, followed by its multivariate version SES. This suggests that using a scoring rule sensitive to distributional shape beyond the first two moments is crucial for distinguishing ID from OOD data.

5. Bold header: Data-Efficient Active Learning

The framework provides a method for selecting new training instances based on epistemic uncertainty, as the active learning loop is then run for 20 rounds, after which a total of 25% of the data has been selected. This utilizes epistemic uncertainty as the selection criterion to minimize loss with as few labels as possible, leading to better final test loss compared to random baselines.

6. Bold header: Model-Agnostic Evaluation

The framework is designed such that the proposed measures are agnostic to the choice of predictive model, allowing for evaluation across diverse architectures like NG, DER, MDN, and SB using Table 1 to summarize performance across different uncertainty representation methods.

Regression tasks, notably in safety-critical domains, require reliable uncertainty quantification, yet the literature remains largely classification-focused. To address this, we introduce a family of measures for total, aleatoric, and epistemic uncertainty in multivariate regression based on strictly proper kernel scores. The framework provides a principled recipe for designing new uncertainty measures whose behavior, such as tail sensitivity or out-of-distribution responsiveness, is governed by the choice of the underlying kernel, while also encompassing existing measures under a joint analysis. We prove explicit correspondences between properties of the kernel and behavior of resulting uncertainty measures, yielding concrete design guidelines for practitioners. Extensive experiments across structured regression tasks, including spatial and functional domains, demonstrate effectiveness on downstream tasks such as out-of-distribution detection and active learning, and reveal that different kernel choices lead to distinct trade-offs, offering practitioners guidance for task-specific selection.