A Three-Way Testing Framework for Quantifying Epistemic Calibration Uncertainty in SBI
stat.ME, stat.ML
Submitted: 2026-09-21
Updated: 2026-09-21
Code: https://github.com/Monoxido45/three_way_CP4SBI
License: http://creativecommons.org/licenses/by/4.0/
The gist: Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods.
Terminology
Abstract
Current experimental scientists increasingly rely on simulation-based inference (SBI) to invert complex models with intractable likelihoods. A primary goal in these settings is to obtain credible regions with valid coverage. While recent model-agnostic conformal calibration methods have succeeded in constructing credible sets with prescribed local Bayesian coverage, their approximate nature introduces inherent epistemic uncertainty in the calibration process. In this work, we propose a novel tool for diagnosing calibration uncertainty. Our approach is based on a simple three-way hypothesis testing procedure. We demonstrate how this tool can be used to effectively assess necessary simulation budgets for calibration sets and analyze the epistemic uncertainty associated with cutoff estimation.
Sources
- Approximate Bayesian Computation with Deep Learning and Conformal prediction
- Conformal Calibration of Statistical Confidence Sets
- CP4SBI: Local Conformal Calibration of Credible Sets in Simulation-Based Inference
- Trustworthy scientific inference with generative models
- Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
- Simulation-Based Inference: A Practical Guide
- Diffusion posterior sampling for simulation-based inference in tall data settings
- Variational Inference with Coverage Guarantees in Simulation-Based Inference
Related papers
- Doubly robust inference via calibration
- Bayesian Empirical Bayes: Simultaneous Inference from Probabilistic Symmetries
- Optimal Watermark Localization in Mixed-Source Large Language Model Texts
- Flexible Nonparametric Inference for Causal Effects under the Front-Door Model
- A Greedy PDE Router for Blending Neural Operators and Classical Methods
- Policy Learning with Weak Signals