Robust Conformalized Selection with Noisy Responses
Chengyao Yu, Hongxin Wei, Bingyi Jing
stat.ML, cs.LG, stat.ME
Submitted: 2026-07-25
Code: https://github.com/ChengyaoYu1/Robust-Conformalized-Selection
License: http://creativecommons.org/licenses/by/4.0/
The gist: Conformalized selection has been widely applied to select high-quality candidates from large datasets with rigorous uncertainty quantification, such as reliable labeling, drug discovery, and the
Terminology
Abstract
Conformalized selection has been widely applied to select high-quality candidates from large datasets with rigorous uncertainty quantification, such as reliable labeling, drug discovery, and the alignment of large language models. Nevertheless, existing methods assume clean responses on calibration data, an assumption that rarely holds in practice. In this paper, we formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values. We demonstrate that existing conformal selection methods fail to control the false discovery rate (FDR) or suffer from severe power loss under contaminated calibration data. To that end, we propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination. The key insight of RCS lies in a novel statistical reduction: by separately conditioning on different classes, we translate the intractable label noise into a localized covariate shift problem, which then enables a covariate-adjusted empirical-Bayes-type estimate of the number of false selections. Statistical properties such as the asymptotic FDR control, power optimality, and robustness of RCS are established. We further develop an instantiation of RCS under randomized response model, and also apply RCS to the task of selecting candidates with large response values. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of RCS.
Sources
- Theoretical Foundations of Conformal Prediction
- Conformal Selective Prediction with General Risk Control
- Probably Approximately Correct Labels
- ACS: An interactive framework for conformal selection
- Selection from Hierarchical Data with Conformal e-values
- Feedback-Enhanced Online Multiple Testing with Applications to Conformal Selection
- Diversifying Conformal Selections
- A Conformal Prediction Score that is Robust to Label Noise
- Training Convolutional Networks with Noisy Labels
- A Unified Framework for Large-Scale Inference of Classification: Error Rate Control and Optimality
- A generalized e-value feature detection method with FDR control at multiple resolutions
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches