Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees
Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang
cs.LG, stat.ML
Submitted: 2026-08-03
Comments: Accepted by ICLR 2026
Code: https://github.com/CAMELLIAxt/12500_
License: http://creativecommons.org/licenses/by/4.0/
The gist: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.
Terminology
Abstract
Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregate-then-Calibrate (AtC), a two-stage framework that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into a consensus ranking using a rank-aggregation model that accounts for annotator reliability. Stage-2 calibrates any predictive model's scores by an isotonic projection onto the order, enforcing ordinal consistency while preserving as much of the model's quantitative information as possible. Theoretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymptotically outperforms model-only assessment. Across semi-synthetic and real-world datasets, AtC consistently improves accuracy and robustness over human-only or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.
Sources
- WED-Net: A Weather-Effect Disentanglement Network with Causal Augmentation for Urban Flow Prediction
- Mitigating Cognitive Biases in Multi-Criteria Crowd Assessment
- Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
- Spatio-Temporal Hierarchical Causal Models
- $L_p$ Isotonic Regression Algorithms Using an $L_0$ Approach
- Differentially Private (Gradient) Expectation Maximization Algorithm with Statistical Guarantees
- Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks