Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration
cs.AI, cs.CL
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: 17 pages, 6 figures
Code: https://github.com/MoonshotAI/Kimi-K3
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task…
Terminology
Abstract
LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task, and post-hoc calibration (mapping LLM to human scores in a small labeled anchor set) has become a popular method to correct for this shift. In this paper we derive the calibration from two positions: that LLMaJ scores are ordinal ratings drawn from a distribution rather than isolated points. We present streaming quantile matching (SQM), a version of a classical method not previously bench marked for LLMaJ calibration, which is built on the two positions by computing the optimal-transport map between the LLM and human score distributions. We develop an evaluation framework and within it we show that unlike point-wise calibrations, SQM respects the ordinality of the human anchor set and achieves the best distributional fidelity and the top overall rank across models and data. We also show that in general, post-hoc calibration narrows the raw error gap between smaller and larger judge models, which opens the possibility of more efficient LLM judges.
Sources
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
- Large Language Models are not Fair Evaluators
- GPTScore: Evaluate as You Desire
- Calibrate Before Use: Improving Few-Shot Performance of Language Models
- Language Models (Mostly) Know What They Know
- A Survey on LLM-as-a-Judge
- LLM-based NLG Evaluation: Current Status and Challenges
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
- The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
- From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
- A Finite-Calibration Regime Map for LLM Judge Panels
- VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
- Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
- Post-hoc Alignment of LLM-judges to Human Judgment Distribution
- Distributional Process Reward Models: Calibrated Prediction of Future Rewards via Conditional Optimal Transport
- Instance-Optimal Estimation with Multiple LLM Judges on a Budget
- gpt-oss-120b & gpt-oss-20b Model Card
- Qwen3 Technical Report
- On Cost-Effective LLM-as-a-Judge Improvement Techniques
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents