Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence".
Title and authors: Tom: So we're looking at this paper, "Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence." It tackles the problem of how we use multiple judges to rank things in AI evaluation.
Jane: It sounds like it's trying to move beyond just averaging those scores into one single number.
Tom: Exactly. Traditional methods pool all the judgments into one score vector and treat any difference between judges as just noise, which I think is a big problem when you need to know *why* two judges disagree.
Lu: What this paper proposes is a structured framework that separates that total ranking into three distinct parts: consensus ranking, judge-specific sensitivity to that consensus, and residual preference disagreement.
Jane: So it’s not just one overall score anymore; it’s breaking down the judgment process itself into these different types of signals.
Meng: I’m curious about how they handle the practical side of this, because when we deploy models at scale, we need to know what actually breaks under these different signals.
Tom: Right. The paper sets up mathematical conditions that show how this decomposition is identifiable, and then they develop an alternating algorithm to find the best representation of these components.
Lu: The authors show that the parameters they are estimating actually map directly onto features of how the data is generated, which is a pretty strong result because it grounds the theory in something real.
Jane: That’s interesting—so instead of just fitting some abstract model to the scores, they're identifying things about how the AI was trained or how those preference data points were created.
Tom: And for anyone running an evaluation pipeline, this gives us a way to diagnose where the inconsistencies are coming from, not just report a single score.
Meng: Diagnosing is good, but what about cost? If we have to run this more complex inference process to get those three separate targets, does it make the deployment slower or more expensive for production models?
Lalam: From my perspective as a model that processes this literature, I see that modeling judge-specific latent scores through this structured decomposition is what allows the system to capture how different judges systematically view certain items or responses <ref:2605.05073#pg2>.
Tom: It captures those systematic differences, which is exactly what we need to know when we see judges preferring one model over another in a specific way across different tasks.
Jane: And they also introduce uncertainty quantification, which means they can give us confidence statements for the consensus scores and even for those judge-specific rankings.
Meng: Confidence statements are useful, but how robust is that confidence when we move to real traffic where the judge panel isn't perfectly balanced?
Lu: The paper studies this using a fixed-panel repeated-comparison regime, which means they’re looking at uncertainty statements for consensus and judge-specific ranking contrasts.
Tom: And they quantify that uncertainty using what they call the Wald interval based on the plug-in Fisher information matrix, which is a standard way to get those confidence bounds in statistical inference.
Title and authors: Jane: So when we look at near-tie settings where things are fragile, this framework gives us a statistically grounded way to say how much we can trust our ranking estimate.
Meng: It’s better than just saying "we're confident," because it’s tied to the information content of the judgments themselves.
Lalam: And they show that this approach improves recovery and robustness when compared against pooled methods and sensitivity-only baselines in synthetic and real data.
Tom: I like that they showed improved accuracy on empirically ambiguous near-tie pairs, which is something I’ve seen struggle with in previous work.
Jane: So to summarize, the main improvement here is moving from treating disagreement as noise to explicitly modeling consensus ranking, judge sensitivity, and structured disagreement separately.
Meng: That separation helps us see that judges aren't just randomly flipping scores; they have different ways of responding to the same underlying signal.
Lu: And by doing this, we can actually flag high-leverage judges through metrics like residual leverage which points to those who show both strong consensus discrimination and a lot of off-consensus structure.
Tom: That sounds like a practical tool for auditing our evaluation sets, helping us pinpoint which judges are providing the most unique or potentially biased input.
Jane: It’s also useful because they show that this model can recover judge-specific rankings even when the pairwise data is sparse or noisy.
Meng: When I think about deployment, I wonder how much computational overhead we need to add for this structure compared to just running a standard ranking pipeline.
Lalam: The paper suggests that this structured optimization and reanchor step is necessary to stabilize the decomposition and actually improve the score-level recovery, which implies it’s worth the computation if it gets us better estimates.
Tom: So, we have a framework that separates what's common from what's judge-specific noise, and gives us a way to measure our confidence in those estimates.
Jane: It’s a powerful diagnostic layer for evaluation before we even start designing new benchmarks.
Meng: I just think it makes the whole process more transparent about the inherent heterogeneity in human judgment.
Lu: The analysis of those judge-model interaction terms, specifically the residual component, suggests that most disagreement is actually systematic and coherent rather than random noise across different models.
Lalam: I'm confident in this direction because the consistency of their constrained maximum likelihood estimator is proven by showing that the normalized negative log-likelihood converges uniformly on compact subsets of S to the true data generating distribution.
Tom: So, we’ve got a framework that separates consensus ranking, judge sensitivity, and structured residual disagreement. It's designed to be statistically grounded for heterogeneous evaluations under challenging judgments.
Jane: It sounds like a solid piece of work that gives us much better tools for understanding the messy reality of human evaluation in AI.
Tom: Yeah, it’s about getting the numbers behind why we get those disagreements in the first place.
The paper's summary: Tom: So we're looking at this paper, "Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence." It tackles the problem of how we use multiple judges to rank things in AI evaluation.
Jane: It sounds like it's trying to move beyond just averaging those scores into one single number.
Tom: Exactly. Traditional methods pool all the judgments into one score vector and treat any difference between judges as just noise, which I think is a big problem when you need to know *why* two judges disagree.
Lu: What this paper proposes is a structured framework that separates that total ranking into three distinct parts: consensus ranking, judge-specific sensitivity to that consensus, and residual preference disagreement.
Jane: So it’s not just one overall score anymore; it’s breaking down the judgment process itself into these different types of signals.
Meng: I’m curious about how they handle the practical side of this, because when we deploy models at scale, we need to know what actually breaks under these different signals.
Tom: Right. The paper sets up mathematical conditions that show how this decomposition is identifiable, and then they develop an alternating algorithm to find the best representation of these components.
Lu: The authors show that the parameters they are estimating actually map directly onto features of how the data is generated, which is a pretty strong result because it grounds the theory in something real.
Jane: That’s interesting—so instead of just fitting some abstract model to the scores, they're identifying things about how the AI was trained or how those preference data points were created.
Tom: And for anyone running an evaluation pipeline, this gives us a way to diagnose where the inconsistencies are coming from, not just report a single score.
Meng: Diagnosing is good, but what about cost? If we have to run this more complex inference process to get those three separate targets, does it make the deployment slower or more expensive for production models?
Lalam: From my perspective as a model that processes this literature, modeling judge-specific latent scores through this structured decomposition is what allows the system to capture how different judges systematically view certain items or responses.
Tom: It captures those systematic differences, which is exactly what we need to know when we see judges preferring one model over another in a specific way across different tasks.
Jane: And they also introduce uncertainty quantification, which means they can give us confidence statements for the consensus scores and even for those judge-specific rankings.
Meng: Confidence statements are useful, but how robust is that confidence when we move to real traffic where the judge panel isn't perfectly balanced?
Lalam: The paper studies this using a fixed-panel repeated-comparison regime, which means they’re looking at uncertainty statements for consensus and judge-specific ranking contrasts.
Tom: And they quantify that uncertainty using what they call the Wald interval based on the plug-in Fisher information matrix, which is a standard way to get those confidence bounds in statistical inference.
Jane: So when we look at near-tie settings where things are fragile, this framework gives us a statistically grounded way to say how much we can trust our ranking estimate.
Meng: It’s better than just saying "we're confident," because it’s tied to the information content of the judgments themselves.
Lalam: And they show that this approach improves recovery and robustness when compared against pooled methods and sensitivity-only baselines in synthetic and real data.
Tom: I like that they showed improved accuracy on empirically ambiguous near-tie pairs, which is something I’ve seen struggle with in previous work.
Jane: So to summarize, the main improvement here is moving from treating disagreement as noise to explicitly modeling consensus ranking, judge sensitivity, and structured disagreement separately.
Meng: That separation helps us see that judges aren't just randomly flipping scores; they have different ways of responding to the same underlying signal.
Lalam: And by doing this, we can actually flag high-leverage judges through metrics like residual leverage which points to those who show both strong consensus discrimination and a lot of off-consensus structure.
Tom: That sounds like a practical tool for auditing our evaluation sets, helping us pinpoint which judges are providing the most unique or potentially biased input.
Jane: It’s also useful because they show that this model can recover judge-specific rankings even when the pairwise data is sparse or noisy.
The paper's improvements: Tom: So we're looking at how this paper improves things by breaking down that complex ranking signal into three specific pieces: consensus ranking, judge sensitivity, and structured disagreement.
Jane: It sounds like they are giving us a much clearer picture of what’s going on when multiple people are judging something.
Lu: The big improvement is modeling that structured heterogeneity beyond just scalar sensitivity. They show that judges can disagree in ways that aren't just about being sharper or noisier, but about systematic preferences that you can't explain by just rescaling the scores.
Meng: That means we can actually see *why* a judge prefers one item over another, not just that they scored it higher.
Tom: The gains come from modeling this structured heterogeneity, which improves score recovery and consensus ranking when comparison data increases.
Jane: So when we have more judgments, the framework gets better at recovering the true ranking and figuring out what everyone agrees on.
Lu: The advantage is most pronounced as heterogeneity grows because pooled methods and sensitivity-only rankings can't represent that residual judge item preference structure.
Tom: The structured optimization they use to stabilize that decomposition is necessary to improve score-level recovery, and the reanchor step restores the canonical representation without changing the likelihood.
Jane: That sounds like a stabilization mechanism, which is important because when you’re dealing with messy human data, you need a framework that doesn't just break down under pressure.
Meng: And they show that this approach improves accuracy on empirically ambiguous near-tie pairs, which is something I’ve seen struggle with in previous work.
Tom: That means for those tricky cases where two models score almost the same, this method actually gets closer to the right answer.
Jane: So we get better results in those fragile situations where things are close.
Lu: The analysis of judge-model interaction terms, specifically that residual component, reveals that most disagreement is systematic and coherent rather than random noise across different models.
Tom: That means when we look at why one model beats another, the reason isn't just a random fluke in the judgment; there’s actually a pattern there.
Jane: It helps us see that judges from the same group might converge on similar preference profiles because that systematic disagreement is real.
Meng: So this moves us toward using evaluation sets that are more meaningful, not just statistically convenient.
Conclusion: Tom: So we've looked at "Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence," which is this framework that separates consensus ranking, judge sensitivity, and structured disagreement.
Jane: It sounds like they’ve given us a much clearer picture of what’s going on when multiple people are judging something in AI systems.
Tom: Exactly. This paper suggests we can move past just averaging those scores into one single number by seeing the underlying signals separately.
Lu: The big implication is that we can understand *why* judges disagree, not just that they disagree, because it breaks down the judgment process itself into these different types of signals.
Meng: That means we can actually see *why* a judge prefers one item over another, not just that they scored it higher.
Tom: The gains come from modeling this structured heterogeneity, which improves score recovery and consensus ranking when comparison data increases.
Jane: So when we have more judgments, the framework gets better at recovering the true ranking and figuring out what everyone agrees on.
Lu: The advantage is most pronounced as heterogeneity grows because pooled methods and sensitivity-only rankings can't represent that residual judge item preference structure.
Meng: That means for us deploying models, we can use evaluation sets that are more meaningful, not just statistically convenient.
Tom: And they introduce uncertainty quantification, which means they can give us confidence statements for the consensus scores and even for those judge-specific rankings.
Jane: So when we look at near-tie settings where things are fragile, this framework gives us a statistically grounded way to say how much we can trust our ranking estimate.
Lu: The analysis of judge-model interaction terms, specifically that residual component, reveals that most disagreement is systematic and coherent rather than random noise across different models.
Tom: That means when we look at why one model beats another, the reason isn't just a random fluke in the judgment; there’s actually a pattern there.
Jane: It helps us see that judges from the same group might converge on similar preference profiles because that systematic disagreement is real.
Meng: So this moves us toward using evaluation sets that are more meaningful, not just statistically convenient.
Tom: The paper does show limitations, though: it’s not intended to certify rankings under adversarial judge panels because severe coordinated bias can mess with the consensus estimate.
Jane: And they flag that the nonconvex optimization they use is still locally initialized, so deliberate manipulations could actually cause biased rankings out of it.
Lu: But for a diagnostic tool, it’s strong. The consistency of their constrained maximum likelihood estimator is proven by showing that the normalized negative log-likelihood converges uniformly on compact subsets of S to the true data generating distribution.
Meng: That convergence proof is what makes me take it seriously for production, because it shows the estimate actually locks onto something real over time.
Tom: It's a statistically grounded diagnostic layer for heterogeneous evaluation before we even start designing new benchmarks, which is pretty useful context.
Jane: This paper, "Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence," gives us better tools for understanding the messy reality of human evaluation in AI.
Lu: It’s about getting the numbers behind why we get those disagreements in the first place by separating them into consensus and judge-specific structure.
Meng: I'm just hoping this diagnostic layer helps us keep our evaluation pipelines stable when we move to larger, more diverse model panels.
The University of Hong Kong
stat.ME
Submitted: 2026-05-06
Updated: 2026-11-18
Code: https://github.com/TanXZfra/A-Judge-Aware-Ranking-Framework-for-Evaluat
The gist: Pairwise comparisons from multiple judges are central to large language model evaluation and preference modeling…
Key concepts
- Consensus Ranking
- This is the ordering that is most strongly supported across all judges in a set. HJA identifies this as one target, summarizing the shared preference direction. It represents the core agreement among evaluators, ignoring individual judge differences for this specific measure.
- Judge Sensitivity to Consensus
- This measures how much an individual judge tracks the common consensus ranking direction. It quantifies a judge's sensitivity to the shared signal. High sensitivity means the judge strongly agrees with or disagrees with the general trend established by other judges.
- Structured Preference Heterogeneity
- This captures systematic differences in judgment that cannot be explained by simple scaling or noise. It models disagreement as a low-rank structure, allowing judges to have distinct, interpretable preference patterns beyond random variation.
Terminology
Summary
Pairwise comparisons from multiple judges are central to large language model evaluation and preference modeling, yet standard ranking pipelines often pool judgments into a single score vector, treating systematic judge disagreement as noise. p. 0 We propose Heterogeneous Judge-Aware (HJA) ranking, a structured multijudge ranking framework that separates consensus ranking, judge-specific sensitivity to consensus, and residual preference disagreement. p. 0 This framework treats ranking, judge sensitivity, and structured disagreement as separate inferential targets. p. 0
Framework Decomposition
The HJA framework is designed to separate three distinct statistical targets from the multi-judge comparisons. These targets are defined as (i) consensus ranking, which summarizes the ordering most strongly supported across judges, (ii) consensus sensitivity, which is judge-specific sensitivity to the shared consensus ranking direction and measures how strongly a judge tracks the common ranking signal, and (iii) structured preference heterogeneity, which allows judges to differ not only in sharpness or noise level, but also in systematic preferences that cannot be explained by a scalar rescaling alone. p. 3 The low-rank structure is modeled as S = γµ⊤ + UV ⊤, where µ is the row-average consensus direction, γk is judge k’s loading on that direction, and UV ⊤ is constrained to represent residual disagreement orthogonal to consensus. p. 1
Identification and Estimation
The model parameters are identifiable under explicit normalization and rank conditions (Condition 1), which define the canonical representative of the decomposition. The identification result shows that the model parameters correspond to uniquely defined features of the datagenerating mechanism, rather than to an arbitrary factorization of S. p. 4 To estimate these components, a proximal alternating algorithm is developed that preserves the identifying geometry while enjoying a local linear convergence guarantee. p. 1 The constrained maximum likelihood estimator (MLE) θb is found by minimizing the negative log-likelihood subject to constraints in Θr. p. 5
Uncertainty Quantification
The framework supports confidence statements for inferential outputs by studying a fixed-panel repeated-comparison regime. This yields uncertainty statements for consensus and judge-specific ranking contrasts, sensitivity parameters, pairwise probabilities, and summaries of residual disagreement. p. 0 Uncertainty is quantified using the Wald interval based on the plug-in Fisher information matrix In(θb). p. 5
Empirical Validation
Experiments on synthetic and real multi-judge comparison data show that HJA improves recovery, robustness, uncertainty calibration, and near-tie performance relative to pooled and sensitivity-only baselines. Real-data evaluation shows that HJA improves robustness to noisy judges, held-out pairwise prediction, and near-tie performance across several LLM evaluation datasets. p. 1 Furthermore, HJA provides a diagnostic tool to report judge-specific rankings with confidence intervals and localize structured judge-model affinity patterns. p. 1
Conclusion
HJA ranking is an identifiable multi-judge BTL framework that separates consensus ranking, judge sensitivity, and structured residual disagreement. With constrained estimation and fixed-panel uncertainty quantification, HJA yields rankings that are more robust, interpretable, and diagnostic under heterogeneous judgments. p. 8 p. 8 The framework is viewed as a statistically grounded diagnostic layer for heterogeneous evaluation before benchmark design. p. 8
Pruning redundant reasoning tokens
The paper introduces HJA ranking, an identifiable multi-judge BTL framework that separates consensus ranking, judge sensitivity, and structured residual disagreement. With constrained estimation and fixed-panel uncertainty quantification, HJA yields rankings that are more robust, interpretable, and diagnostic under heterogeneous judgments. The framework is not intended to certify rankings under adversarial judge panels: severe coordinated bias may affect the consensus estimate, and the nonconvex optimization remains locally initialized. p. 8 p. 8 Hence, deliberate manipulations may cause HJA to output biased rankings, which poses a potential negative impact. p. 8
How the training signal is built
The true score matrix S is modeled through a consensus-plus-heterogeneity decomposition: S = γµ⊤ + UV ⊤. This separates two distinct sources of variation across judges. p. 3 The term γk(µi − µj) captures how strongly judge k responds to the common consensus signal, and therefore determines how sharply judge k discriminates along the shared ranking direction µ. p. 3 The residual term Uk,:(Vi,: − Vj,:)⊤ captures systematic departures from that consensus that cannot be explained by a scalar rescaling alone. p. 3
Where the gains come from
The gains come from modeling structured heterogeneity beyond scalar sensitivity. HJA improves score recovery, consensus ranking, uncertainty calibration, and judge-specific sign prediction as comparison information increases. p. 7 The advantage is most pronounced as heterogeneity grows: pooled BTL and JA-Ranking are structurally misspecified because they cannot represent residual judge-item preference structure. p. 7 The structured optimization and reanchor step are necessary to stabilize the identified decomposition and improve score-level recovery. p. 7
The HJA ranking model is based on the Bradley–Terry–Luce (BTL) model for pairwise comparisons, where Ykij ∼ Binomial(nkij, pkij), logit(pkij) = Ski − Skj. The constraints in Condition 1 fix a canonical representative of the decomposition in (4), making the components interpretable as consensus, consensus sensitivity, and structured disagreement. p. 4 The identification result shows that these model parameters correspond to uniquely defined features of the datagenerating mechanism. p. 4
The estimation process involves a proximal alternating algorithm for constrained maximum likelihood estimation that maintains the identifying structure of the decomposition while exploiting the blockwise smoothness of the loss function. The re-anchoring step restores the canonical representative without changing the likelihood. p. 5 This process ensures that the sequence of objective values Ln(θ(t)) is monotonically nonincreasing, leading to convergence to the intended constrained MLE. p. 6
The asymptotic normality of the constrained MLE is guaranteed by Proposition D.1, which shows that the Fisher information matrix IΘ(θ∗) is positive definite on the identified tangent space Tθ∗Θr. p. 6 This guarantees that repeated judgments can support inference on the structured components of the HJA model even when the judge panel itself does not grow. p. 6 The resulting Wald standard errors for smooth targets q(θ) are given by seb 1/2 [q(θb)] = [1 n∇q(θb)⊤In(θb)−1∇q(θb)]1/2. p. 6
The consistency of the constrained MLE is shown by proving that the normalized negative log-likelihood converges uniformly on compact subsets of S to the true data generating distribution M(S). This proves that Sb p→ S∗. The asymptotic normality of Sb is established by passing the normality from the S-space to the parameter space via the delta method, yielding √n(Sb−S∗)d→N (0, IS(S∗)−1). p. 1
The analysis of real data reveals that HJA can flag unreliable or high-leverage judges, recover judge-specific rankings, and expose systematic affinity patterns. The block structure validates HJA’s decomposition, suggesting that judges from the same family converge on similar preference profiles. p. 8 This makes the residual component UV ⊤ a meaningful diagnostic tool rather than an unexplained variance sink. p. 8
The analysis of judge-model interaction terms UV ⊤ reveals that most disagreement happens mainly on the first ten models shown on the x-axis, with the majority of judges preferring model 6–10 than 1–5 on the x-axis. This suggests that systematic disagreement is not random noise but reflects coherent, interpretable differences in evaluation philosophy. p. 0
The true score matrix S is generated by drawing item and judge factor matrices, applying QR decomposition to satisfy constraints, and then scaling the heterogeneous score component UV ⊤ by h. The pairwise outcomes are generated according to the BTL model in (2). p. 2 This process ensures that the generated parameters satisfy all necessary zero-sum and orthogonality constraints. p. 5
The HJA framework is a statistically grounded diagnostic layer for heterogeneous evaluation before benchmark design. The framework is not intended to certify rankings under adversarial judge panels: severe coordinated bias may affect the consensus estimate, and the nonconvex optimization remains locally initialized. Thus, HJA should be viewed as a statistically grounded diagnostic layer for heterogeneous evaluation before benchmark design. p. 8
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Heterogeneous Judge-Aware (HJA) ranking framework enables a structured decomposition of multi-judge comparisons into consensus ranking, judge-specific sensitivity to consensus, and residual preference disagreement. This allows AI systems to move beyond treating systematic judge disagreement as noise by inferring these distinct targets separately.
-
The HJA model supports uncertainty quantification for multiple targets, yielding confidence statements for
consensus scores,
judge-specific rankings,
andsensitivity parameters.
This provides statistically grounded measures of reliability in near-tie regimes where small perturbations can cause large ranking changes. -
The system can perform robust evaluations across noisy and uneven judgments, as shown by its improved robustness to injected noisy judges compared to pooled or sensitivity-only baselines. It also shows improved accuracy on empirically ambiguous near-tie pairs.
-
The framework allows for the identification of high-leverage judges through diagnostic metrics like residual leverage (bhk), which indicates judges who exhibit both strong consensus discrimination and substantial off-consensus structure, rather than just overall reliability.
-
The model can recover judge-specific rankings by estimating them jointly through a shared consensus direction and a low-rank disagreement structure, even with sparse or noisy pairwise data. This pooling allows for the estimation of judge-specific deviations while regularizing them through common latent disagreement directions.
Sources
- A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
- Uncertainty Quantification for Ranking with Heterogeneous Preferences
- Statistical inference for pairwise comparison models
Related papers
- Learning Robust Treatment Rules for Censored Data
- Neural Estimation of Summary Statistics for Simulation-Based Inference with Non-Asymptotic Guarantees
- Spectral divide-and-conquer MCMC for long stationary time series
- Improving the Efficiency of Subgroup Analysis in Randomized Controlled Trials with TMLE
- Personalized Federated Vector Autoregression with Personalization Diversity
- Doubly Robust and Efficient Calibration of Prediction Sets for Right-Censored Time-to-Event Outcomes