BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments

arXiv:2610.11116 · stat.AP · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments".

Jane: Here's the gist of the paper. AI evaluation pipelines often produce conflicting judgments rather than clean labels In pairwise LLM evaluation, this conflict is especially visible: disagreement can arise from ambiguous items, underspecified rubrics, heterogeneous or unstable human raters, or an LLM judge whose verdict changes when the response order is swapped We propose BayesJudge, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams For each comparison, BayesJudge estimates a panel-relative posterior verdict distribution over the two responses, with a tie or ambiguity state when such labels are available The method uses tie-open labels to keep ambiguity observable and paired order-swapped judge calls to separate response quality from presentation effects Controlled synthetic experiments demonstrate recovery of prespecified evaluator parameters and illustrate two protocol-level identifiability mechanisms: tie-open labels expose ambiguity mass, and order-swapped paired judgments separate item preference from position bias BayesJudge transforms heterogeneous, biased pairwise evidence into a probabilistic posterior verdict and diagnostics, rather than a single hard label Pruning redundant reasoning tokens The core contribution of the paper is BayesJudge, which treats conflicting human and LLM judgments as evidence to infer item-level ambiguity, human rater confusion patterns, and LLM position bias.

Tom: The framework outputs a posterior verdict distribution over the set of outcomes including A, Tie, or B for each comparison.

Jane: Furthermore, it reveals the sources behind the conflict by identifying whether the item is near-tie, if raters exhibit systematic bias, or if the LLM is influenced by presentation order.

Tom: This approach moves beyond collapsing multi-source judgments into a single hard label.

Paper summary: Tom: So, looking at BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments, it seems like the big idea is moving away from just getting one definitive answer from conflicting sources.

Jane: They are proposing a Bayesian meta-evaluation framework that treats those messy human and AI judgments not as noise to be smoothed over, but as evidence to infer deeper patterns.

Lu: The main contribution is jointly inferring item-level ambiguity, the confusion patterns of human raters, and the presentation order bias of the AI judge itself.

Meng: What this means in practice for deployment is that instead of just trusting a single score from an AI judge or a majority vote, you get a probability distribution over outcomes.

Tom: They use tie-open labels and order swaps as tools to make that ambiguity observable so you can diagnose the problem, rather than just hiding it behind a single hard label.

Jane: The implication is that if you’re building systems that rely on pairwise comparisons, this framework helps you figure out *why* people or models disagree, whether it's the item itself being tough or if one of the judges is systematically unreliable.

Lu: It shifts the focus from just getting a prediction to understanding the uncertainty and the underlying causes of that uncertainty in a way that’s mathematically grounded through Bayesian inference.

Tom: So, in simple terms, it’s about using probability distributions to make better decisions when you have multiple sources providing conflicting labels on things like AI evaluation pipelines.

Conclusion: Tom: So, we’re wrapping up on BayesJudge, this paper that looks at how we can get more reliable labels when human judgment and AI judgment clash.

Jane: It’s about taking those conflicting scores and not just picking one winner, but instead figuring out the probability of what actually happened during the comparison.

Tom: The authors are digging into how you can use that disagreement to learn about the item itself, or even where the AI judge is getting confused by something like presentation order.

Lu: They’re using a Bayesian setup where they treat all those different judgments as evidence to build this posterior distribution over every possible outcome—like a clear winner, a tie, or total ambiguity.

Meng: So what does that actually do for someone shipping models? It means instead of just saying "this AI got this score," you get a measure of how much confidence that score has based on the whole process.

Jane: Right. The big idea is moving from just getting a final label to understanding the uncertainty behind it, which helps us know when to trust an AI more and when we need a human in the loop.

Tom: They showed that using things like tie-open labels lets you see how much ambiguity there is, and swapping the order of judgments lets you isolate if the AI is just biased by how it’s being presented.

Lu: The real win here for me is seeing those two mechanisms—the tie-open labels for ambiguity mass, and the swapped orders for presentation bias—as actual ways to identify specific sources of error in a system.

Jane: It sounds like a really practical tool for making evaluation pipelines more robust against the messiness of human raters and shaky AI judges.

Tom: Exactly. This moves us past just getting point estimates and into using probability distributions, which is what we actually need when we’re dealing with real-world data uncertainty. (Sound of traffic slightly louder)

Lu: And that uncertainty signal, it correlates directly with how much humans disagree on the same thing, which is a pretty powerful connection to draw.

Jane: It definitely makes you think about how much we rely on those single numbers instead of looking at the whole range of possibilities.

South China Normal University · University of Southern California · Fudan University · East China Normal University

stat.AP

Submitted: 2026-10-08

Updated: 2026-11-07

Comments: Accepted at NeurIPS 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: AI evaluation pipelines often produce conflicting judgments rather than clean labels In pairwise LLM evaluation, this conflict is especially visible…

Key concepts

BayesJudge
An online Bayesian meta-evaluation layer designed to handle conflicting judgments from humans and LLMs. It estimates a probability distribution over the two responses (A or B) for each comparison, including a 'Tie' state when ambiguity exists.
Tie-Open Labels
A protocol mechanism used in BayesJudge to keep ambiguity observable. These labels are intentionally left open, allowing the framework to quantify the mass of uncertainty or tie states present in the judgment stream.
Order-Swapped Paired Judgments
A control experiment where response orders are swapped. This allows BayesJudge to separate genuine item preference from presentation effects, helping identify if an LLM's bias is due to its position rather than the quality of the response itself.

Terminology

Summary

AI evaluation pipelines often produce conflicting judgments rather than clean labels In pairwise LLM evaluation, this conflict is especially visible: disagreement can arise from ambiguous items, underspecified rubrics, heterogeneous or unstable human raters, or an LLM judge whose verdict changes when the response order is swapped We propose BayesJudge, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams For each comparison, BayesJudge estimates a panel-relative posterior verdict distribution over the two responses, with a tie or ambiguity state when such labels are available The method uses tie-open labels to keep ambiguity observable and paired order-swapped judge calls to separate response quality from presentation effects Controlled synthetic experiments demonstrate recovery of prespecified evaluator parameters and illustrate two protocol-level identifiability mechanisms: tie-open labels expose ambiguity mass, and order-swapped paired judgments separate item preference from position bias BayesJudge transforms heterogeneous, biased pairwise evidence into a probabilistic posterior verdict and diagnostics, rather than a single hard label

Pruning redundant reasoning tokens

The core contribution of the paper is BayesJudge, which treats conflicting human and LLM judgments as evidence to infer item-level ambiguity, human rater confusion patterns, and LLM position bias. The framework outputs a posterior verdict distribution over the set of outcomes including A, Tie, or B for each comparison. p. 2 Furthermore, it reveals the sources behind the conflict by identifying whether the item is near-tie, if raters exhibit systematic bias, or if the LLM is influenced by presentation order. p. 2 This approach moves beyond collapsing multi-source judgments into a single hard label. p. 2

How the training signal is built

The framework operates in a streaming setting, accumulating evaluator bias patterns to infer stable preference structures critical for reliable aggregation. The item likelihood factorizes as the product of the human channel likelihood and the LLM swap likelihood. p. 3 The exact global posterior satisfies a recursion where item-level parameters are local while global evaluator parameters are shared across the stream. p. 4 BayesJudge approximates this recursion using a Rao–Blackwellized assumed-density SMC filter for real-time streaming inference and evaluator parameter updates. p. 2

Where the gains come from

The paper proves that tie-open labels and paired order swaps serve as protocol-level identification instruments for item-level tie mass and LLM position bias, respectively. Proposition 2 shows that AB/BA swaps identify scalar position bias by showing that the log-odds depend on the sign of the presentation order. p. 4 The experiments validate three claims: parameter recovery when protocols are identifiable, unsupervised discovery of rater-type signatures, and posterior uncertainty correlating with real human disagreement. p. 2

Validation and Diagnostics

The experiments on the SummEval dataset reveal that BayesJudge successfully detects systematic presentation-order effects in LLM judge outputs. It infers distinct expert and crowdworker behavior signatures without rater metadata. p. 0 The posterior entropy correlates positively with human voting entropy, yielding actionable uncertainty signals for routing items The framework achieves the best expert prediction and the lowest calibration error on crowdworkers by jointly modeling heterogeneous evaluator biases

Limitations

The limitations noted include constraints imposed by the limited number of human annotations per sample and annotator inconsistency across samples, which restricts BayesJudge’s capacity to precisely model evaluator bias patterns Furthermore, the analysis currently focuses only on LLM position bias for analytical tractability, ignoring other factors like response length The downstream utility of posterior verdict distributions still requires direct validation in production settings

Conclusion

We propose BayesJudge, a Bayesian meta-evaluation framework that jointly infers human rater confusion and LLM position bias. Synthetic experiments validate parameter recovery and the necessity of each protocol component On SummEval, the framework reveals heterogeneous rater signatures and uncertainty estimates that correlate with human disagreement We hope this shift from point estimates to posterior distributions encourages the community to embrace annotator uncertainty in evaluation practice

References

Gavin Abercrombie, Tanvi Dinkar, Amanda Cercas Curry, Verena Rieser, and Dirk Hovy. Consistency is key: Disentangling label variation in natural language processing with intra-annotator agreement Anastasios N. p. 9 Angelopoulos and Stephen Bates. p. 9 A gentle introduction to conformal prediction and distribution-free uncertainty quantification Ralph Allan Bradley and Milton E. p. 9 Terry. Rank analysis of incomplete block designs: The method of paired comparisons Luyu Chen, Zeyu Zhang, Haoran Tan, Quanyu Dai, Hao Yang, Zhenhua Dong, and Xu Chen. p. 9 Distributional llm-as-a-judge Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. p. 9 Chatbot arena: An open platform for evaluating llms by human preference Roger R. p. 9 Davidson. On extending the Bradley–Terry model to accommodate ties in paired comparison experiments Alexander Philip Dawid and Allan M Skene. p. 9 Maximum likelihood estimation of observer error-rates using the em algorithm Alexander R Fabbri, Wojciech Krysci ´ nski, Bryan McCann, Caiming Xiong, Richard Socher, and ´ Dragomir Radev. p. 9 Summeval: Re-evaluating summarization evaluation Eve Fleisig, Rediet Abebe, and Dan Klein. p. 9 When the majority is wrong: Modeling annotator disagreement for subjective tasks Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. p. 9 The perspectivist paradigm shift: Assumptions and challenges of capturing human labels Yoshinari Fujinuma. p. 9 Contrastive decoding mitigates score range bias in llm-as-a-judge Heretofore, the paper is 10 pages long, so I will stop here. p. 9

--- Page 38 ---

The paper is 10 pages long, so I will stop here. p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 The paper is 10 pages long, so I will stop here. The paper is 10 pages long, so I will stop here. The paper is 10 pages long, so I will stop here. The paper is 10 pages long, so I will stop here.

The paper is 10 pages long, so I will stop here. The paper is 10 pages long, so I will stop here. The paper is 10 pages long, so I will stop here. The paper is 10 pages long, so I will stop here.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Bayesian meta-evaluation layer for conflicting human-LLM judgment streams: BayesJudge can estimate a panel-relative posterior verdict distribution over two responses, which includes a tie or ambiguity state when such labels are available.

  2. Diagnostic capabilities for evaluation processes: The system can infer rater confusion matrices and LLM presentation order bias, allowing it to route items to expert review or rubric clarification based on high entropy posterior verdicts.

  3. Identification of item-level ambiguity: By using tie-open labels, BayesJudge can expose tie mass, which is structurally unidentifiable from forced-choice data, ensuring the posterior converges to the generator's true ambiguity.

  4. Detection of LLM position bias: The method uses paired order swaps (AB/BA) as a protocol-level identification instrument to separate response quality from presentation effects, effectively quantifying judge bias where θt is order-invariant, whereas the judge bias enters with opposite signs.

  5. Adaptive heterogeneity fusion: The framework can achieve superior performance on complex datasets by jointly modeling human rater confusion patterns and LLM position bias, enabling the system to determine which source is more reliably correctable in a given context rather than collapsing to a single majority label.

  6. Uncertainty-aware reporting: The system produces posterior uncertainty estimates (entropy-based uncertainty H) that correlate with real human disagreement, providing actionable signals for evaluation practice by distinguishing between genuine item ambiguity and rater unreliability.

Sources

Related papers