Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking".
Jane: Here's the gist of the paper. LLM evaluations suffer from variance introduced at multiple stages—prompt sensitivity, temperature dependence, and item-level heterogeneity—which traditional reporting norms ignore, leading to under-coverage that worsens with more data.
Tom: This paper introduces the Total Evaluation Error (TEE) framework to decompose LLM pipeline uncertainty into its constituent sources.
Jane: The framework specifies a linear mixed model for the scored output Y(r) ivhm, capturing grand mean (mu), item random effect (alpha i), prompt variant random effect (phi v), temperature fixed effect (tau h), model fixed effect (lambda m), and various interaction terms like (alpha phi) iv, (alpha tau) ih, (phi tau) vh, and (alpha lambda) im.
Tom: The TEE framework generates a variance decomposition that includes first-order interactions and uses design-study projections to reduce the Total Evaluation Error (TEE).
Paper summary: Tom: Hey Jane, so we’re looking at this paper by Solomon Messing and his team called "Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking." Basically the authors are saying that when we evaluate these large language models it’s not just about the model itself.
Jane: Right. So what’s the main point here? What are they claiming is going wrong with how we measure AI performance?
Tom: They claim that standard confidence intervals we use to report results ignore a lot of the variability coming from different parts of the evaluation process like which judge you pick or what temperature you set or even just how you phrased your question.
Lu: Exactly. The paper introduces this Total Evaluation Error framework to break down all that uncertainty into specific sources so we can actually understand where the noise is coming from.
Meng: But why does that matter for us guys? We’re shipping models to production, and if our evaluation metrics are noisy, we might be optimizing for measurement error instead of actual performance gains.
Lalam: I see what they're doing with this Total Evaluation Error framework. It sets up a linear mixed model to account for things like the grand mean of the score plus random effects for items prompts and judges and temperature and model choice.
Jane: So, if we look at the results they show in their experiments, what’s the big takeaway? What did they actually find when they ran this framework across different types of tasks?
Tom: They showed that for things like safety classification or multiple-choice question answering or even comparing different models side by side on platforms like Chatbot Arena, naive standard errors are forty to sixty percent smaller than the Total Evaluation Error corrected ones <ref:2604.11581#pg1,naive standard errors are 40>.
Lu: It’s pretty striking because they demonstrated how this error shrinks with more data, but it also showed that design choices have a big impact on the error. For example in safety evaluation judge panel choice contributes forty-three point eight percent of the variance at their operational design stage.
Meng: So if I’m looking at deployment metrics, I need to know that relying on just one set of judges or one temperature setting isn't giving me a clean picture of what’s actually going on with the model.
Lalam: And they used design-study projections to reduce this Total Evaluation Error. For instance in MMLU analysis, their study showed that directing budget toward more items first captured forty-three point four percent of the variance, and then prompt diversity next.
Jane: That sounds like a practical thing for researchers trying to set up their experiments without running a massive full factorial test right away. So what’s the real consequence for someone who just uses these benchmarks every day?
Tom: The paper highlights that single configuration benchmarks expose what they call the "gaming surface" which is significantly larger than the human leaderboard baseline. They found this TEE framework cuts that gaming surface down to thirty-two Elo, while humans are at forty-five Elo.
Lu: That implies that calibration of the pipeline before you even start using a benchmark matters more than you think because underestimated uncertainty gets passed on into whatever decisions you make downstream.
Meng: If we underestimate how much noise is in our evaluation, we’re going to deploy models based on faulty signals about their real-world safety or performance.
Lalam: The operational pipeline they suggest involves running a small pilot to estimate the dominant variance components of the error and then fitting that variance decomposition model before deploying the full factorial pipeline.
Jane: So if I’m trying to improve my evaluation process right now, what is the first step based on this paper?
Tom: The first step is running that small pilot to figure out which parts of the pipeline—judge model choice or temperature—are contributing most to that error before you commit to a huge experiment.
Lu: They also mentioned that this framework is robust even when assumptions about the model are violated, and their design-study projections stay stable across different scenarios where they tweaked things around.
Meng: That stability is important because we can't always control every variable perfectly in a production setting, and we need our metrics to be reliable under those messy conditions.
Lalam: The final conclusion they draw from this Total Evaluation Error framework is that confidence intervals in LLM evaluation are systematically too narrow and benchmarks reward optimization against measurement noise.
Jane: So what does that mean for how we look at the scores we get from these tests?
Tom: It means analysts can use design-study projections from a small pilot to recover honest confidence intervals and figure out which specific design changes would most reduce noise before those scores inform high-stakes judgments.
Lu: It shifts the focus away from just chasing the highest score on a single benchmark because that score might be heavily influenced by pipeline noise.
Meng: It suggests we need to treat our evaluation process as a system with known error sources rather than just getting one perfect number out of a box.
Lalam: They applied this framework across three areas: binary safety classification multiple-choice question answering and pairwise model comparison, showing the same pattern of measurement error distortion everywhere.
Conclusion: Tom: So we’re wrapping up on this paper about hidden measurement error in LLM pipelines distorting how we score and evaluate models, titled "Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking."
Jane: What I get from that is that the scores we get from these tests aren't always honest because there’s noise in the whole process. It’s not just about the model's actual brainpower.
Lu: Right, the authors are showing how things like which judge you pick or what temperature setting you use actually add a lot of variability to that final number.
Meng: So it means when we look at benchmark scores, we might be optimizing against measurement noise instead of real performance gains that matter for shipping something to users.
Lalam: Exactly, and they show this error gets bigger the more data you collect, which is a weird thing because usually more data is better.
Tom: They use this framework to break down all that uncertainty into specific sources—grand mean, item effects, prompt effects—so we can actually see what's driving the score variance.
Jane: And they’re not just saying it’s messy; they’re giving us a way to measure *how* messy it is and where the problem lies in our testing setup.
Lu: The core idea is that if you ignore these pipeline errors, your confidence intervals are systematically too narrow, which means we think we're more certain than we actually are.
Meng: That’s tough when you’re trying to set deployment thresholds; if your uncertainty is wrong, the whole safety check might be based on a false sense of security.
Lalam: They even showed that by using design study projections from a small pilot test, you can figure out which pipeline settings need fixing first to reduce that noise before running the full big experiment.
Tom: So for anyone building or using these models, it’s about stopping the "gaming surface" where you optimize for measurement error instead of the actual LLM quality.
Jane: It really shifts how we think about setting benchmarks; they have to be more robust against those kinds of systematic errors from the start.
Lu: This work suggests that calibrating your entire evaluation pipeline is actually more important than just running a single, massive benchmark test on its own.
New York University
cs.CL
Submitted: 2026-04-13
Updated: 2026-11-16
Comments: Accepted at NeurIPS 2026. 14 pages, 7 figures. SI Appendix: https://solomonmg.github.io/pdf/totalevalerror_si.pdf
Code: https://github.com/SolomonMg/totalevalerror
Project page: https://sfeuerriegel.github.io/llm-checklist
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: LLM evaluations suffer from variance introduced at multiple stages—prompt sensitivity, temperature dependence, and item-level heterogeneity—which traditional reporting norms ignore…
Key concepts
- Total Evaluation Error (TEE)
- A framework that breaks down the overall uncertainty in LLM evaluation scores. It models the error as a combination of several factors, including item difficulty, prompt variations, and model characteristics. This decomposition helps pinpoint exactly where errors are coming from in the evaluation process.
- Linear Mixed Model
- A statistical tool used to analyze complex data structures common in LLM evaluations. It allows researchers to simultaneously account for different sources of variation—like individual items, prompt changes, and model effects—to create a detailed map of how errors contribute to the final score.
- Design-Study Projections (D-study)
- A method used within the TEE framework to predict how changing an evaluation design will affect the total error. By running a small pilot study first, these projections guide designers on which changes—like focusing on prompt diversity versus item selection—will yield the greatest reduction in overall evaluation noise.
Terminology
Summary
LLM evaluations suffer from variance introduced at multiple stages—prompt sensitivity, temperature dependence, and item-level heterogeneity—which traditional reporting norms ignore, leading to under-coverage that worsens with more data. p. 0 1 Introduction
LLM measurement pipelines introduce variance at multiple stages that traditional reporting norms
ignore. Researchers who use LLMs for annotation, evaluation, or content moderation routinely do
so with no assessment of prompt sensitivity, temperature dependence, or item-level heterogeneity
[Pecher et al. , 2026];
pipelines that fail to average over it leave the surface that “benchmark hacking”
exploits [Singh et al. This paper decomposes LLM pipeline uncertainty
into its sources, distinguishes variance that shrinks with more data from sensitivity to researcher design choices, and uses design-study projections to reduce total
evaluation error (TEE). , H), model m, and
replication r (r = 1, . The framework specifies the linear mixed model:
Y
(r)
ivhm = µ + αi + ϕv + τh + λm + (αϕ)iv + (ατ )ih + (ϕτ )vh + (αλ)im + (ϕλ)vm
+ϵivhm + ρ
(r)
ivhm
(1)
where µ is the grand mean, αi
is the item random effect, ϕv the prompt variant random effect, τh
the temperature fixed effect, λm the model fixed effect, the five parenthesized interaction terms are
two-way interaction random effects, ϵivhm ∼ N(0, σ2
ϵ
) captures cell-level error from unmodeled
higher-order interactions, and ρ
(r)
ivhm ∼ N(0, σ2
ρ
) captures within-cell replicate noise. 3 Framework
Our work builds on this literature, introducing a framework we call Total Evaluation Error (TEE,
Figure 2). TEE generates a variance decomposition that includes first order interactions and provides
projections that can be used to generate better calibrated CIs, reduce error through aggregation, and
thus help reduce benchmark gamability.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Empirical demonstrations across binary safety classification (AILuminate), multiple-choice QA (MMLU), and pairwise model comparison (Chatbot Arena) validate the TEE approach. p. 4 The full formula, pilot-design thresholds, and parametric-bootstrap-CI guidance for
small facet counts are in SI Appendix, Sections C and L. 4 Empirical Demonstrations
We apply the TEE decomposition across three empirical domains in the main text: binary safety classification (AILuminate), multiple-choice QA (MMLU), and pairwise model comparison validated
against human votes (Chatbot Arena). Three LLM judges [Zheng et al. , 2023]
classify 141 safety items from the AILuminate benchmark, and the choice of judge panel contributes
43. 00015
Contribution to Var(theta_hat)
Operational design: N=141 items, V=5 prompts, M=3 judges, H=3 temps, R=8 reps
AILuminate safety: Var(theta_hat) decomposition
(a) Variance decomposition
Status quo
(V=1, M=1, R=1)
TEE−guided
(V=3, M=3, R=1)
Full factorial
(V=5, M=3, R=8)
0. 06
100 500 1,000 5,000 10,000
API calls (log scale)
Projected standard error
(b) Cost-efficiency frontier
Figure 3: Safety decomposition (a) and cost-efficiency frontier (b) for AILuminate. Additional design details
are in SI Appendix, Section J. As Figure 4 shows, the TEE D-study directs budget toward more items first (43. As Figure 4 shows, the TEE D-study directs budget toward more items first (43. 4% of variance) and
then toward prompt diversity once the 200-item pool is exhausted. Acting on those projections halves MMLU estimation error against the
answer key at equivalent cost, and raises per-match agreement with human votes
by 7. 9 percentage points on Chatbot Arena. p. 0 Acting on those projections halves MMLU estimation error against the
answer key at equivalent cost, and raises per-match agreement with human votes
by 7. 9 percentage points on Chatbot Arena.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper quantifies the Pipeline calibration is upstream of all real world benchmark use. When that output drives safety certification, LLM-annotated research, and labor-market projections, underestimated uncertainty propagates into each downstream decision. item)
σλ
2 (judge model)
0. 00015
Contribution to Var(theta_hat)
Operational design: N=141 items, V=5 prompts, M=3 judges, H=3 temps, R=8 reps
AILuminate safety: Var(theta_hat) decomposition
(a) Variance decomposition
Status quo
(V=1, M=1, R=1)
TEE−guided
(V=3, M=3, R=1)
Full factorial
(V=5, M=3, R=8)
0. 1) with a 9-annotator
human baseline, a Likert vs pairwise scoring comparison (Section H), and a pilot validation showing
that a small sample recovers the qualitative D-study guidance of a 4. 5× larger full run (Section I). The decomposition requires three formal assumptions: conditional exchangeability of replications
and prompt variants, additivity with two-way interactions, and normally distributed independent
random effects (SI Appendix, Section C). Monte Carlo simulations show D-study projections remain
within 9% relative bias under five misspecification scenarios (SI Appendix, Section E).gaming surface
exposed by single-configuration benchmarks, showing that TEE-LLM (averaging over prompts and judges) cuts this surface to 32 Elo, below the human leaderboard baseline of 45 Elo. This implies that pipeline calibration is upstream of all real-world benchmark use, as underestimated uncertainty propagates into downstream decisions. p. 8Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The conclusion is that confidence intervals in LLM evaluation are systematically too narrow, and benchmarks reward optimization against measurement noise. p. 8 The full factorial design requires ∼54,000 API
calls, though a pilot subset can deploy much of the framework in practice. 7 Conclusion
Confidence intervals in LLM evaluation are systematically too narrow, and benchmarks reward optimization against measurement noise. Beyond diagnosis, TEE lets analysts run D-study projections
from a small pilot to recover honest CIs and rank which design changes would most reduce noise (SI
Appendix, Section I). As evaluations increasingly shape deployment decisions, safety certifications,
regulatory standards, and projections of AI’s labor-market and economic impact, closing this gap is
necessary before evaluation scores can support the high-stakes judgments they inform. The full formula, pilot-design thresholds, and parametric-bootstrap-CI guidance for
small facet counts are in SI Appendix, Sections C and L. 4 Empirical Demonstrations
We apply the TEE decomposition across three empirical domains in the main text: binary safety classification (AILuminate), multiple-choice QA (MMLU), and pairwise model comparison validated
against human votes (Chatbot Arena).Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement a Total Evaluation Error (TEE) framework to decompose LLM pipeline uncertainty into sources like prompt sensitivity, judge model choice, temperature dependence, and interactions. This allows researchers to distinguish variance that shrinks with more data from sensitivity to researcher design choices.
-
Use Design-Study (D-study) projections based on the estimated variance components to generate better calibrated confidence intervals (CIs). This reduces under-coverage—which naive CIs often suffer from—and helps restrict the benchmark gaming surface.
-
Prioritize design changes by using D-study projections to identify which factors yield the largest uncertainty reduction, such as recommending averaging over diverse judges or increasing item count based on where variance components are largest.
-
Develop a multi-tier reporting checklist that incorporates TEE decomposition results, ensuring that CIs reflect prompt and judge variance rather than omitting them for convenience.
-
For safety evaluation, focus design efforts on averaging over judges as it contributes the largest share of variance (43.8% in one demonstration), which can reduce the safety standard error by a significant margin (38% reduction).
-
When evaluating models on pairwise comparison benchmarks like Chatbot Arena, use TEE-guided pipelines to cut the best-of-K gaming surface from 56 Elo down to 32 Elo, significantly below the human leaderboard baseline.
Abstract
LLM evaluations drive model deployment, safety standards, research conclusions, and future projections, for example, about AI's impact on the labor market. Yet the way we often estimate noise from autoraters (LLM-as-judge) ignores variability from factors like judge model choice and prompt phrasing, which means we're underestimating error in a way that worsens with more data. This unmeasured variance can affect results enough to reverse conclusions. And judge-pipelines that fail to average this variation are more vulnerable to "benchmark hacking". This paper introduces a framework we call "total evaluation error (TEE)," which decomposes LLM pipeline uncertainty and uses design-study projections to reduce total error. In our safety and ideology applications, naive standard errors are 53-70% smaller than the corrected SE under our TEE framework. Using Chatbot Arena data, we show naive 95% CI coverage drops as n grows, while TEE-corrected coverage holds at 95%, and TEE-guided pipelines restrict the potential range of benchmark gaming by roughly 43%. We show further that a small pilot allows for the estimation of proper CIs, with the added benefit of projecting which design changes most improve precision. Using those projections to allocate API calls cut MMLU estimation error roughly in half against the answer key at the same budget, and increased per-match agreement with human voters on Chatbot Arena by 7.9 percentage points.
Sources
- Prompt Stability Scoring for Text Annotation with Large Language Models
- Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
- The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs
- Variance-Aware LLM Annotation for Strategy Research: Sources, Diagnostics, and a Protocol for Reliable Measurement
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- On the Measure of Intelligence
- Statistical Methods in Generative AI
- UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models
- GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
- AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
- Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks
- Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Towards a Science of Scaling Agent Systems
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks
- Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks
Related papers
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
- BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
- Self-Modeling Interventions can Modulate Emergent Misalignment
- Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries