Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing
cs.CL, cs.AI
Submitted: 2026-10-27
Updated: 2026-11-05
Comments: 55 pages (11 with no appendix), 10 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument.
Terminology
Abstract
LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reasoning in order to optimize accuracy. In contrast, we investigate whether validated prompt variants intended to preserve evaluation semantics while differing in surface realization (paraphrastic rephrasings) yield consistent judgments under a fixed evaluation task. We introduce Prompt Lottery, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments. Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings. A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9 times larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. This indicates that single-prompt evaluation is insufficiently robust for model ranking. Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.
Sources
- Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness
- JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data