CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
cs.CL
Submitted: 2026-03-02
Updated: 2026-11-03
Terminology
Sources
- Do Repetitions Matter? Strengthening Reliability in LLM Evaluations
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- The Llama 3 Herd of Models
- A Survey on LLM-as-a-Judge
- HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
- LLM-as-Judge on a Budget
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- Self-Preference Bias in LLM-as-a-Judge
- Qwen2.5 Technical Report
- Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data