RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges
cs.CL
Submitted: 2026-08-25
Updated: 2026-08-25
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or distillation from
Terminology
Abstract
LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or distillation from stronger teachers. In this work, we eliminate external gold supervision from the RL training reward: the model's own evaluative capability generates learning signals for its optimization -- a closed-loop setting of bounded recursive self-improvement (RSI) termed Recursive Self-Evaluation (RecurSE). We study two central questions: when can self-improvement occur, and when must it stop? First, RecurSE pairs a trainable judge evaluating candidate responses under per-rule rubrics (Pass 1) with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward (Pass 2). To enable learning, interface decoupling structurally isolates the checker's scalar score from the judge's verdict tokens, eliminating a degenerative token-copying shortcut that inflates self-assigned rewards. Second, because unanchored recursive learning is inherently bounded, Pairwise Advantage Validity (PAV) serves as an unbiased validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify the optimal early-stopping window. Across Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B, RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks. Ablations demonstrate that synchronized judge-checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation. Furthermore, preference pairs curated by our judge effectively enhance downstream policy alignment. Bounded RSI for LLM-as-judge is thus viable when self-produced reward validity is explicitly decoupled and monitored.
Sources
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
- On the Expressive Power of Tree-Structured Probabilistic Circuits
- EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Reinforcement Learning from Meta-Evaluation: Aligning Language Models Without Ground-Truth Labels
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- RLSR: Reinforcement Learning from Self Reward
- Gemma 4 Technical Report
- Co-Evolving LLM Evaluators and Policies via DynamicRubric
- Self-Taught Evaluators
- Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
- More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering