LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
cs.AI
Submitted: 2026-07-05
Updated: 2026-11-16
Code: https://github.com/bbang3/LLM-as-a-Tutor
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals.
Terminology
Abstract
Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design preserves the original prompt while adding constraints to increase difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, on average and on four of five metrics, suggesting prompt adaptation as an underexplored axis of policy-awareness in non-verifiable RL for constraint-heavy instruction following.
Sources
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
- Distilling the Knowledge in a Neural Network
- R-Zero: Self-Evolving Reasoning LLM from Zero Data
- VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
- Language Self-Play For Data-Free Training
- SPICE: Self-Play In Corpus Environments Improves Reasoning
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
- Proximal Policy Optimization Algorithms
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- WizardLM: Empowering large pre-trained language models to follow complex instructions
- Qwen3 Technical Report
- Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents