RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

Abstract Summary

Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. The paper proposes RISE-RL (Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and 3.3 points at the 14B scale, including a 6.0-point gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.

Introduction Summary

Open-ended question answering is a core capability of large language models, spanning open-domain dialogue, creative writing, health assistance, and scientific explanation. Unlike mathematical reasoning or code generation, open-ended tasks admit no single correct trajectory and require responses to satisfy diverse criteria, including factuality, completeness, style, structure, safety, and human preferences. Moreover, substantially different responses may be equally valid, giving rise to a multi-peaked policy distribution with multiple desirable generation modes. Reinforcement learning for open-ended tasks must therefore improve rubric alignment without sacrificing autonomous exploration or response diversity.

Existing methods still struggle to improve open-ended capabilities while preserving exploration and diversity. Rubric-based RL provides fine-grained criterion-level feedback but aggregates criterion-wise scores into a scalar sequence-level reward for policy optimization. Consequently, criterion-specific failures may not be directly translated into targeted policy updates. OPD provides dense token-level supervision by distilling a teacher distribution on states visited by the student policy, but offers limited control over which guided behaviors are selectively internalized. Mixed-policy optimization further combines privileged trajectories with on-policy samples in the same group-relative objective, allowing privileged samples to affect the reward normalization and relative advantages of natural rollouts, while limiting independent control over the strength and granularity of guided supervision.

To address these limitations, the paper proposes RISE-RL, a selective-guidance paradigm for open-ended tasks. RISE-RL uses rubric feedback to identify capability gaps, filters guided trajectories by complete-rubric reward, and re-evaluates the retained trajectories under the original prompt to emphasize behaviors weakly supported by the natural policy. The resulting signal is optimized through a separate auxiliary objective, keeping guided learning outside natural group-relative optimization. As its benefit diminishes, guidance is removed and training continues with unguided on-policy exploration, facilitating selective internalization while retaining autonomous exploration and response diversity.

The paper evaluates RISE-RL on Qwen3-4B and Qwen3-14B across four RubricHub domains and eight downstream benchmarks. RISE-RL achieves higher mean scores than standard Rubric-RL on every evaluated benchmark at both model scales, with average gains of 1.3 and 3.3 points at the 4B and 14B scales, respectively. On Qwen3-14B, the gains reach 8.0 points on Arena-Hard-v2 and 6.0 points on CreativeWriting-V3. In the creative-writing evaluation, RISE-RL also improves output diversity by 5.8%. These gains further extend to objectively scored tasks, including improvements of 3.3 points on MedQA and 3.6 points on GPQA-Diamond.

The main contributions are threefold:

• Criterion-Level Selective Guidance for Open-Ended RL: RISE-RL retains the fine-grained diagnostic information that is normally collapsed into a scalar rubric reward. By identifying high-value criteria that remain repeatedly unsatisfied across natural rollouts, RISE-RL constructs targeted guidance for behaviors that are rarely elicited through unguided exploration.

• Selective Internalization by Gain and Policy Support: RISE-RL retains only privileged trajectories that improve the complete-rubric reward over the natural-rollout baseline. It then removes the privileged criteria and re-evaluates the retained trajectories under the original prompt, using token probabilities to concentrate learning on beneficial behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective, preventing privileged trajectories from affecting the reward normalization and relative-advantage estimation of natural on-policy rollouts.

• Broad Empirical Evaluation across Models and Domains: Experiments with Qwen3-4B and Qwen3-14B across writing, chat, health, and science show that RISE-RL achieves higher mean scores than Rubric-RL on every evaluated benchmark at both scales under guidance-free evaluation. The gains are particularly pronounced on challenging open-ended benchmarks at the 14B scale, while improvements on objectively scored tasks further suggest within-domain transfer to verifiable tasks.

Related Work Summary

Rubric-based RL characterizes the multidimensional quality of open-ended responses through fine-grained evaluation criteria and aggregates criterion-level scores into a scalar reward for policy optimization. However, this aggregation can discard part of the diagnostic information provided by the judge, making failures on specific criteria difficult to translate into targeted policy updates. A complementary line of work provides dense supervision through policy distillation. On-policy distillation (OPD) queries a teacher distribution on states visited by the student policy, while on-policy self-distillation (OPSD) constructs the teacher using the same model under privileged contexts. Rubric-Guided Self-Distillation further supplies instance-specific rubrics as privileged information to the teacher. Although these methods offer token-level guidance, their objectives primarily emphasize matching the teacher distribution, without explicitly coordinating dense supervision with the preservation of autonomous policy exploration.

Recent work has begun to combine natural on-policy rollouts with externally guided trajectories. LUFFY integrates on-policy rollouts with off-policy expert demonstrations and introduces policy shaping to balance imitation and exploration. Critique-GRPO and RGR-GRPO further construct refined responses from critiques and rubrics, respectively, and jointly optimize them with the original rollouts. RuscaRL instead injects rubric-based scaffolds into rollout prompts, producing trajectories conditioned on additional guidance. These methods demonstrate that guided trajectories can expose the policy to high-quality behaviors that are difficult to discover through autonomous exploration alone. However, guided and natural trajectories often share the same optimization objective, coupling the learning signal from privileged trajectories with the relative advantages of natural samples. RISE-RL differs by filtering guided trajectories with the complete rubric, estimating their policy support after removing the privileged context, and selectively internalizing them through a separate auxiliary objective.

Method Summary

The method builds on Group Relative Policy Optimization (GRPO) for natural-prompt policy optimization. Given a prompt q, the old policy samples a group of G responses, each assigned a sequence-level reward. GRPO estimates the advantage of each response by normalizing its reward within the group. For each prompt q, a rubric C(q) = ck K k=1 is associated, where criterion ck has weight wk. For response oi, the rubric grader produces a binary judgment zi,k ∈ 0,1, and GRPO uses the aggregated sequence-level reward.

Criterion-Level Selective Feedback: Although suitable for policy optimization, the scalar reward obscures which important criteria are repeatedly missed across natural rollouts. Given natural rollouts, the priority of criterion ck is defined as pk = wk Σ(1 − zi,k), which jointly captures criterion importance and failure frequency within the current rollout group. The M highest-priority criteria are then selected as targeted feedback: C fb(q) = TopM(C(q); pk K k=1). This selection concentrates guidance on high-value capabilities that remain insufficiently covered by the current policy.

Quality-Filtered Selective Guidance: The selected criteria are appended to the original prompt and a group of privileged trajectories is sampled: q priv = q ⊕ C fb(q), opriv j ∼ πθold(· q priv). Because privileged conditioning may improve the injected criteria while degrading other quality dimensions, each candidate is re-evaluated using the complete original rubric. The reference advantage is computed as Aref j = r priv j − r̄nat, where r̄nat is the mean natural reward. Only reward-improving trajectories are retained: Oref = opriv j Aref j > 0.

To identify important yet under-supported tokens in reward-improving trajectories, the selected criteria are removed and each reference trajectory is teacher-forced under the original prompt: pj,t = πθ(oref j,t q, oref j,<t). Re-evaluating the retained trajectory under the original prompt estimates its support without privileged criteria: a low pj,t indicates that the corresponding token, despite appearing in a higher-reward trajectory, remains weakly supported by the natural policy. Following the saturating transformation in LUFFY, the policy support factor is defined as ρj,t = pj,t / (pj,t + γ), where γ > 0 controls the sensitivity to low-probability tokens. As pj,t decreases, −log ρj,t increases, assigning stronger guidance to weakly supported behavior while suppressing updates to already-supported tokens.

The trajectory-level weight is defined as Aej = clip(Aref j, 0, Amax), which discards non-improving trajectories and caps excessively large reward improvements. The selective guidance loss is then Lguide = −(1/Oref) Σ Σ (Aej/oref j) log ρj,t. This objective combines two forms of selectivity: reward filtering removes non-improving privileged trajectories, while policy support weighting focuses learning on higher-reward behaviors weakly supported by the current policy. It thus enables selective internalization rather than uniform imitation of the guided response. The token probabilities are recomputed under the current policy and backpropagation occurs only through it, treating trajectories, rewards, and trajectory weights as fixed. If Oref = ∅, Lguide = 0.

Decoupled Optimization and Guidance Removal: The filtered guidance signal is incorporated through a separate auxiliary objective: L(θ, s) = LGRPO(θ) + λguide(s)Lguide(θ), where s is the training step. Natural rollouts are optimized by GRPO, while filtered privileged trajectories contribute only through the auxiliary loss. This keeps them outside the reward normalization and relative-advantage estimation of natural rollouts, while allowing independent control over guidance strength and token-level weighting.

As the policy internalizes the guided behaviors, the reward advantage of privileged rollouts over natural rollouts gradually decreases. This additional benefit is measured by ∆r(s) = r̄priv(s) − r̄nat(s). A large ∆r(s) indicates that privileged feedback still elicits behaviors less accessible to the natural policy, whereas a narrowing gap suggests diminishing marginal benefit. Based on a preliminary run, sswitch is set near the onset of the plateau in the smoothed reward-gap curve, and λguide(s) = λ0 I[s < sswitch], removing guidance thereafter to continue training with pure GRPO.

Experiments Summary

Experimental Setup: The training corpus is constructed from RubricHub, a large-scale, multi-domain dataset equipped with fine-grained rubric-based evaluation criteria. The writing, health, chat, and science subsets are selected for reinforcement learning training and validation. The complete reinforcement learning corpus contains 86,355 prompts, including 29,418 Science examples, 29,681 Health examples, 17,444 Writing examples, and 9,812 Chat examples. Each reinforcement learning instance consists of an open-ended prompt and a set of weighted, instance-specific evaluation criteria. For each generated response, gpt-oss-120b independently determines whether each rubric criterion is satisfied.

Evaluation is conducted across four domains using open-ended generation and verifiable reasoning tasks: (1) Writing: WritingBench and CreativeWriting-V3; (2) Chat: Arena-Hard-v2; (3) Health: HealthBench, LLMEval-Med, and MedQA; (4) Science: ResearchQA and GPQA-Diamond. All evaluations follow official protocols. Baselines include SFT (fine-tuned on high-quality responses from RubricHub), OPD (on-policy distillation using Qwen3-14B as a teacher for the 4B model, and Qwen3-32B for the 14B model), Rubric-RL (standard RL using scalar rewards aggregated from fine-grained rubric criteria), and RuscaRL (a scaffold-mixed baseline that jointly optimizes rubric-conditioned and natural responses within a single group-relative objective).

For policy optimization, GRPO is used for the natural rollout branch. In RISE-RL, the shaping parameter is γ = 0.1, and the guidance weight λguide is set to 0.05 for writing and chat, and 0.01 for health and science domains. The number of injected criteria M is 5. All RL methods are trained separately for each domain.

Main Results: RISE-RL consistently achieves the strongest results across both Qwen3-4B and Qwen3-14B scales. In creative domains (Writing and Chat), RISE-RL shows substantial gains (up to +6.0 and +8.0 points on the 14B scale, respectively). On challenging benchmarks like CreativeWriting-V3, standard RL often struggles to discover high-reward trajectories or suffers from diversity collapse. RISE-RL's ability to target failed criteria exposes the model to rare creative modes, while its guidance removal mechanism ensures that the policy returns to unguided exploration, preserving creative openness.

In knowledge-dense domains (Health and Science), RISE-RL demonstrates that aligning with open-ended rubrics simultaneously reinforces performance on objective benchmarks. Training on Health and Science subsets leads to a +3.3 point gain on MedQA and a +3.6 point gain on GPQA-Diamond. RISE-RL maintains a significant lead over OPD and RuscaRL. OPD relies on teacher distillation, whose effectiveness depends on both the incremental value and the learnability of teacher trajectories. RuscaRL directly optimizes scaffold-conditioned samples, but the resulting behaviors do not always transfer reliably once the scaffold is removed. RISE-RL decouples privileged guidance from natural exploration, enabling more stable autonomous on-policy optimization.

Effect of Guidance Removal on Training Dynamics: RISE-RL exhibits a distinct two-stage evolution. (1) Expansion: During the early guided stage, policy entropy increases substantially, indicating a broader policy distribution—from 0.9 to 3.5 in Writing and from 0.5 to 2.5 in Health. This increase coincides with performance gains. (2) Consolidation: After guidanc…

Improvements to AI Systems:

1. Adaptive Criterion-Failure Targeting: The AI system can dynamically identify which specific evaluation criteria (e.g., factual accuracy, style, safety) are repeatedly failed during training and prioritize guidance on those exact gaps, rather than optimizing a single aggregate score. This enables more efficient learning of weak capabilities.

2. Selective Trajectory Internalization with Policy-Support Weighting: The system can filter high-reward trajectories by re-evaluating them without privileged context and weight learning toward tokens that are both high-value and weakly supported by the current policy. This prevents over-imitation of already-mastered behaviors and focuses updates on genuinely novel, beneficial actions.

3. Decoupled Auxiliary Guidance with Automatic Removal: The system can train with a separate auxiliary loss for guided trajectories, keeping them isolated from natural rollout reward normalization. It can automatically detect when guidance benefit diminishes (via reward-gap plateau) and remove guidance to resume pure on-policy exploration, preventing overfitting or instability.

4. Two-Stage Entropy Management: The system can deliberately expand policy entropy during early guided training (broadening exploration) and then consolidate after guidance removal (stabilizing at a higher entropy floor than baseline). This preserves creative diversity while improving performance, avoiding the diversity collapse common in standard RL.

5. Domain-Adaptive Guidance Strength: The system can adjust guidance weight based on domain characteristics—higher for creative tasks (writing, chat) and lower for knowledge-dense tasks (health, science)—to balance exploratory openness versus factual constraint adherence.

What the Improved AI System Can Do:

• Generate higher-quality open-ended responses (e.g., creative writing, dialogue, health advice, scientific explanations) that satisfy multiple rubric criteria simultaneously, with average score gains of 1.3–3.3 points over standard rubric-based RL.

• Maintain output diversity (+5.8% diversity gain in creative writing) while improving quality, avoiding the mode collapse typical of reward-maximizing policies.

• Transfer rubric-aligned skills to verifiable tasks: Training on open-ended rubrics improves performance on objective benchmarks (e.g., +3.3 points on MedQA, +3.6 points on GPQA-Diamond), enabling multi-task capability gains.

• Stably learn rare, high-value behaviors (e.g., consistently advising consultation with an obstetric provider in health responses) that unguided exploration fails to discover, reducing persistent criterion failures from 8/8 to 0/8.

• Self-regulate its training process: Automatically transition from guided to unguided exploration when marginal benefit diminishes, preventing performance degradation from over-guidance and maintaining long-term stability.