Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

The paper introduces learnability as a complementary dimension of task value in reinforcement learning (RL) post-training for large language models (LLMs), distinct from the commonly used snapshot-based notion of solvability. The authors define learnability as a regime-conditional measure of expected positive response to continued training under a fixed RL post-training regime. They argue that tasks with similar current solvability (e.g., similar pass rates) can differ sharply in how they respond to continued optimization, and that existing task-valuation methods, which rely on instantaneous performance signals, fail to capture this residual axis.

The paper makes three main contributions:

1. It distinguishes learnability from solvability and operationalizes it as a task's regime-conditional positive response to continued training.

2. It provides empirical evidence that learnability is reproducible across independently sampled training contexts and predictive of downstream utility, even among tasks with matched initial solvability.

3. It proposes TRAJVAL, a lightweight probe-based estimator of learnability that can be used either as a standalone static prior for task sampling or as a multiplicative prior for existing online task schedulers.

Key findings and methodology:

• Heterogeneous per-task learnability: By analyzing per-task reward trajectories over 20 epochs of GRPO training on a diagnostic pool of 2,048 math tasks, the authors observe that tasks with similar initial pass rates can follow sharply different reward trajectories. They define six coarse trajectory profiles (Stable, Sluggish, Ineffective, Mastered, Unlearned, Forgetting) based on reward gain (Δ), standard deviation (σ), and linear-fit R2.

• Context-robustness and predictive value: Learnability is shown to be reproducible across independently sampled training contexts (Cohen's κ = 0.776 for six profiles, 0.879 for binary learnable grouping on Qwen3-1.7B; similar results on Llama-3.2-3B). Restricting training to learnable tasks (≈41% of the pool) matches full-pool converged accuracy in 3.6× fewer steps, and an oracle-based soft prior outperforms random sampling (0.410 vs 0.387).

• TRAJVAL estimator: The method applies two reductions to make oracle learnability practical:

• Reduction I: Replaces dense trajectory tracking with an endpoint-based summary using early-phase and late-phase mean rewards (r̄ early, r̄ late) and their difference Δ.

• Reduction II: Replaces full-pool training with a short probe run on a small uniformly sampled subset (P ≪ D), using the probe-trained model as a proxy for a later training state.

The continuous learnability score is defined as: s(x) = (1 − r̄ early(x)) · max(0, Δ(x)), where the first term captures learning headroom and the second captures directional alignment. The score is converted to sampling weights via power compression and a probability floor: w(x) = max(s(x) α, ε), with α = 0.3 and ε = 0.05 as defaults.

• Integration with online schedulers: TRAJVAL composes multiplicatively with existing online schedulers: w final(x, t) = w TRAJVAL(x) · w online(x, t), providing a stable offline prior that complements online solvability estimates, particularly early in training when online evidence is noisy.

Experimental results:

• Main results: Across Math (DAPO-Math, 17,398 tasks) and Logic (GURU Logic subset, 4,998 tasks) domains, with Qwen3-1.7B and Qwen3-4B models, TRAJVAL consistently improves both convergence speed and peak performance over uniform sampling (BASE-GRPO). As a plug-in prior, it improves Best Acc and AUC for BOTS and GRESO in every configuration, with S2B reductions in all cases (e.g., 40% S2B on Logic at 4B scale).

• Within-pre-bin analysis: To isolate learnability from solvability, tasks are partitioned into 3 pre-bins (low, mid, high) based on r̄ early, then split into high-Δ and low-Δ sub-pools with matched average pre. Across all six bin-domain combinations, high-Δ sub-pools uniformly outperform low-Δ counterparts in both Best Acc and AUC. For example, in the Low-pre Math bin, high-Δ tasks reach 0.434 Best Acc versus 0.338 for low-Δ tasks.

• Ablations:

• Score composition: All single-dimension variants (pre-only, post-only, Δ-only) underperform full TRAJVAL, indicating headroom and directional alignment are complementary.

• Probe set size: P = 512 (≈3% of pool) provides the best overall trade-off.

• Hyperparameter sensitivity: Across α ∈ 0.1, 0.3, 0.5, all settings improve over Random.

• Computational overhead: TRAJVAL adds a one-time probe run on ≈3% of the pool and two inference passes; per-step sampling overhead is negligible (≈0.01% of total pipeline time).

• Cross-regime reuse: Same-family transfer (Qwen3-1.7B→Qwen3-4B) retains useful signal (S2B 66.0%), while cross-family transfer (Llama-3.2-3B→Qwen3-4B) helps less (S2B 84.0%); in-regime estimation achieves the strongest performance.

Limitations: The method is evaluated on mathematical and logical reasoning tasks with binary reward signals; transfer to non-binary or model-graded rewards (e.g., LLM judge) remains open. Evaluation is confined to text-only settings, and experiments are conducted on models up to 8B parameters.

Improvements to AI systems:

1. Task-aware curriculum scheduling: The AI system can dynamically prioritize training on tasks with high learnability (high headroom × directional alignment) rather than only high solvability. This means it can converge to peak performance in up to 3.6× fewer steps, reducing compute cost while maintaining or improving final accuracy.

2. Early-training task selection: The system can use TRAJVAL as a static prior before online training begins, avoiding the noise of early reward signals. This allows it to avoid wasting gradient updates on tasks that are either already mastered or will not improve, focusing compute on tasks with the most residual learning potential.

3. Robust cross-scale transfer: The system can transfer learnability priors from a smaller model (e.g., 1.7B) to a larger one (e.g., 4B) within the same family, retaining 66% of the speed-to-benefit advantage. This enables efficient hyperparameter-free warm-starting of larger models without full re-estimation.

4. Multiplicative integration with online schedulers: The system can combine a stable offline learnability prior with existing online solvability-based schedulers (e.g., BOTS, GRESO). This yields consistent improvements in both best accuracy and area-under-curve across all tested configurations, with up to 40% reduction in steps-to-benefit on logic tasks.

5. Low-overhead task valuation: The system can estimate learnability with a one-time probe run on only 3% of the task pool (e.g., 512 tasks) and two inference passes, adding negligible per-step overhead (0.01% of pipeline time). This makes it practical for large-scale RL post-training without significant engineering cost.

6. Profile-aware data filtering: The system can classify tasks into six trajectory profiles (e.g., Stable, Sluggish, Ineffective, Mastered, Unlearned, Forgetting) and use this to automatically exclude or down-weight tasks that are Unlearned or Forgetting during training, preventing catastrophic forgetting and improving training stability.

7. Solvability-matched performance gains: The system can achieve higher final accuracy even when initial task pass rates are matched. For example, in low-solvability math tasks, selecting high-learnability tasks yields 0.434 best accuracy vs. 0.338 for low-learnability tasks—a 28% relative improvement, enabling better use of limited training budgets.

8. Compute-efficient scaling: The system can apply learnability priors to reduce total training steps for a target accuracy, effectively lowering the carbon footprint and cost of RL post-training for LLMs, especially in domains with large task pools (e.g., 17k+ math tasks).