RLTL;DR: Self-improvement by Internalizing Self-generated Feedback
cs.LG, cs.AI, cs.CL, stat.ML
Submitted: 2026-09-29
Updated: 2026-09-29
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones.
Terminology
Abstract
The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.
Sources
- Context Bootstrapped Reinforcement Learning
- Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
- A General Language Assistant as a Laboratory for Alignment
- Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
- Nudging the Boundaries of LLM Reasoning
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Contextual Drag: How Errors in the Context Affect LLM Reasoning
- STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving
- GLM-5: from Vibe Coding to Agentic Engineering
- iGRPO: Self-Feedback-Driven LLM Reasoning
- Reinforcement Learning via Self-Distillation
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- SPICE: Self-Play In Corpus Environments Improves Reasoning
- Understanding R1-Zero-Like Training: A Critical Perspective
- Policy and World Modeling Co-Training for Language Agents
- SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
- Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning
- CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- ECHO: Terminal Agents Learn World Models for Free
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks