Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States".

Jane: Here's the gist of the paper. The gist The core question is not whether to use off-policy data, but where in the sequence it should appear, which leads to Thinking Seeds, a token-level mix-policy framework that uses historical checkpoints as off-policy prefixes and on-policy continuations to preserve gradient quality.

Tom: On-policy reinforcement learning for language models suffers from a fundamental tension where policy entropy collapses as training progresses, causing the model to “forget” its own earlier exploratory capacity Page 1, "as training progresses, policy entropy collapses and sampling diversity diminishes, causing the model to “forget” its own earlier exploratory capacity." Existing methods mix entire trajectories at the sequence level, which introduces severe policy mismatch and training instability Page 1, "existing methods mix entire trajectories at the sequence level, introducing severe policy mismatch and training instability." The paper reveals that intra-sequence entropy decreases along positions in standard on-policy training without mini-batching Page 3, "As shown in Figure 2, under standard on-policy training without mini-batching (Figure 2a), policy entropy decreases sharply along sequence positions, indicating that the model becomes increasingly deterministic as generation progresses within each rollout." Furthermore, when mini-batch updates are employed, the high-entropy tokens are immediately clipped and fail to contribute effective gradients Page 3, "When mini-batch updates are employed (Figure 2b), typically introduced for computational efficiency to enable larger effective batch sizes, a side effect is that the policy drifts from the reference policy across mini-batches, creating a mild off-policy effect that raises entropy." Thinking Seeds Framework The Thinking Seeds framework operationalizes the where-not-whether principle by using historical checkpoints as “thinking seeds” Page 2, "the model’s own historical checkpoints serve as natural “thinking seeds.” Rather than importing trajectories from external stronger models (which introduces distribution incompatibility and resource overhead), Thinking Seeds recycles the model’s own earlier diversity by using historical checkpoint outputs as sequence prefixes, providing diverse thinking starting points, while the critical continuation that drives policy updates is generated on-policy." The method constructs a complete sample oi by truncating the behavior policy output sθbehavior to form a prefix and then conditioning the reference policy πθold to generate the suffix sθ conditioned on this prefix Page 5, "For a given prompt q, we truncate the response generated by πθbehavior as: sθbehavior ∼ truncate(s), s ∼ πthetabehavior(· q), and conditioned on this prefix, the reference policy πthetaold continues to generate the suffix: sθ ∼ πthetaold (·

q, sthetabehavior: )." Token-Level Importance Ratios To manage the distribution shift introduced by mixing data sources, Thinking Seeds employs token-level importance ratios to fine-grained control gradient contributions Page 5, "During the optimization phase, we further distinguish the sampling sources at the token level and compute the corresponding importance ratio accordingly." The importance ratio ri,t(θ) is defined as a function that accounts for tokens in both the off-policy prefix and on-policy suffix Page 5, "its importance ratio is defined as: ri,t(θ) = min (πtheta(oi,j q) / πthetabehavior(oi,j q), oi,j ∈ sthetabehavior, πtheta(oi,k

q, sthetabehavior: ) / πthetaold (oi,k

q, sthetabehavior: ), oi,k ∈ sθ." This fine-grained modeling circumvents the sample-level clipping saturation problem by allowing tokens in the off-policy prefix to continuously contribute effective gradient signals within a controlled shift range Page 5, "Under the token-level GRPO objective function inherited from DAPO (Equation 1), this fine-grained importance ratio modeling circumvents the sample-level clipping saturation problem." Experimental Results and Analysis Extensive experiments across models and mathematical reasoning benchmarks demonstrate that Thinking Seeds consistently outperforms standard on-policy training and existing off-policy extensions Page 5, "Through experiments, Thinking Seeds consistently outperforms on-policy training and existing off-policy extensions across models and mathematical reasoning benchmarks." The method maintains higher effective entropy, reduces gradient loss from clipping, and expands the explorable solution space Page 5, "Our analysis reveals that the method maintains higher effective entropy, reduces gradient loss from clipping, and expands the explorable solution space." Specifically, Thinking Seeds exhibits more stable reward behavior compared to standard on-policy methods, characterized by smaller fluctuations in the reward curve and a smoother convergence process Page 8, "Thinking Seeds exhibits more stable reward behavior compared to the standard on-policy method, characterized by smaller fluctuations in the reward curve and a smoother convergence process." The analysis confirms that the position-aware design successfully resolves the entropy-clipping dilemma identified in prior work Page 9, "This confirms that the position-aware design, placing off-policy tokens in the prefix where they enrich context and on-policy tokens in the suffix where they must contribute gradients, successfully resolves the entropy-clipping dilemma identified in our motivation." Thinking Seeds highlights that fine-grained, token-level mix-policy modeling, guided by positional awareness, is a practical and effective direction for advancing RL in LLMs Page 7, "This yields our core insight: the question is not whether to use off-policy data, but where in the sequence it should appear." The paper concludes that position-aware integration within individual samples can resolve the entropy-clipping dilemma that plagues both pure on-policy training and naive off-policy mixing Page 7, "Our key insight is that the question of off-policy data is not whether but where: position-aware integration within individual samples can resolve the entropy-clipping dilemma that plagues both pure on-policy training and naive off-policy mixing." The proposed framework uses historical checkpoints as off-policy prefixes to preserve diversity while keeping on-policy continuations for gradient quality Page 7, "we propose Thinking Seeds, which uses the model’s own historical checkpoints as off-policy prefixes that preserve diversity, while keeping on-policy continuations for gradient quality." The method consistently outperforms other baseline methods across different training strategies and models Page 8, "Thinking Seeds consistently outperforms both standard on-policy training and existing off-policy methods across two model backbones." This work demonstrates that position-aware mix-policy modeling improves both exploration and final performance in LLM RL Page 1, "clarifying how position-aware mix-policy modeling improves both exploration and final performance in LLM RL." The limitations noted include the lack of guarantee that the chosen off-policy policy is optimal for exploration or credit assignment Page 14, "the off-policy prefix is generated from a historical policy checkpoint that is refreshed every T batches, it does not guarantee that the chosen off-policy policy is optimal for exploration or credit assignment." The paper suggests exploring more adaptive strategies, such as dynamically selecting off-policy checkpoints or incorporating heterogeneous policy sources Page 14, "Exploring more adaptive strategies, such as dynamically selecting off-policy checkpoints or incorporating heterogeneous policy sources, is an interesting direction for future work." The final conclusion is that the method provides a practical and effective direction for advancing RL in LLMs Page 7, "Thinking Seeds highlights that fine-grained, token-level mix-policy modeling, guided by positional awareness, is a practical and effective direction for advancing RL in LLMs." The work confirms that the gains arise from unified single-sample mix-policy modeling rather than merely introducing additional newly sampled tokens Page 8, "Nevertheless, all non-zero ratio settings outperform pure on-policy training (ratio = 0%), indicating that the gains arise from unified singlesample mix-policy modeling rather than merely introducing additional newly sampled tokens." The final finding is that Thinking Seeds consistently leverages off-policy data at both the single-sample and cross-temporal levels Page 8, "Overall, together with the ratio ablation, these results show that Thinking Seeds consistently leverages off-policy data at both the single-sample and cross-temporal levels." The paper shows that different truncation strategies may exhibit varying effectiveness across models and settings Page 8, "At the same time, we observe that different truncation strategies may exhibit varying effectiveness across models and settings.

Title and authors: Tom: So we're looking at this paper called "Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States," and it tackles a really fundamental problem in training language models for reinforcement learning. It seems to be about how we can use old, diverse reasoning paths to keep the model from getting stuck in a narrow loop during its main training process.

Jane: That sounds like it could be huge if we're trying to get these models to reason better and explore more ideas without losing that initial spark of creativity. What's the core idea behind this "Thinking Seeds" approach?

Tom: The main thing is that they argue the question isn't whether you should use data from before, but rather where in the sequence you put it. They propose a token-level mix-policy framework where they use the model’s own historical checkpoints as off-policy prefixes, which are like diverse starting points for reasoning, while they keep the critical continuation on-policy to make sure we get good gradient quality.

Lu: I think what's interesting about this is that instead of bringing in external models or importing whole trajectories from a different policy, Thinking Seeds recycles the model’s own earlier diversity by using those historical checkpoint outputs as prefixes, which they say avoids distribution incompatibility and resource overhead.

Meng: From my side, I'm always worried about stability at scale. If we mix data poorly, it just introduces noise that makes deployment unpredictable. How does this method handle that instability when you're dealing with mini-batch updates?

Tom: That's where the paper gets pretty detailed on the math. They introduce token-level importance ratios to fine-tune exactly how much gradient each token contributes from both the off-policy prefix and the on-policy suffix, which they say circumvents that sample-level clipping saturation problem.

Jane: So it’s not just mixing sequences at a high level; it’s managing the influence of individual tokens within the sequence, which is a much more precise way to handle distribution shifts during training. What are some of the results they show?

Tom: They show that Thinking Seeds consistently outperforms standard on-policy training and existing off-policy extensions across different models and mathematical reasoning benchmarks. They also report maintaining higher effective entropy and reducing gradient loss from clipping, which suggests it expands the explorable solution space rather than just adding noise.

Lu: The analysis confirms that by placing off-policy tokens in the prefix where they enrich context, and on-policy tokens in the suffix where they need to contribute gradients, they successfully resolve the entropy-clipping dilemma that plagued previous work.

Title and authors: Meng: If it’s resolving a problem like that, does it mean we can push for longer chain-of-thought generations without worrying as much about the policy collapsing early on? Because right now, those long sequences are really fragile.

Tom: Exactly. They show more stable reward behavior compared to standard on-policy methods, with smaller fluctuations in the reward curve and a smoother convergence process, which is pretty important for reliable AI systems.

Jane: That stability sounds like it could translate directly into more reliable performance in complex tasks where the reasoning needs to be sustained over a long generation. So this "where-not-whether" approach really seems to solve the core tension they identified earlier.

Lu: It does, because they are using the model’s own historical checkpoints, which are distributionally compatible and require no external models for that diversity, which is a big win for training efficiency.

Meng: But I have to ask about the practical deployment side. How complex is this token-level importance ratio modeling to implement in a production system? Is it just adding more computation overhead?

Tom: The paper shows they are using a token-level GRPO objective function inherited from DAPO, and that this fine-grained importance ratio modeling actually circumvents the sample-level clipping saturation problem, which is a key technical detail for gradient control.

Jane: That sounds like it’s a more controlled way to handle the data flow than just relying on larger batch sizes to smooth things out. So what about those specific results they show regarding different position ratios?

Tom: They show that under standard on-policy training without mini-batching, entropy collapses along positions, but with mini-batch updates, the clipping ratio surges proportionally when you look at tokens in the off-policy prefix versus the on-policy suffix.

Lu: But Thinking Seeds uses those historical checkpoints which are distributionally compatible and require no external models for that diversity, which is a big win for training efficiency.

Meng: I still have to wonder about the limitations they admit. What is it that this method doesn't guarantee? Because in my experience, if you can't guarantee something critical like optimal credit assignment, it’s not ready for real traffic yet.

Tom: The paper states a limitation clearly: the off-policy prefix is generated from a historical policy checkpoint that is refreshed every T batches, and it does not guarantee that the chosen off-policy policy is optimal for exploration or credit assignment.

Jane: So they’re essentially saying that while this works well for improving exploration and final performance in mathematical reasoning, we still need to look at more adaptive strategies, like dynamically selecting off-policy checkpoints.

Title and authors: Lu: That’s a fair point; exploring more adaptive strategies, such as dynamically selecting off-policy checkpoints or incorporating heterogeneous policy sources, is an interesting direction for future work. It suggests the current static approach isn't the final answer for long-term exploration.

Meng: If we look at this in terms of deployment cost, does this complexity translate into a significant increase in inference time when we actually start serving these models? Because running more complex sampling logic adds latency that matters for real-time applications.

Tom: The paper shows different truncation strategies may exhibit varying effectiveness across models and settings, so the choice you make has implications for both exploration and performance metrics.

Jane: So to wrap this up on "Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States," it’s a framework that uses historical checkpoints as off-policy prefixes to preserve diversity, while keeping on-policy continuations for gradient quality. It seems like a very practical way to manage the entropy-clipping dilemma in language models.

Lu: Overall, together with the ratio ablation, these results show that Thinking Seeds consistently leverages off-policy data at both the single-sample and cross-temporal levels. It's about unifying single-sample mix-policy modeling rather than just introducing additional newly sampled tokens.

Meng: I see how it addresses the gradient clipping issue, which is a major stability concern for me when training large models, but I still need to see if the overhead of calculating those token importance ratios is manageable for our current infrastructure.

Tom: It does show that this position-aware design successfully resolves the entropy-clipping dilemma identified in prior work. It’s a clear demonstration of how where you place things matters for training stability and final performance.

Jane: So, the big picture here is that fine-grained, token-level mix-policy modeling guided by positional awareness is a practical direction for advancing RL in language models. We get to see how this helps explore more effectively while keeping the learning process stable.

Tom: It’s clear that Thinking Seeds consistently outperforms both standard on-policy training and existing off-policy methods across two model backbones, which is a strong indicator of its utility here.

Lu: This work confirms that position-aware mix-policy modeling improves both exploration and final performance in LLM RL, clarifying how position-aware mix-policy modeling improves both exploration and final performance in LLM RL.

Meng: For production deployment, the fact that it maintains higher effective entropy is good because it suggests the model isn't overfitting too quickly to a narrow set of solutions during training.

Jane: It’s definitely a solid piece of research for understanding how to make these models more robust when we try to guide them through complex reasoning tasks. We’ll be keeping an eye on those future work directions, like dynamically selecting those checkpoints.

The paper's summary: Tom: So, we're talking about "Thinking Seeds" now that we’ve seen the setup. The core idea is that instead of just dumping old data in, they use those historical checkpoints as a starting point—a prefix—and then keep generating new material on-policy to make sure the updates are good.

Jane: So, if I get it right, the paper is saying it’s not about whether you use off-policy data, but where exactly you put it in the sequence of generation. It’s a token-level mix-policy framework.

Lu: Exactly. They're using the model's own past reasoning states as "thinking seeds" to keep that diversity, while the actual continuation that drives policy updates is generated on-policy. It’s about recycling internal variety instead of pulling in external models.

Meng: I still have to wonder how they handle the noise from those older paths when you start updating the model with new data. It sounds complicated managing two different sources of tokens at once.

Tom: That’s where they get technical, Meng. They introduce these token-level importance ratios to fine-tune exactly how much gradient each token contributes from both the old prefix and the new suffix. This is meant to prevent that sample-level clipping saturation problem we see in standard training.

Jane: So instead of just mixing whole sequences, they're controlling which individual tokens are important for learning based on their position. It’s a very precise way to manage that shift in data distribution during training.

Lu: The results show this approach is consistent across different models and mathematical reasoning benchmarks, actually outperforming existing methods. They maintain higher effective entropy and reduce gradient loss from clipping, which means the model explores more solutions.

Tom: That’s what I like to hear—a clean ablation showing that mixing at a fine-grained level actually helps performance instead of just being a computational headache.

Jane: It sounds like this isn't just about getting better numbers; it’s about stabilizing the training process and giving the model more room to think during complex tasks.

Lu: It addresses the entropy-clipping dilemma by placing those off-policy tokens where they enrich context, and the on-policy tokens where they need to provide gradient signals. That position awareness is really key there.

Meng: I see how that resolves some of my stability concerns, Jane. If it keeps the gradient quality high in the suffix, we’re less likely to get those sudden drops in performance when we update with new batches.

Tom: Yeah, and they show it leads to more stable reward behavior overall. Smaller fluctuations in the reward curve means smoother convergence when you're training these models for reasoning.

Jane: So what does this mean for us? It suggests that instead of just pushing the model harder on-policy, we can strategically use its own history to guide exploration and maintain quality during the learning phase.

Lu: It confirms that unified single-sample mix-policy modeling is a practical way to improve both exploration and final performance in language model reinforcement learning.

Tom: But they are careful about what they don't guarantee. They state that because the prefix comes from a checkpoint refreshed every T batches, it doesn't guarantee that the chosen off-policy policy is optimal for credit assignment or exploration.

Jane: That’s an important caveat. It tells us we still need to look at more adaptive strategies, like deciding which historical checkpoints to use dynamically, instead of just picking one static starting point.

Lu: Right, exploring those adaptive strategies would be a natural next step for improving the robustness of this whole framework.

Meng: I think that’s realistic. If we can figure out how to make the checkpoint selection smarter or more dynamic, that’s where you start thinking about deployment costs versus gain.

Jane: So, while this paper gives us a very strong way to handle the "where" of mixing data in a single sample, the next challenge is figuring out the best "when."

The paper's improvements: Tom: So, we’re talking about how they suggest we take this Thinking Seeds framework and make it even better for real deployment. It’s not just about getting the numbers right on a benchmark; they want to know what happens when you try to push its limits.

Jane: Right, so once you have this token-level mix-policy running, what’s the next step? Are they suggesting we change how we choose those historical checkpoints or how we define the importance ratios?

Lu: They suggest exploring more adaptive strategies. They acknowledge that using a fixed checkpoint refreshed every T batches isn't always optimal for credit assignment or exploration.

Tom: I like that. It means moving beyond just a static setup where you pick one old state and stick with it. It points toward a system that can intelligently select the best historical starting point based on what’s happening right now during training.

Jane: So, it’s not just about making the initial idea work, but building a mechanism that decides which past ideas are most useful for the current task. That shifts the focus from just having good data to being smart about how you use that data over time.

Lu: They also mention incorporating heterogeneous policy sources. This means instead of relying solely on one type of historical model output, you could mix in outputs from different types of reasoning models to see what helps the overall performance.

Meng: If we’re talking about deployment, I wonder if that "heterogeneous policy" approach is just going to make our inference pipeline way more complicated to run. We need things simple and fast on production hardware.

Tom: The goal there is still performance, Meng. They are aiming for a framework that balances the exploration benefit of diversity with the practical constraints of training stability and gradient quality in a production environment.

Jane: It sounds like they're trying to solve the trade-off between having enough diverse ideas to explore and keeping the learning process smooth enough not to break down when you’re running massive models.

Lu: That’s the core implication: it pushes us toward more sophisticated control over the mixing process, moving from a fixed recipe to a dynamic one.

Tom: It shows that this isn't just one solution for RL in LLMs, but a direction where we need to look at how the data is selected and how that selection changes as training progresses.

Jane: So, once you have this framework, the next big question for us will be designing those adaptive strategies—how do we automate that decision-making process?

Conclusion: Tom: So we’re wrapping up on "Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States." The whole point is that this token-level mix-policy modeling, guided by position awareness, is a practical way to handle the entropy-clipping dilemma.

Jane: That means they successfully showed that you don't have to choose between having diverse starting points and keeping your training stable. It’s about finding the sweet spot in how you blend old and new data within a single generation sample.

Lu: The main takeaway is that this position-aware integration resolves the issue where high-entropy tokens get clipped while learning actually happens in the suffix. It clarifies exactly where each piece of information should contribute its gradient signal.

Meng: From a deployment standpoint, it’s promising because it keeps things stable during training, which means we don't have to constantly worry about sudden crashes when we introduce new data batches.

Lalam: I’m confident in my answer here; the method consistently leverages off-policy data at both single-sample and cross-temporal levels, which is a huge win for keeping the model’s reasoning robust.

Tom: It definitely holds up against other methods, showing it outperforms standard on-policy training and existing off-policy extensions across different model backbones.

Jane: So, we're looking at a framework that improves both exploration and final performance in mathematical reasoning by focusing on this fine-grained control over tokens.

Lu: Yes, the position-aware design is what actually makes the difference, because it manages the distribution shift much more effectively than just mixing entire trajectories.

Meng: Still, we have to remember their caveat: they don't guarantee that the off-policy prefix is optimal for credit assignment or exploration in every single case.

Tom: Exactly; they admit that it’s a starting point, and that’s where future work needs to go—making the selection of those historical checkpoints smarter.

Jane: So, the direction now is looking at more adaptive strategies, like dynamically picking the best checkpoints based on what the model is currently trying to learn.

Lalam: I think that's a solid path forward for making this framework truly production-ready and versatile.

Lu: It opens up a lot of research on how to make those checkpoint selections more intelligent and less arbitrary.