BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation

arXiv:2610.09804 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation".

Jane: The gist The BoT-GRPO algorithm proposes a critic-free extension of Group Relative Policy Optimization that uses a length-invariant bag-of-tokens aggregation to efficiently assign per-token advantages from external token rewards,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: It suggests that the core question for efficient reasoning isn't just what to reward—like a specific rule or a judge—but rather how to fold any such token-level signal into GRPO so that it accelerates convergence without adding an unbounded cost.

Tom: The paper is looking at Monte-Carlo estimation methods and verifier models, which usually have high costs because they either roll out many continuations per step or require separate training pipelines for the reward model itself #pg2.

Lu: They address this by proposing BoT-GRPO as an orthogonal solution that doesn't depend on the choice of signal, whether it’s a generative judge or a rule-based one, because it focuses on how that signal is aggregated into per-token advantages fairly across variable-length rollouts #pg3.

Meng: So they are trying to decouple the design of the step-level signal itself from the cost of calculating its credit assignment in the RL loop.

Jane: That’s right, they are focusing on creating a fair way to assign per-token advantages using group statistics that normalize for sequence length #pg1.

Tom: It’s about turning any existing token-level reward into advantages without needing a separate value network or an auxiliary model to train just for that purpose.

Lu: The mathematical mechanism involves collecting all token-level rewards across rollouts and weighting each by the inverse of its source sequence length, then computing per-token advantages relative to those weighted group statistics #pg1.

Jane: I think what’s interesting is how they handle the local signal S(k)t within that aggregation, showing that even a bounded local signal doesn't break the length-invariant normalization if the global term is strong enough #pg2.

Meng: So, they are essentially saying that as long as you can get a token reward, this framework gives you credit assignment for free in terms of new infrastructure.

Tom: It’s about densifying feedback by folding those granular signals into GRPO to accelerate convergence without adding an unbounded cost that grows with the length of reasoning.

Lu: The idea is that this approach is applicable wherever token-level rewards are available, making it a versatile tool for various supervision methods #pg2.

Jane: It’s about showing that token-level credit assignment can be achieved in a way that's robust to the specific type of external reward you're using.

Tom: So, we’ve established the mechanics; now we see how those mechanics translate into real performance gains across different types of reasoning tasks.

The paper's summary: Jane: They show significant efficiency gains on both front-end code generation and AIME mathematical reasoning, which is pretty impressive because those are two very different types of tasks.

Tom: For React front-end code generation, BoTGRPO reaches an eighty percent compile rate up to one point nine times faster than GRPO and it converges faster than GSPO, DAPO, and PURE.

Lu: That speedup means the system can reach target quality in markedly fewer steps while also achieving higher final compile and VLM-judged win rates #pg1.

Meng: And for AIME mathematical reasoning, they reported absolute Pass@k gains up to eight point one percent over GRPO in half the steps. That’s a substantial jump for that kind of problem.

Jane: These results are important because they hold across three base-model families: Qwen2 point 5-3B, SmolLM3-3B, and Phi4 mini reasoning, showing that the advantage is domain- and architecture-agnostic #pg1.

Tom: The fact that it’s domain agnostic is what makes this interesting; it suggests this technique isn't just a niche fix for one problem area.

Lu: It demonstrates that token-level credit assignment is an efficiency lever that works across different architectures and domains, which was the main goal when they designed BoT-GRPO #pg3.

Meng: From a deployment standpoint, if we can get this kind of performance boost without needing to train a massive value network, it drastically simplifies our training pipeline requirements.

Jane: It really shifts the focus from just finding the best reward signal to figuring out how to aggregate whatever signal you have into GRPO effectively #pg1.

Tom: It’s about being able to use existing feedback mechanisms more effectively without incurring the massive cost of building new supervision infrastructure.

The paper's improvements: Jane: To sum it up, BoT-GRPO is a critic-free extension of GRPO that consumes any external token-level reward and assigns per-token advantages through a length-invariant bag of tokens aggregation #pg1.

Tom: It achieves convergence speedups without the cost of value networks or auxiliary models, making it well suited for reasoning model training under real efficiency constraints.

Lu: The overall contribution is that token-level credit assignment can be used as a domain- and architecture-agnostic efficiency lever, which is pretty significant #pg3.

Meng: It’s a method that densifies feedback by folding token-level signals into GRPO to accelerate convergence without adding an unbounded cost.

Jane: We’re looking at how this changes the conversation around process supervision—it shows that we don't always need new models to get better credit assignment.

Tom: It’s a method that lets us get faster results on complex reasoning tasks like math and code generation using feedback we already have in place.

Lu: We hope this work encourages more research into how different token-level signals can be integrated into RL frameworks in this efficient way #pg1.

Conclusion: Tom: So to wrap up, BoT-GRPO is this critic-free way to use any external token reward by aggregating them using a length-invariant bag of tokens to speed up convergence in reasoning tasks.

Jane: Right, so it takes whatever signal you have—whether it’s from a rule or another model—and folds it into the Group Relative Policy Optimization loop without needing those extra value networks.

Lu: The core idea is that by using this weighted group statistics approach, you get per-token advantages even when you’re dealing with variable sequence lengths, which is usually where things break in standard GRPO setups.

Meng: I like the fact that it doesn't require training a separate auxiliary model just to handle the rewards; it just works if you already have those token-level signals available.

Lalam: The paper shows that this mechanism can incorporate any combination of a global outcome and a local signal, as long as you set the weighting coefficients right. I’m pretty sure it’s very robust because of that length-invariant normalization you mentioned earlier.

Jane: Exactly, so even if your local signal is noisy or bounded, the global outcome term dominates enough to prevent those longer sequences from completely dominating the advantage calculation.

Tom: And on the empirical side, they showed real speedups. We’re talking about eighty percent compile rate improvement in code generation and a substantial Pass@k gain on AIME math reasoning compared to standard GRPO variants.

Lu: That domain-agnostic result is what really stands out; it proves that token-level credit assignment is a universal lever, not just a fix for one specific type of reward signal.

Meng: From my side, the implication is that we can actually deploy this kind of feedback mechanism on production models without adding complexity or latency that scales with sequence length.

Jane: It means we can move closer to having models learn from more granular, fine-grained feedback in a way that’s computationally efficient for real-world use.

Lalam: I think the main thing is how it changes the cost structure of reinforcement learning for complex tasks; you get credit assignment without adding unbounded costs.

Tom: So, we’ve seen how BoT-GRPO takes existing token rewards and makes them work efficiently inside GRPO, achieving tangible speedups across different reasoning benchmarks.

Lu: It’s a solid piece of work because it tackles the credit assignment problem directly within the policy update mechanism itself.

Jane: Yeah, it gives us a concrete recipe for how to design reward functions that actually accelerate training when we have those kinds of signals available.

Meng: I just hope that when we look at this at scale, the aggregation mechanism doesn't become a bottleneck in our inference pipeline.

Lalam: The authors acknowledge that the AIME task still needs an LLM judge for step evaluation, which means it’s not a complete end-to-end solution yet.

Tom: Still, it’s a massive step forward because the efficiency gains alone make it worth looking at seriously for our next generation of reasoning models.

Lu: We should keep an eye on this because I think this approach to folding local signals into group statistics will be useful for other representation learning problems too.

Jane: Right, so BoT-GRPO is a strong foundation for making RL feedback more granular and efficient across the board.

Tom: Next time we look at a paper, we’ll see if they can push this even further with zero-shot adaptation to entirely new reward structures.

Yingxiang Yang, Weihang Xiao, Zhunxuan Wang

Amazon AGI · Virginia Tech

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Comments: Published at the COLM 2026 Workshop on Efficient Reasoning

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist The BoT-GRPO algorithm proposes a critic-free extension of Group Relative Policy Optimization that uses a length-invariant bag-of-tokens aggregation to efficiently assign per-token

Key concepts

Bag-of-Tokens Aggregation: This technique collects all tokens from multiple generated sequences into one pool. Each token is weighted by the inverse of its source sequence length (wk = 1/Lk). This process aggregates tokens into a single group statistic, ensuring that the contribution of each sequence to the overall learning signal is normalized and equal, regardless of how long the original text was.
Length-Invariant Normalization: This crucial design principle prevents longer sequences from dominating the advantage calculation. By weighting tokens inversely by their source length (wk = 1/Lk), the algorithm ensures that shorter sequences contribute proportionally to the group statistics. This corrects a bias found in standard GRPO, making each sequence's impact on learning equally weighted.
Token Reward Construction: Token rewards are constructed as a combination of a global outcome (Gk) and a local signal (S(kt)) using coefficients wg and ws: r(k)t = wg · Gk + ws · S(kt). This allows the algorithm to incorporate any external token-level reward, balancing general success with specific, bounded local feedback from teachers or rules.
Critic-Free Extension: BoT-GRPO extends Group Relative Policy Optimization (GRPO) without using a separate value network or auxiliary models. Instead of relying on complex critic estimations, it directly assigns per-token advantages based on the aggregated token rewards. This makes the method highly efficient and suitable for training large reasoning models under strict computational constraints.

Terminology

Aggregation — 1× in this paper · explained in 1 episode
GRPO — 1× in this paper · explained in 6 episodes

Summary

The gist The BoT-GRPO algorithm proposes a critic-free extension of Group Relative Policy Optimization that uses a length-invariant bag-of-tokens aggregation to efficiently assign per-token advantages from external token rewards, accelerating convergence in reasoning tasks.

How it works

BoT-GRPO extends GRPO by consuming any external token-level reward and assigning per-token advantages through a bag of tokens aggregation mechanism The algorithm collects all tokens across rollouts into a single pool, weights each by wk = 1/Lk (inverse source sequence length), computes weighted group statistics, and calculates per-token advantages relative to these statistics The length-invariant normalization ensures that each sequence contributes equally to the group statistics regardless of length which corrects the response-length bias plaguing standard GRPO This mechanism allows BoT-GRPO to make no assumptions about the source of token rewards, as it can incorporate any combination of a global outcome and a bounded local signal works

Key Components and Design Principles

The core innovation lies in how token-level signals are integrated into the GRPO framework The method constructs token rewards as r(k)t = wg · Gk + ws · S(k)t (Equation 4), where Gk is the global outcome, S(k)t is a local signal from a fixed teacher or rule-based model, and wg, ws are weighting coefficients The length-invariant normalization ensures that each sequence contributes equally to the group statistics regardless of length which prevents longer sequences from dominating the baseline calculation Furthermore, BoT-GRPO departs from standard Monte Carlo estimators by constructing token rewards as r(k)t = wg · Gk + ws · S(k)t The influence of local rewards is controlled via ws/wg, and when local rewards are bounded (S(k)t ≤ B), the local component perturbs each sequence’s length-normalized contribution to the group statistics by at most a length-independent constant

Empirical Results and Performance

The algorithm demonstrates significant efficiency gains on both front-end code generation and AIME mathematical reasoning For React front-end code generation, BoTGRPO reaches 80% compile rate up to 1.9× faster than GRPO and converges faster than modern GRPO variants like GSPO, DAPO, and PURE while reaching higher final compile and VLM-judged win rates On the AIME mathematical reasoning task, BoT-GRPO delivers absolute Pass@k gains up to 8.1% over GRPO in half the steps The results hold across three base-model families (Qwen2.5-3B, SmolLM3-3B, and Phi-4-mini reasoning), showing that the advantage is domain- and architecture-agnostic

Practical Reward Recipe and Limitations

The paper provides a practical recipe for reward models, emphasizing that reward stability matters more than richness: clean, bounded, stable fine-grained signals consistently accelerate learning where noisier alternatives stall For code generation, the reward construction involves combining global code quality with line-level error localization using deterministic static analysis and LLM/VLM judges A key finding is that relying only on compiler and linter signals leads to reward hacking where the policy learns to emit a syntactically plausible JSX skeleton that renders a blank page, necessitating the addition of an LLM code judge to restore quality gains The paper notes limitations, such as the AIME reward needing an LLM judge for step evaluation, which adds cost and latency

Conclusion

BoT-GRPO is presented as a critic-free extension of GRPO that consumes any external token-level reward and assigns per-token advantages through a length-invariant 'bag of tokens' aggregation It achieves convergence speedups without the cost of value networks or auxiliary models, making it well suited for reasoning model training under real efficiency constraints The findings establish token-level credit assignment as a domain- and architecture-agnostic efficiency lever The overall contribution is a method that densifies feedback by folding token-level signals into GRPO to accelerate convergence without adding an unbounded cost

--- Page 1 ---

Key Components and Design Principles

Empirical Results and Performance

The algorithm demonstrates significant efficiency gains on both front-end code generation and AIME mathematical reasoning For React front-end code generation, BoTGRPO reaches 80% compile rate up to 1.9× faster than GRPO and converges faster than modern GRPO variants like GSPO, DAPO, and PURE while reaching higher final compile and VLM-judged win rates On the AIME mathematical reasoning task, BoT-GRPO delivers absolute Pass@k gains up to 8.1% over GRPO in half the steps The results hold across three base-model families (Qwen2.5-3B, SmolLM3-3B, and Phi-4 mini reasoning), showing that the advantage is domain- and architecture-agnostic

--- Page 2 ---

Improvements for AI systems

textbfProcessing Efficiency via BoT-GRPO for Token-Level Credit Assignment: BoT-GRPO reaches 80% compile rate up to 1.9× faster than GRPO and converges faster than modern GRPO variants (GSPO, DAPO, PURE) in React front-end code generation. This allows the system to reach target quality in markedly fewer steps, accelerating convergence without adding a value network or per-token cost that grows with sequence length: BoT-GRPO is critic-free, and is a drop-in replacement wherever GRPO is used when token-level reward is available.
textbfHandling Reward Hacking via Bounded Auxiliary Signals: The local component degenerates into unterminated attributes and repeated or placeholder tokens when only compiler/linter signals are used, resulting in a blank page. BoT-GRPO mitigates this by using bounded local rewards: "Because the global term dominates (wg >> ws·B), a mislabeled local step cannot override the outcome of the sequence it belongs to."
textbf Achieving Domain-Agnostic Reasoning Gains: The efficiency gains transfer to a very different domain, with BoT-GRPO achieving absolute Pass@k gains up to 8.1% over GRPO in half the steps on AIME mathematical reasoning. This demonstrates that token-level credit assignment is a domain- and architecture-agnostic efficiency lever.

Abstract

Reinforcement learning is now central to eliciting reasoning in large language models, while in the popular algorithm Group Relative Policy Optimization (GRPO) every token in a rollout receives the same advantage. We ask how to make process supervision efficient: accelerating convergence and improving final quality without the cost of value networks. We propose Bag-of-Tokens Group Relative Policy Optimization (BoT-GRPO), which extends GRPO to token-level reward models through a length-invariant "bag of tokens" aggregation: it collects all token-level rewards across rollouts, weights each by the inverse of its source sequence length, and computes per-token advantages relative to weighted group statistics. BoT-GRPO is critic-free, and is a drop-in replacement wherever GRPO is used when token-level reward is available. On React front-end code generation, BoT-GRPO reaches 80% compile rate up to 1.9 times faster than GRPO and converges faster than modern GRPO variants (GSPO, DAPO, PURE) while reaching higher final compile and VLM-judged win rates. On a second task, AIME mathematical reasoning, BoT-GRPO delivers absolute Pass@ k gains up to 8.1% over GRPO in half the steps. For both tasks we compare the algorithm's performance on reasoning vs. non-reasoning base-model families (Qwen2.5-3B, SmolLM3-3B, Phi-4-mini-reasoning). Our experiments also yield a practical recipe for the reward model itself: reward stability matters more than richness: clean, bounded, stable fine-grained signals consistently accelerate learning where noisier alternatives stall.