BoT-GRPO: Efficient Process-Reward RL for Reasoning via Bag-of-Token Aggregation
BoT-GRPO is a critic-free extension of Group Relative Policy Optimization that uses a length-invariant bag-of-tokens aggregation to assign per-token advantages from external token rewards. It works by combining global outcomes and local signals into token rewards, ensuring equal contribution from all sequences regardless of length, which significantly speeds up convergence in reasoning tasks without needing value networks.