STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens
cs.CL, cs.AI
Submitted: 2026-02-17
Updated: 2026-10-28
Comments: NeurIPS 2026
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and…
Terminology
Abstract
Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training. We identify a key factor behind this instability: a small fraction of tokens, termed spurious tokens (around 0.01%), which contribute little to the reasoning outcome but receive disproportionately amplified gradient updates due to inheriting the full sequence-level reward. We present a unified framework for evaluating token-level optimization impacts across spurious risk, gradient norms, and entropy changes. Building on the analysis of token characteristics that severely disrupt optimization, we propose the Silencing Spurious Tokens (S2T) mechanism to efficiently suppress their gradient perturbations. Incorporating this mechanism into a group-based objective, we propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement. Across six mathematical benchmarks and three model scales, STAPO consistently demonstrates superior entropy stability and achieves average accuracy improvements over baselines including GRPO, 20-Entropy, and JustRL under two widely-adopted evaluation settings.
Sources
- GPT-4 Technical Report
- Qwen3 Technical Report
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Soft Adaptive Policy Optimization
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
- A Survey of Reinforcement Learning for Large Reasoning Models
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
- GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
- BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
- OpenAI o1 System Card
- Proximal Policy Optimization Algorithms
- VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
- When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
- Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data