GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
cs.CL, cs.AI, cs.LG
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: Findings of the 2026 Conference on Empirical Methods in Natural Language Processing
Code: https://github.com/outongyiLv/GMTS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating
Terminology
Abstract
Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.
Sources
- GPT-4 Technical Report
- LLaDA2.1: Speeding Up Text Diffusion via Token Editing
- SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
- Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index
- FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
- CommonsenseQA 2.0: Exposing the Limits of AI through Gamification
- GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
- Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
- Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
- VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank
- Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
- Diversity-Aware Policy Optimization for Large Language Model Reasoning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering