CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
cs.LG, cs.AI, cs.CL
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 28 pages, 11 figures, 5 tables
Code: https://github.com/huggingface/Math-Verify
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation.
Terminology
Abstract
Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose Cancellation-Aware Response Masking (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to 3.13 percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by 2.88 points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.
Sources
- Program Synthesis with Large Language Models
- Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
- Proximal Policy Optimization Algorithms
- Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
- Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
- Qwen3 Technical Report
- GLM-5: from Vibe Coding to Agentic Engineering
- Group Sequence Policy Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks