The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-03
Comments: Accepted to NeurIPS 2026 (Poster). Code is available at https://github.com/lianmin-chen/the_cost_of_mismatch
Code: https://github.com/lianmin-chen/the_cost_of_mismatch
License: http://creativecommons.org/licenses/by/4.0/
The gist: Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training…
Terminology
Abstract
Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training, yet its reliability in reinforcement learning with verifiable rewards (RLVR) remains largely unexplored. Motivated by the distinct data regimes of on-policy RL and fixed-data optimization, we develop a generic variance-theoretic framework to investigate what limits the reliability of ZO gradient estimation. Our analysis reveals that estimation reliability is structurally determined by whether the two finite-difference evaluations are performed on matched data. On-policy reinforcement learning (RL) under the mismatched-data regime produces a 1/μ squared oracle-noise amplification that sets an irreducible noise floor as the perturbation scale μ decreases, whereas Boltzmann Targeted SFT (BOLT) under the matched-data regime eliminates this amplification via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. These results motivate the ZO-BOLT algorithm, which builds on BOLT's matched-data advantage for reliable ZO optimization of an RLVR-aligned objective. Controlled perturbation-scale sweeps and matched-versus-mismatched end-to-end comparisons validate the predicted effect of data matching. Experiments across three models and two mathematical reasoning benchmarks further show that ZO-BOLT consistently improves over the pretrained baseline while retaining a near-inference memory footprint. The code is publicly available at https://github.com/lianmin-chen/the cost of mismatch.
Sources
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization
- Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
- Qwen2.5 Technical Report
- Measuring Mathematical Problem Solving With the MATH Dataset
- Proximal Policy Optimization Algorithms
- AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
- ZoAQ: Adaptive Zeroth-Order Querying via Query-Reuse Coupling
- Reinforcement Learning: An Overview
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning