Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference
The research introduces VeXact, a zero-mismatch rollout engine designed to eliminate small numerical disagreements between training and inference engines (TIM). By comparing standard methods against VeXact, the study proves that TIM is a critical destabilizing factor in Reinforcement Learning training collapse when using the REINFORCE objective. This shows TIM fundamentally changes the optimization problem, skewing gradients before other effects become visible.