COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
Coupled Off-Policy Correction (COPC) addresses advantage staleness in asynchronous LLM reinforcement learning by introducing two correction channels: policy-side masking and advantage-side weighting. The research shows these channels interact multiplicatively, suggesting a coordinated approach is better. COPC achieves state-of-the-art performance and improved training stability by coupling these corrections.