Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
SORL stabilizes off-policy training for LLM agents by combining turn-level importance sampling and clipping-triggered normalization. It fixes a granularity mismatch between token optimization and turn structure, while using adaptive scaling to suppress high-variance updates from unreliable samples. This results in more robust learning dynamics for multi-turn reasoning tasks.