Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Trajectory-Aware Policy Optimization (TAPO) addresses training instability in long-horizon LLM reinforcement learning caused by compounding state distribution shifts. Existing methods fail because they only correct local action mismatches. TAPO derives a principled state-distribution correction from the off-policy gradient, uses a geometric mean for variance reduction, and employs length-aware clipping to stabilize training while improving performance on reasoning and coding tasks.