Ascent Conditions for Length-Normalized Single-Rollout Policy Updates
The analysis investigates whether length normalization preserves an ascent direction for expected task rewards in a softmax policy rollout. It finds that for binary rewards, length normalization can cause an early decrease in expected success, suggesting it doesn't universally preserve reward-centered updates. The paper also introduces unbiased statistics to estimate the return direction.