Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspective
The research analyzes training instability in large recommender systems using a Gradient-Weight-Loss (G-W-L) cascade model. It identifies that shifts like data distribution changes trigger a failure chain: exploding gradients lead to weight explosion, culminating in loss explosion. The paper proposes a comprehensive mitigation strategy by applying targeted fixes to gradients, weights, and the loss function simultaneously for superior stability.