Adaptive Optimization via Momentum on Variance-Normalized Gradients
The episode discusses MVN-Grad, a new optimizer from UT Austin and Google Research, based on 'Adaptive Optimization via Momentum on Variance-Normalized Gradients.' It addresses instability in standard optimizers like Adam by replacing the second moment with variance for normalization and applying momentum after normalization. The authors prove this method is more stable, avoids loss spikes, and matches or beats current state-of-the-art models.