Adaptive Variance Reduction without Knowing Oracle Smoothness
The method optimizes non-convex functions using only stochastic gradients when oracle smoothness is unknown. It uses a two-phase approach: first, comparing two gradients to switch to normalized momentum; second, applying a variance reduction update. This yields expected gradient norm bounds of O(T - 1/3) under average smoothness, O(T - 1/4) under standard smoothness, and O(T - 1/2) noiselessly for a known budget T.