Modified Loss of Momentum Gradient Descent: Fine-Grained Analysis
The paper analyzes Polyak heavy-ball momentum optimization to show it is equivalent to plain gradient descent with a modified loss on an exponentially attractive manifold. It proves the algorithm can be approximated by a memoryless iteration with arbitrary precision, deriving continuous modified equations and revealing combinatorial structures in the approximation coefficients.