PowerStep: Memory-Efficient Adaptive Optimization via p-Norm Steepest Descent

Name: PowerStep: Memory-Efficient Adaptive Optimization via p-Norm Steepest Descent
Uploaded: 2026-10-03T06:05:57.889529+00:00
Duration: 472 s
Description: PowerStep is a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics, unlike Adam. It applies a nonlinear transform to a momentum buffer, proving optimal O(1/√T) convergence for non-convex optimization while matching Adam's speed on large models.

PowerStep is a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics, unlike Adam. It applies a nonlinear transform to a momentum buffer, proving optimal O(1/√T) convergence for non-convex optimization while matching Adam's speed on large models.

Open the full episode