Gefen: Optimized Stochastic Optimizer
Gefen is a memory-efficient optimizer designed to reduce AdamW's large memory footprint by about 8 times. It achieves this by automatically grouping parameters whose squared gradients are similar and sharing second-moment estimates within these groups. This method maintains AdamW's performance while requiring no extra hyperparameters or architecture-specific metadata.