Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule
The discussion of 'Activation-Keyed Momentum' details a new optimization method using the Delta Rule to manage gradient history. This approach allows for targeted, direction-aware forgetting of stale data, leading to significant efficiency gains. Experiments show this method achieves comparable results to AdamW in up to forty-six point three nine percent fewer training steps on large language models.