Remember to Forget: Gated Adaptive Positional Encoding
The GAPE augmentation modifies Rotary Positional Encoding by adding adaptive gates that control how context is contracted and which key positions are protected from distance-based suppression. This decouples distance penalties from token importance, allowing the model to maintain long-context robustness up to eight times the training length while selectively suppressing irrelevant background information.