Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
A new method fine-tunes transformer models using sparse attention that works with any KV cache policy and runs efficiently on moderate hardware. This technique combines nested activation checkpointing, CPU offloading, and exploiting linear recurrence in the KV cache to handle long sequences. Experiments show this approach often outperforms exact attention methods like sequence parallelism.