NOSA: Native and Offloadable Sparse Attention
NOSA introduces a trainable sparse attention mechanism designed to improve decoding throughput when GPU memory is limited by large KV caches. It achieves significant speedups by adding a locality constraint during training, forcing selected tokens to be reused across decoding steps. This results in higher throughput without sacrificing performance on long-context and long-generation tasks.