A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
The paper addresses large LLM KV-cache size by proposing Universal Attention (UA), a new architecture that replaces standard RoPE embeddings with a composite decay mechanism. This mechanism combines three types of decay functions—query-aware, key-aware, and similarity decay—to create an adaptive pruning criterion. UA enables highly expressive, end-to-end trainable pruning that removes tokens least contributing to attention computation, achieving up to 10x compression while improving performance over unpruned models.