TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference
cs.LG, cs.AI
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/HanzhiZhang-Ulrica/TriAttention
License: http://creativecommons.org/licenses/by/4.0/
The gist: Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained.
Abstract
Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interactions differ in their precision requirements: some tolerate low-precision scores while remaining sensitive to removal. We therefore introduce TriAttention, which assigns query-key (QK) tiles to skipping, 4-bit, or 8-bit computation, using 4-bit to retain quantization-tolerant interactions at lower cost. To represent compactly, we encode each head's computation regions with two diagonal-distance boundaries, keeping policy storage independent of context length. Offline calibration then evaluates candidate boundary pairs using normalized projected-output errors and complete-tile execution costs. Using these measurements, a model-wide allocator selects each head's boundaries to minimize calibration error under a shared computation budget. During prefill, the selected boundaries generate precision-grouped traversals for a fused kernel that skips omitted tiles and combines contributions from both QK precisions into one normalized attention output. The quantized keys and values prepared during prefill are then reused in a compressed key-value (KV) cache for decoding, reducing repeated reads of prompt data. Experiments show that TriAttention achieves end-to-end generation speedups of up to 7.88x over the Torch attention and 2.65x over the BF16 baseline. Meanwhile, TriAttention maintains quality comparable to the original models on general-purpose and long-context tasks. These results show how a compact retention-and-precision policy translates local sensitivities into attention computation and repeated KV-cache access. The implementation is https://github.com/HanzhiZhang-Ulrica/TriAttention.
Sources
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- The Llama 3 Herd of Models
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
- XAttention: Block Sparse Attention with Antidiagonal Scoring
- LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
- SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis