Read What Matters: Query-Adaptive Quantization for KV Caches
cs.LG, cs.CL, cs.PF
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: 50 pages, 3 figures, 12 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Yi: Open Foundation Models by 01.AI
- Optimal compression of approximate inner products and dimension reduction
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Error-Correcting Data Structures
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- QAQ: Quality Adaptive Quantization for LLM KV Cache
- ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
- Mistral 7B
- PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
- On Locally Decodable Source Coding
- Landmark Attention: Random-Access Infinite Context Length for Transformers
- Softmax is $1/2$-Lipschitz: A tight bound across all $\ell_p$ norms
- Loki: Low-rank Keys for Efficient Sparse Attention
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
- Qwen2.5 Technical Report
- No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
- RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks