Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
cs.CL, cs.AI
Submitted: 2026-05-08
Updated: 2026-11-06
Code: https://github.com/gkamradt/LLMTest_NeedleInAHaystack
Terminology
Sources
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Longformer: The Long-Document Transformer
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
- Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
- SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
- CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction
- The Llama 3 Herd of Models
- AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Efficient Attentions for Long Document Summarization
- Mistral 7B
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data