LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
cs.LG, cs.CL
Submitted: 2026-11-04
Updated: 2026-11-04
Terminology
Sources
- Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifold
- Palu: Compressing KV-Cache with Low-Rank Projection
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- QAQ: Quality Adaptive Quantization for LLM KV Cache
- Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA
- The Llama 3 Herd of Models
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
- KVzap: Fast, Adaptive, and Faithful KV Cache Pruning
- KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
- MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
- On First-Order Meta-Learning Algorithms
- Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
- Linear Predictability of Attention Heads in Large Language Models
- KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
- RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations
- Gemma 2: Improving Open Language Models at a Practical Size
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
- ThinK: Thinner Key Cache by Query-Driven Pruning
- ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis