Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning

arXiv:2610.05685 · cs.CL · Submitted 2026-10-05 · Read on arXiv

cs.CL

Submitted: 2026-10-05

Updated: 2026-10-07

Comments: 17 pages, 4 figures, 13 tables

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Decode — 1× in this paper · explained in 2 episodes

Sources

• Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
• Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
• Palu: Compressing KV-Cache with Low-Rank Projection
• xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
• Value-Aware Stochastic KV Cache Eviction for Reasoning Models
• SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
• SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
• QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
• ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
• Measuring Mathematical Problem Solving With the MATH Dataset
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering