SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
cs.NI, cs.AI, cs.PF
Submitted: 2026-04-23
Updated: 2026-10-08
Code: https://github.com/ggml-org/llama.cpp
Terminology
Sources
- The Llama 3 Herd of Models
- GPT-4 Technical Report
- Gemini: A Family of Highly Capable Multimodal Models
- Qwen3 Technical Report
- Mistral 7B
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Compute Or Load KV Cache? Why Not Both?
- LLM as a System Service on Mobile Devices
- XAttention: Block Sparse Attention with Antidiagonal Scoring
- SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
- Efficient Attentions for Long Document Summarization
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
- xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Related papers
- HiFiNet: Hierarchical Fault Identification in Wireless Sensor Networks via Edge-Based Classification and Graph Aggregation
- PACC: Protocol-Aware Cross-Layer Compression for Compact Network Traffic Representation
- TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces
- Embodied AI in 6G Networks: From Intelligent Connectivity to Physical Intelligence
- Lightweight GenAI for Network Traffic Generation: Fidelity, Augmentation, and Classification
- NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning