LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention
cs.AI
Submitted: 2026-10-03
Updated: 2026-10-03
Code: https://github.com/MuLabPKU/TransArch
Terminology
Sources
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- Longformer: The Long-Document Transformer
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
- xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
- Generating Long Sequences with Sparse Transformers
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- MiniMax Sparse Attention
- PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
- MoBA: Mixture of Block Attention for Long-Context LLMs
- WebGPT: Browser-assisted question-answering with human feedback
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
- You Only Cache Once: Decoder-Decoder Architectures for Language Models
- CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
- KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems