Dual Dimensionality for Local and Global Attention
cs.CL, cs.AI
Submitted: 2026-06-17
Updated: 2026-11-01
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Program Synthesis with Large Language Models
- Longformer: The Long-Document Transformer
- Datasheet for the Pile
- SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Training Compute-Optimal Large Language Models
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Fast Transformer Decoding: One Write-Head is All You Need
- Efficient Streaming Language Models with Attention Sinks
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data