Large Language Models without QK-Normalization
cs.CL, cs.AI, cs.LG
Submitted: 2026-11-07
Updated: 2026-11-07
Code: https://github.com/yongding-tao/QK-MAP
Terminology
Sources
- Controlling changes to attention logits
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Chameleon: Mixed-Modal Early-Fusion Foundation Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Scaling Vision Transformers to 22 Billion Parameters
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Gemma 2: Improving Open Language Models at a Practical Size
- Gemma 3 Technical Report
- QK-Normed MLA: QK normalization without full key caching
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Time2Vec: Learning a Vector Representation of Time
- Kimi K2: Open Agentic Intelligence
- Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention
- Pointer Sentinel Mixture Models
- NoLiMa: Long-Context Evaluation Beyond Literal Matching
- Transformers without Tears: Improving the Normalization of Self-Attention
- 2 OLMo 2 Furious
- Olmo 3
- Methods of improving LLM training stability
- NormFormer: Improved Transformer Pretraining with Extra Normalization
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data