One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse
cs.LG
Submitted: 2026-08-03
Updated: 2026-09-30
Code: https://github.com/xieTwim/one-qk-channel-artifact
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- Controlling changes to attention logits
- Chameleon: Mixed-Modal Early-Fusion Foundation Models
- PaLM: Scaling Language Modeling with Pathways
- Pretraining large language models with MXFP4 on Native FP4 Hardware
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- DeepSeek-V3 Technical Report
- Scaling Vision Transformers to 22 Billion Parameters
- Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
- Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training
- Deep Learning with Limited Numerical Precision
- Query-Key Normalization for Transformers
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
- Kimi K2: Open Agentic Intelligence
- ZClip: Adaptive Spike Mitigation for LLM Pre-Training
- Oscillation-free Quantization for Low-bit Vision Transformers
- Mixed Precision Training
- Online normalizer calculation for softmax
- A Theory on Adam Instability in Large-Scale Machine Learning
- 2 OLMo 2 Furious
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks