Query Expansion and Key Specialization in Transformer Attention Geometry

arXiv:2609.34273 · cs.AI, cs.LG · Submitted 2026-09-28 · Read on arXiv

cs.AI, cs.LG

Submitted: 2026-09-28

Updated: 2026-09-28

Terminology

• Transformer — 1× in this paper · explained in 2 episodes

Sources

• Self-attention Networks Localize When QK-eigenspectrum Concentrates
• What Does BERT Look At? An Analysis of BERT's Attention
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
• Transformers Are Born Biased: Structural Inductive Biases at Random Initialization and Their Practical Consequences
• SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• DeepNet: Scaling Transformers to 1,000 Layers

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems