Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel
cs.CL
Submitted: 2026-01-15
Updated: 2026-11-02
Code: https://github.com/ymgw55/proj-kernel-attn-subspace
Project page: https://transformerlensorg.github.io/TransformerLens/generated/demos/Main_Demo.html
Terminology
Sources
- Layer Normalization
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
- The geometry of BERT
- Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
- Sparse Attention Decomposition Applied to Circuit Tracing
- Qwen2 Technical Report
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data