Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras
cs.LG
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: 14 pages, 2 figures. Preprint
License: http://creativecommons.org/licenses/by/4.0/
The gist: We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators T=OV nearly closes under composition, T 2 aboutαT.
Terminology
Abstract
We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators T=OV nearly closes under composition, T 2 aboutαT. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment P at least0.9, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, T=Q OKQ V and T 2=Q O(KDK)Q V, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation of K while preserving singular values, norms, factor spans, and principal angles reduces median closure from 0.336 to 1.04 times10-4; trained orientation wins for 98.64% of heads and in every layer. Constructive searches show that high closure is feasible in every surveyed layer, but usually not attained. Retrospective trajectories in three independently trained lineages further separate broadly available capacity from the orientations attained by final strong heads. Under exact value sharing, headwise closure extends to a right-action algebra, T iT j=α jT i. Seven-model experiments verify the approximate law and reveal distinct oblique projections with a shared value-defined kernel. These results characterize scaled idempotence as a sparse trained orientation within broadly available geometric capacity and show how value sharing extends a headwise relation into a local operator algebra.
Sources
- Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits
- Mistral 7B
- In-context Learning and Induction Heads
- Qwen2.5 Technical Report
- 2 OLMo 2 Furious
- The Communication Map of a Transformer
- Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel
- Qwen3 Technical Report
- TinyLlama: An Open-Source Small Language Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks