A Structural Theory of Position Bias in Transformers
The paper develops a structural theory of position bias in Transformers using residual-aware cumulative attention rollout. It shows that causal masking and residual connections create broad, U-shaped influence profiles at finite depth, resolving discrepancies with simpler models. This theory explains how architectural forces shape biases like primacy and recency, proving that residual connections fundamentally change deep attention dynamics.