A Structural Theory of Position Bias in Transformers

The gist Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood, and this bias is closely connected to the Lost-in-the-Middle phenomenon where models underutilize information placed in the middle of the context.

Structural Theory of Position Bias

The paper develops a structural theory of position bias based on residual-aware cumulative attention rollout, showing that this behavior can arise from the architecture of causal Transformers themselves At finite depth, causal masking and residual connections induce broad, often U-shaped, influence profiles At infinite depth, the framework resolves a discrepancy between prior attention-only collapse theory and practical Transformer behavior by showing that residual connections fundamentally change cumulative attention dynamics.

Architectural Forces Shaping Position Bias

The authors formally prove that four architectural forces shape position bias at finite depth:

1. Causal masking induces primacy drift This is shown by Proposition 4.2, which states that for every prefix length k < n, the prefix mass is monotonically increasing in depth.

2. Residual connections induce recency drift Proposition 4.3 shows that stronger residual connections (i.e., decreasing λt) induce a recency drift in the last-row rollout distribution.

3. Positional encodings induce recency drift Proposition 4.4 demonstrates that recency-favoring positional encodings induce a recency drift in the last-row rollout distribution.

4. Content contributions modulate positional drift Proposition 4.5 shows that positive diagonal content induces a recency drift, while negative diagonal content induces a primacy drift.

Infinite-Depth Dynamics and Collapse Dichotomy

The analysis moves to infinite depth, where the asymptotic behavior of the residual-aware attention rollout P(T) is fully determined by the summability of the mixing coefficients.

(i) Finite total mixing ⇒ no collapse.

If P∞t=1 λt 0, meaning attention does not collapse to the first token.

(ii) Infinite total mixing ⇒ collapse.

If P∞t=1 λt = ∞, then for every token i, limT→∞ P(T)i1 = 1, and for all 1 0 such that P(T)ij → 0 for all j > 1 as T → ∞.

Empirical Validation against Measured Token Influence

The theory is validated by comparing predicted influence profiles to empirically estimated input-output token influence in pretrained language models.

(a) Attention-only rollout

The attention-only rollout exhibits collapse to the first token and fails to reproduce the empirical profile.

(b) Residual-aware rollout

The residual-aware architectural rollout with measured schedules produces a broad U-shaped profile, which closely matches the observed distributions.

(c) Residual-aware rollout with constant content

Incorporating the constant-plus-diagonal content abstraction modulates the U-shape by shifting mass toward later positions and shows very close alignment with measured distributions.

Quantitative Agreement and Conclusion

The normalized 1-Wasserstein distance between predicted and empirically measured input token influence distributions is consistently small, demonstrating that the predicted influence distributions from the rollout theory closely match the empirical distributions in shape. This agreement is particularly pronounced for larger models, suggesting that the simplifying assumptions underlying the theory become increasingly accurate with depth. The results support the view that Transformer architectures induce a strong U-shaped positional prior even in the absence of content.

The structural theory provides a quantitatively faithful approximation of influence propagation in pretrained Transformer models. The paper concludes that residual connections fundamentally alter deep attention dynamics, preventing inevitable collapse and establishing a principled perspective on position bias.

--- Page 1 ---

A Structural Theory of Position Bias in Transformers

Hanna Herasimchyk

University of Hamburg

Hamburg, Germany

hanna.herasimchyk@uni-hamburg.de

Robin Labryga

University of Hamburg

Hamburg, Germany

robin.labryga@uni-hamburg.de

Abstract

Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that Lost-in-the-Middle-type behavior can arise from the architecture of causal Transformers itself. To do so, we develop a structural theory of position bias based on residual-aware cumulative attention rollout. At finite depth, causal masking and residual connections induce broad, often U-shaped, influence profiles. At infinite depth, our framework resolves a discrepancy between prior attention-only collapse theory and practical Transformer behavior: residual connections fundamentally change cumulative attention dynamics. Empirically, the predicted profiles closely match measured input-token influence in pretrained language models.

1 Introduction

Despite their widespread success, the mechanisms by which Transformers propagate information across depth remain poorly understood. In particular, modern Transformer models exhibit strong and systematic position biases that persist across tasks, inputs, and even lexical scrambling [Rahimi et al., 2026]. These include primacy bias (preference for early tokens), recency bias (preference for recent tokens), and the Lost-in-the-Middle (LiM) phenomenon, where information in the center of long contexts is underutilized [Liu et al., 2024]. A growing body of empirical evidence suggests that these effects are not primarily semantic, but instead reflect architectural structure. Recent theoretical work has sought to explain position bias through attention rollout, interpreting stacked attention layers as a stochastic information propagation process across tokens. In a recent fundamental analysis, Wu et al. [2025] show that, in attention-only causal Transformers, cumulative attention must collapse onto the first token as depth increases. This prediction contradicts empirical results in modern Transformers, which do not collapse even at substantial depth. Wu et al. [2025] identify this mismatch as an open problem. We attribute this gap to a key omission: prior analyses ignore residual connections, despite their central role in modern Transformers. We develop a residual-aware theory of cumulative attention rollout and formally prove that four architectural forces shape position bias at finite depth: causal masking induces primacy; residual connections induce recency; relative positional biases such as ALiBi strengthen recency; and coarse content contributions modulate both. Their interaction yields broad, often U-shaped influence profiles, forming a structural prior consistent with Lost-in-the-Middle Preprint.

arXiv:2602.16837v2 [cs.LG] 27 May 2026

--- Page 1 ---

A Structural Theory of Position Bias in Transformers

Hanna Herasimchyk

University of Hamburg

Hamburg, Germany

hanna.herasimchyk@uni-hamburg.de

--- Page 2 ---

Contributions.

1. A residual-aware cumulative attention rollout mechanism can be implemented to model information propagation, which allows for a structural prior consistent with empirically observed position-dependent effects even in the absence of content-dependent mechanisms. This enables the system to generate predictions that match measured input-output token influence in pretrained language models.

2. The system can dynamically adjust its attention mixing strength based on layer magnitude, utilizing the residual mixing sequence-level coefficient λt(x) derived from the total signal strength to ensure stable and accurate representation learning across different depths.

3. The model can be architecturally tuned to counteract the Lost-in-the-Middle phenomenon by incorporating a content abstraction of the form constant plus diagonal self-attention, which provides a structural prior that shifts mass toward later positions, as shown in Proposition 4.5 and Figure 1.

Transformer models systematically favor certain token positions, yet the architectural origins of this position bias remain poorly understood. This bias is closely connected to the Lost-in-the-Middle phenomenon, where models underutilize information placed in the middle of the context. We show that causal Transformer architectures induce a systematic position-bias prior that can help explain the influence profiles associated with Lost-in-the-Middle. To characterize this prior, we develop a structural theory based on cumulative attention rollout with layerwise residual mixing. At finite depth, we isolate the competing positional drifts induced by causal masking, residual mixing, and positional encoding. Their interaction produces broad, often U-shaped profiles. At infinite depth, prior work predicts inevitable collapse under attention-only theory and identifies the discrepancy with observed Transformer behavior as an open problem. Under the stated boundedness assumptions, our residual-aware analysis replaces inevitable collapse with a dichotomy governed by cumulative attention mixing. Empirically, across 30 model-dataset settings, our adaptive residual-aware rollout closely matches measured input-token influence and reduces the average profile discrepancy by approximately 85% relative to both the attention-only and fixed-residual rollout baselines.