Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval
cs.CL, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Code: https://github.com/ericenouen/shiftmech
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with global attention (SWA
Terminology
Abstract
Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with global attention (SWA NoPE). However, how these choices shape in-context retrieval remains unclear. To study this question, we take a mechanistic view, tracing how positional encoding (PE) choice shapes the internal mechanisms models use for in-context retrieval. Across 22 open-weight models spanning eight families, we find that standard RoPE models rely primarily on positional retrieval, while PE hybrids shift toward semantic retrieval. We further show on a controlled pre-training ablation that confining positional encoding to local layers produces this semantic shift, degrading representations of positional information. Finally, we show that the reported long-context gains of PE hybrids mask a retrieval trade-off: SWA NoPE improves over RoPE on multiple-target retrieval and QA, but degrades when distinguishing competing keys. We show that these behavioral differences better track the mechanism shift from positional toward semantic mechanisms than a uniform improvement in long-context retrieval.
Sources
- Round and Round We Go! What makes Rotary Positional Encodings useful?
- Longformer: The Long-Document Transformer
- NVIDIA Nemotron 3: Efficient and Open Intelligence
- Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
- RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
- Gemma 4 Technical Report
- Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
- The Llama 3 Herd of Models
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Gemma 3 Technical Report
- Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
- Kimi Linear: An Expressive, Efficient Attention Architecture
- Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
- Interpretability Can Be Actionable
- Language Models use Lookbacks to Track Beliefs
- SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
- Rethinking the Role of Efficient Attention in Hybrid Architectures
- Qwen3.5-Omni Technical Report
- Gemma 2: Improving Open Language Models at a Practical Size
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering