Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
cs.CL, cs.AI
Submitted: 2026-05-28
Updated: 2026-09-02
Comments: 8 page + 10 pages of bibliography and appendix
Code: https://github.com/LequeuISIR/DSTG-encoder
License: http://creativecommons.org/licenses/by/4.0/
The gist: Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood.
Terminology
Abstract
Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood. Modern PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval. Hence, a better understanding of the internal positional mechanism could help design better PE. Building on evidence that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, we modify an encoder Transformer to process three explicitly disentangled streams: semantic, absolute positional (AP) and relative positional (RP), and confine the masked-language-modeling (MLM) objective to the semantic stream. This decoupling enables a clean mechanistic study and yields three take-aways. (1) The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the structure of the document; (2) Attention heads specialize into structure and semantic-oriented groups, with RP exclusively supporting the latter; (3) Standard positional encodings do not robustly retain macroscopic structure: RoPE and RP only weakly encode it, and entangled AP loses it in the final layers under MLM pressure. The disentangled approach preserves positional encoding, which improves linguistic representation on 49 of the 65 linguistic phenomena of the Flash-Holmes probing benchmark.
Sources
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
- Contextual Position Encoding: Learning to Count What's Important
- Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Rethinking Positional Encoding in Language Pre-training
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Functional Interpolation for Relative Positions Improves Long Context Transformers
- GLU Variants Improve Transformer
- Uncovering hidden geometry in Transformers via disentangling position and context
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Decoupled Weight Decay Regularization
- Decoupling Positional and Symbolic Attention Behavior in Transformers
- Pointer Sentinel Mixture Models
- YaRN: Efficient Context Window Extension of Large Language Models
- Retrieval Head Mechanistically Explains Long-Context Factuality
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering