Higher-Dimensional Rotary Position Embedding
cs.LG, cs.AI, cs.CL
Submitted: 2026-08-30
Updated: 2026-08-30
Comments: Accepted to EMNLP 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Transformers rely on position embedding mechanisms in long context modeling in most cases.
Terminology
Abstract
Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.
Sources
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
- Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
- The Llama 3 Herd of Models
- Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
- Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
- Scaling Laws of RoPE-based Extrapolation
- Extending Context Window of Large Language Models via Positional Interpolation
- The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
- LieRE: Lie Rotational Positional Encodings
- YaRN: Efficient Context Window Extension of Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- Self-Attention with Relative Position Representations
- SlimPajama-DC: Understanding Data Combinations for LLM Training
- Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Rope to Nope and Back Again: A New Hybrid Attention Strategy
- GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks