Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
The study investigated using only a small fraction of hidden dimensions for Rotary Positional Embedding (RoPE) to save memory while maintaining performance. Results show applying RoPE to about 10% of dimensions achieves convergence comparable to full RoPE, offering significant memory savings, especially for long contexts.