ConvPE: Convolutional Position Embedding Improves Transformers
cs.LG
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/wzhagi/convpe
Terminology
Sources
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- The Impact of Positional Encoding on Length Generalization in Transformers
- Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
- TileLang: A Composable Tiled Programming Model for AI Systems
- Kimi K3: Open Frontier Intelligence
- GLU Variants Improve Transformer
- Fast Transformer Decoding: One Write-Head is All You Need
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Pointer Sentinel Mixture Models
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Adam: A Method for Stochastic Optimization
- HellaSwag: Can a Machine Really Finish Your Sentence?
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Rotary Position Embedding for Vision Transformer
- Longformer: The Long-Document Transformer
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis