Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation
cs.LG
Submitted: 2026-11-01
Updated: 2026-11-08
Code: https://github.com/homw314159-spec/causal-ot-attention
Terminology
Sources
- Layer Normalization
- Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge
- On the Convergence of Semi-Relaxed Sinkhorn with Marginal Constraint and OT Distance Gaps
- Why Do Accumulated Transformations Extrapolate?
- RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings
- Scaled-Dot-Product Attention as One-Sided Entropic Optimal Transport
- You Need Better Attention Priors
- First Passage Attention
- Limitations of Normalization in Attention Mechanism
- Scalable-Softmax Is Superior for Attention
- Unbalanced Optimal Transport, from Theory to Numerics
- Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection
- Towards Infinite Length Extrapolation: A Unified Approach
- Group Representational Position Encoding
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis