Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
stat.ML, cs.AI, cs.LG
Submitted: 2026-10-09
Updated: 2026-10-09
Code: https://github.com/wuqianyue09/OAL
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Linear Transformers with Learnable Kernel Functions are Better In-Context Models
- Simple linear attention language models balance the recall-throughput tradeoff
- Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity
- Transformers Are Born Biased: Structural Inductive Biases at Random Initialization and Their Practical Consequences
- The Llama 3 Herd of Models
- Massive Activations in Large Language Models
- Retentive Network: A Successor to Transformer for Large Language Models
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Efficient quadratic entropy with distance sketches
- Embedding-Bias in Conditional Independence Testing
- From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier