When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt, Martin Potthast, Gerhard Heyer
cs.CL
Submitted: 2026-08-04
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Neural Combinatorial Optimization with Reinforcement Learning
- Longformer: The Long-Document Transformer
- Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
- RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
- Gemma 2: Improving Open Language Models at a Practical Size
- Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
- LLaMA: Open and Efficient Foundation Language Models
- BloombergGPT: A Large Language Model for Finance
- Baichuan 2: Open Large-scale Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering