Attention at Absolute Zero
cs.LG, cs.AI, cs.NE
Submitted: 2026-11-09
Updated: 2026-11-19
Code: https://github.com/NeoVand/witnesses
Project page: https://neovand.github.io/witnesses
Terminology
Sources
- Transformers Struggle to Learn to Search
- Transformers, parallel computation, and logarithmic depth
- Systematic Generalization with Edge Transformers
- Logic and the $2$-Simplicial Transformer
- Fast and Simplex: 2-Simplicial Attention in Triton
- Linearized 2-Simplicial Attention
- Attention Is All You Need
- Weisfeiler--Leman and Graph Spectra
- The CLRS Algorithmic Reasoning Benchmark
- FloydNet: A Learning Paradigm for Global Relational Reasoning
- Exponential Separations in Symmetric Neural Networks
- Poly-attention: a general scheme for higher-order self-attention
- Provably Powerful Graph Networks
- Graph as Point Set
- Overcoming a Theoretical Limitation of Self-Attention
- Scalable-Softmax Is Superior for Attention
- A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
- On Universal Equivariant Set Networks
- Neural Algorithmic Reasoning with Causal Regularisation
- Neural Algorithmic Reasoning Without Intermediate Supervision
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis