Clustering in pure-attention hardmax transformers and its role in sentiment analysis
cs.CL, cs.LG, math.DS, stat.ML
Submitted: 2024-06-26
Updated: 2026-11-17
Code: https://github.com/DCN-FAU-AvH/clustering-hardmax-transformers
Terminology
Sources
- Exact Sequence Interpolation with Transformers
- Emergence of meta-stable clustering in mean-field transformer models
- A deep language model to predict metabolic network equilibria
- The Llama 3 Herd of Models
- Effects of padding on LSTMs and CNNs
- A mathematical perspective on Transformers
- Dynamic metastability in the self-attention model
- Adam: A Method for Stochastic Optimization
- GPT-4 Technical Report
- Generative Language Modeling for Automated Theorem Proving
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data