Underscoring the Problem: Why Softpick Fails at Initialization
cs.LG
Submitted: 2026-10-04
Updated: 2026-10-04
Sources
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Generating Long Sequences with Sparse Transformers
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- Scaling Laws for Neural Language Models
- Sparse and Continuous Attention Mechanisms
- Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention
- GLU Variants Improve Transformer
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- LLaMA: Open and Efficient Foundation Language Models
- Attention Is All You Need
- Replacing softmax with ReLU in Vision Transformers
- Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks