Neural Collapse Is Forbidden: Information Floors in Language Models
Bruno Abrahao
cs.LG, cs.CL, stat.ML
Submitted: 2026-07-10
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Neural Networks Learn Statistics of Increasing Complexity
- Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data
- Generalized Neural Collapse for a Large Number of Classes
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training
- Pointer Sentinel Mixture Models
- 2 OLMo 2 Furious
- In-context Learning and Induction Heads
- The Geometry of Categorical and Hierarchical Concepts in Large Language Models
- Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
- Imbalance Trouble: Revisiting Neural-Collapse Geometry
- Linguistic Collapse: Neural Collapse in (Large) Language Models
- Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
- Structure Before Collapse: Transient semantic geometry in next-token prediction
- A Geometric Analysis of Neural Collapse with Unconstrained Features
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks