Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models
cs.LG, cs.CL
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/state-spaces/mamba
Terminology
Sources
- Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
- The Zamba2 Suite: Technical Report
- Kimi Linear: An Expressive, Efficient Attention Architecture
- ZClip: Adaptive Spike Mitigation for LLM Pre-Training
- No Language Left Behind: Scaling Human-Centered Machine Translation
- FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
- Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis