KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers
cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/l-trochelmann/KiteNorm
Terminology
Sources
- Layer Normalization
- Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
- Don't be lazy: CompleteP enables compute-efficient deep transformers
- The Unreasonable Ineffectiveness of the Deeper Layers
- Training Compute-Optimal Large Language Models
- Lambda-Skip Connections: the architectural component that prevents Rank Collapse
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
- Decoupled Weight Decay Regularization
- A Johnson--Lindenstrauss Framework for Randomly Initialized CNNs
- In Search of Adam's Secret Sauce
- GLU Variants Improve Transformer
- The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
- The Curse of Depth in Large Language Models
- Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis