Do Orthogonality Constraints Help Language-Model Training Beyond Sphere Optimization?
cs.LG
Submitted: 2026-10-31
Updated: 2026-10-31
Terminology
Sources
- Demystifying Manifold Constraints in LLM Pre-training
- LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
- OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models
- Mano: Restriking Manifold Optimization for LLM Training
- Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
- Muown Implicitly Performs Angular Step-size Decay
- Normalized Low-Rank Adaptation
- Muown: Row-Norm Control for Muon Optimization
- Muon is Scalable for LLM Training
- Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining
- The Llama 3 Herd of Models
- Rethinking Language Model Scaling under Transferable Hypersphere Optimization
- Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
- Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
- L2 Regularization versus Batch and Weight Normalization
- Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
- Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
- Hyperball May Not Be a Free Lunch
- Controlled LLM Training on Spectral Sphere
- Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis