Learning Rate Transfer and Feature Learning Across Depth for Constrained Spectral Optimizers: Complete Scion
cs.LG, math.OC
Submitted: 2026-11-11
Updated: 2026-11-11
Code: https://github.com/elinepot/CompleteSCION
Project page: http://leloykun.github.io/ponder/spectral-clipping
Terminology
Sources
- Correction of Decoupled Weight Decay
- Don't be lazy: CompleteP enables compute-efficient deep transformers
- Optimal Scaling Needs Optimal Norm
- Training Compute-Optimal Large Language Models
- On the Role of Batch Size in Stochastic Conditional Gradient Methods
- Kimi K2: Open Agentic Intelligence
- Kimi K2.5: Visual Agentic Intelligence
- Attention Residuals
- Adam: A Method for Stochastic Optimization
- Weight Decay may matter more than muP for Learning Rate Transfer in Practice
- Training Transformers with Enforced Lipschitz Constants
- Training Neural Networks at Any Scale
- Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales
- Rethinking Language Model Scaling under Transferable Hypersphere Optimization
- Learning Rate Transfer in Normalized Transformers
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Arcee Trinity Large Technical Report
- mHC: Manifold-Constrained Hyper-Connections
- On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer
- A Spectral Condition for Feature Learning
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis