On the Design of Hyper-Connections: Simplification, Stability, and Scalability
cs.LG, cs.CL
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/aHapBean/SimpleHC
Terminology
Sources
- Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- go-$m$HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- The Llama 3 Herd of Models
- oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- Muon is Scalable for LLM Training
- Spectral-Sphere-Constrained Hyper-Connections
- Decoupled Weight Decay Regularization
- Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
- Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections
- Multi-Stream Attention Residuals
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis