Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

arXiv:2607.18130 · cs.LG · Submitted 2026-07-20 · Read on arXiv

cs.LG

Submitted: 2026-07-20

Updated: 2026-10-07

Code: https://github.com/mlco2/codecarbon

Terminology

• Manifold-Constrained Hyper-Connections — 2× in this paper
• Transformer — 2× in this paper · explained in 2 episodes
• Benchmark — 1× in this paper · explained in 2 episodes

Sources

• Lessons from the Trenches on Reproducible Evaluation of Language Models
• PIQA: Reasoning about Physical Commonsense in Natural Language
• go-$m$HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
• Deep Residual Learning for Image Recognition
• Measuring Mathematical Problem Solving With the MATH Dataset
• LoRA: Low-Rank Adaptation of Large Language Models
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
• VeRA: Vector-based Random Matrix Adaptation
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Spectral-Sphere-Constrained Hyper-Connections
• Decoupled Weight Decay Regularization
• TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes
• 2 OLMo 2 Furious
• Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections
• JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
• Overtrained Language Models Are Harder to Fine-Tune

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks