Can an MLP Absorb Its Own Skip Connection Exactly?
cs.LG
Submitted: 2026-04-26
Updated: 2026-10-05
Code: https://github.com/KellerJordan/modded-nanogpt
Sources
- Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability
- DeepSeek-V3 Technical Report
- The Llama 3 Herd of Models
- Gemma: Open Models Based on Gemini Research and Technology
- KV-weights are all you need for skipless transformers
- Most ReLU Networks Admit Identifiable Parameters
- Approximating Continuous Functions by ReLU Nets of Minimal Width
- Cutting the Skip: Training Residual-Free Transformers
- Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
- GLU Variants Improve Transformer
- LLaMA: Open and Efficient Foundation Language Models
- mHC: Manifold-Constrained Hyper-Connections
- Hyper-Connections
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks