M3Former: Fine-Grained Cross-Layer Mixing for Transformers
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/Kingchet10/m3former
Terminology
Sources
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- The Unreasonable Ineffectiveness of the Deeper Layers
- Scaling Laws for Neural Language Models
- Attention Residuals
- Multi-Head Attention Residuals
- Multi-Stream Attention Residuals
- GLU Variants Improve Transformer
- DeepNet: Scaling Transformers to 1,000 Layers
- mHC: Manifold-Constrained Hyper-Connections
- TinyLlama: An Open-Source Small Language Model
- Frac-Connections: Fractional Extension of Hyper-Connections
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning