DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction
cs.CL
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/jianke0604/DepthGraft
License: http://creativecommons.org/licenses/by/4.0/
The gist: Scaling depth is a key route to stronger large language models, but Pre-LN architectures often suffer from depth-wise information dilution, limiting the effective use of deep layers.
Terminology
Abstract
Scaling depth is a key route to stronger large language models, but Pre-LN architectures often suffer from depth-wise information dilution, limiting the effective use of deep layers. While cross-layer connectivity has proven effective at mitigating this problem, existing designs usually introduce additional computation or memory access. We revisit cross-layer KV sharing as an efficient form of such connectivity. Beyond cache compression, each KV reuse edge creates a direct gradient path from deep layers to source KV projections, turning KV sharing into an implicit structural regularizer for KV representation geometry. We provide a systematic analysis of this mechanism, showing that proper sharing topologies sharpen key retrieval directions while enriching the value content subspace, whereas existing adjacent or global-anchor designs do not fully exploit this effect. Motivated by this, we propose DepthGraft, a hierarchical KV-sharing architecture that reconstructs deep-layer KV from stride-aligned shallow-middle source pairs. DepthGraft distributes non-local gradient flow across source layers, while preserving prefill early completion and reducing KV cache by roughly one third. Experiments on dense and MoE models from 0.5B to 16B parameters show consistent downstream improvements under both GQA and MLA. Notably, on the 16B MoE model, DepthGraft achieves a 7.0% absolute improvement on MMLU-Pro, along with gains of 6.6% on CMMLU and 6.4% on C3. These results validate the potential of DepthGraft as a scalable design for large-scale pretraining. The project page is available at https://github.com/jianke0604/DepthGraft.
Sources
- Bridging Linguistic Structure and Mechanistic Interpretability for Conceptual Interpretation in Language Models
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Do Language Models Use Their Depth Efficiently?
- The Unreasonable Ineffectiveness of the Deeper Layers
- Measuring Massive Multitask Language Understanding
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
- Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Multi-Stream Attention Residuals
- How Instruction-Tuning Imparts Length Control: A Cross-Lingual Mechanistic Analysis
- The Curse of Depth in Large Language Models
- Kimi K2: Open Agentic Intelligence
- Attention Residuals
- HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing
- Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
- Efficient Streaming Language Models with Attention Sinks
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data