What Breaks and What Comes Back: Capability Damage and Recovery in Depth-Pruned Small Language Models
cs.CL, cs.LG
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/piyushdhoka/depth-pruned-slm-recovery
Terminology
Sources
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- The Unreasonable Ineffectiveness of the Deeper Layers
- Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
- PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery
- DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
- Reassessing Layer Pruning in LLMs: New Insights and Methods
- A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
- On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
- What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity
- Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2
- The Llama 3 Herd of Models
- Qwen2.5 Technical Report
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- 2 OLMo 2 Furious
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
- Instruction-Following Evaluation for Large Language Models
- Training Verifiers to Solve Math Word Problems
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data