Compression Technique Interactions in Small Language Models: A Factorial Ablation Study
cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: 13 pages, 3 figures, 11 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- A Systematic Study of Compression Ordering for Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- Measuring Massive Multitask Language Understanding
- Distilling the Knowledge in a Neural Network
- LoRA: Low-Rank Adaptation of Large Language Models
- SqueezeLLM: Dense-and-Sparse Quantization
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- Pointer Sentinel Mixture Models
- Compact Language Models via Pruning and Knowledge Distillation
- KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- A Simple and Effective Pruning Approach for Large Language Models
- UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents