When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale
cs.AI
Submitted: 2026-11-05
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by/4.0/
The gist: Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores.
Terminology
Abstract
Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. Such scores record only the loss that remains at the output. Inside the model, compression damages some components more than others, and the rest of the network can partly compensate. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate. How the two combine, and whether their balance changes with model size, remains unclear. We study this compensation for in-context retrieval, a capability carried by a small, identifiable set of attention heads. To separate the two, the expected loss from ablating these heads is decomposed exactly into the part restored by the rest of the network, attributed to the components responsible, and the part that reaches the output. This decomposition is then compared across model sizes. For damage of comparable size, smaller models restore most of the loss, whereas larger models retain most of it, because the compensating responses offset less of it as models grow. Compressing the heads produces the same pattern. Compression that is nearly harmless in small models can therefore be costly in large ones.
Sources
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Perplexity Can Miss SAE Feature Damage Under Quantization
- The Llama 3 Herd of Models
- Training Compute-Optimal Large Language Models
- What Do Compressed Deep Neural Networks Forget?
- Copy Suppression: Comprehensively Understanding an Attention Head
- The Hydra Effect: Emergent Self-repair in Language Model Computations
- Does quantization affect models' performance on long-context tasks?
- A White Paper on Neural Network Quantization
- The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
- Same Predictions, Different Reasons: The Effect of Quantization on Model Explanations
- Explorations of Self-Repair in Language Models
- Interpreting the Effects of Quantization on LLMs
- Massive Activations in Large Language Models
- Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- OPT: Open Pre-trained Transformer Language Models
- From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents