Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage
Averaged evaluation masks crucial trade-offs in calibration for LLM pruning. The study found that calibration perplexity correlates positively with General knowledge but negatively with Math/Code retention, showing no single source is universally strong across all capabilities. The solution is capability-balanced multi-source calibration and Information-Guided Self-Calibration (IGSP), which dynamically constructs optimal, diverse training data pools to maximize capability preservation during high-sparsity pruning.