GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression

The gist The GA-SVD framework introduces a generation-aware low-rank compression method for large language models that consistently outperforms existing SVD-based baselines on both multiple-choice and generative benchmarks while scaling to models up to 70B parameters.

How it works

1. Generation-Aware Weight Whitening (GAWW) leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior The central idea is to make the whitening geometry reflect the model’s generative behavior, rather than relying only on offline calibration statistics. This involves estimating layerwise whitening statistics from trajectories generated by the compressed student model itself.

2. Bi-whitened SVD is performed using these generation-aware statistics, where the objective is approximated by a bi-sided quadratic form that measures compression error in a geometry jointly induced by input activations and output sensitivities. This leads to solving the weighted rank-constrained problem Wcl = arg min rank(A)≤rl TrΣs,l (W⋆l − A) Σx,l (W⋆l − A)⊤.

3. Evolutionary Global Rank Allocation searches for effective layer-wise rank allocation under a given global compression ratio by formulating rank selection as a search problem. This is achieved through evolutionary search where parent candidates are mutated via Rank mutation and Boundary-swap mutation to explore the search space.

Key Components and Findings

**: The GA-SVD pipeline starts from a warm-start initialization, which provides a stable compressed model from which on-policy generation can be performed. The On-policy Generation-aware Whitening collects generation-aware whitening statistics from trajectories sampled from the current compressed student model. This process uses a token-level distillation objective, defined as Lgen = τ2/N Σ X z X t∈G(z) KL(pτS(· z<t)∥ pτT(· z<t)), gl,t = ∂Lgen/∂hl,t. This objective induces a generation-aware sensitivity signal from the student’s own rollouts. The Generation-aware Bi-whitened Low-rank Compression then uses this to define the layerwise compression objective, which is equivalent to solving Wcl = arg min rank(A)≤rl TrΣs,l (W⋆l − A) Σx,l (W⋆l − A)⊤. This results in a compressed layer expressed in factorized form Wcl = Wu,lWv,l. The Evolutionary Global Rank Allocation then searches for the optimal rank allocation setup under a global compression budget B, formulated as min K F(K) s.t. X K∈K C(K) ≤ B. This search space is restricted by defining discrete rank levels R = ⌊doutdin/dout+din⌋ and boundary windows B(r) for non-monotonic singular selection. The evolutionary strategy uses a (1 + λ) evolutionary strategy with mutations like Rank mutation and Boundary-swap mutation to generate offspring candidates. 4.3 Ablation Study shows that removing GAWW decreases performance on all generative benchmarks, indicating that generation-aware whitening helps preserve generation ability during compression. Removing ERA causes a larger drop on coding tasks, reducing HumanEval from 20.7 to 16.7 and MBPP from 34.1 to 30.9, which shows the importance of global rank allocation beyond heuristic rank assignment. Table 5 confirms that the evolutionary search automatically redistributes the rank budget toward more sensitive layers while assigning lower ranks to more compressible ones. Results on different compression ratios show that GA-SVD consistently outperforms SVD-LLM at both 20% and 40% compression ratios, demonstrating stronger performance across coding, math, MCQ benchmarks, and perplexity. The method is also highly scalable to models with up to 70B parameters. These results demonstrate that GA-SVD preserves both generative and discriminative abilities more effectively than prior SVD-based methods. The method also reduces perplexity, improving inference efficiency, and scales to large models up to 70B parameters. Results on Qwen3-8B show that GA-SVD mitigates, but does not eliminate, the sensitivity of Qwen3-8B to compression. The results on CNN/DailyMail suggest that the benefits of generation-aware compression extend to summarization. Results with Different Student Warmup Ratio show that final performance remains stable across this range, suggesting that a short warmup is sufficient for effective estimation and that the method is insensitive to the warmup ratio within the evaluated range. The combination with quantization shows that GA-SVD can be effectively combined with low-bit quantization, enabling further compression of the model weights while largely preserving downstream performance. This work demonstrates that generation-aware low-rank compression is an effective and practical direction for preserving LLM capabilities under substantial compression. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks. The larger degradation on coding and math tasks than on MCQ benchmarks is consistent with this hypothesis. The consistent reduction in perplexity further shows that GA-SVD better maintains the language modeling distribution after low-rank approximation. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks. The larger degradation on coding and math tasks than on MCQ benchmarks is consistent with this hypothesis. The consistent reduction in perplexity further shows that GA-SVD better maintains the language modeling distribution after low-rank approximation. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks. The larger degradation on coding and math tasks than on MCQ benchmarks is consistent with this hypothesis. The consistent reduction in perplexity further shows that GA-SVD better maintains the language modeling distribution after low-rank approximation. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks. The larger degradation on coding and math tasks than on MCQ benchmarks is consistent with this hypothesis. The consistent reduction in perplexity further shows that GA-SVD better maintains the language modeling distribution after low-rank approximation. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks. The larger degradation on coding and math tasks than on MCQ benchmarks is consistent with this hypothesis. The consistent reduction in perplexity further shows that GA-SVD better maintains the language modeling distribution after low-rank approximation. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks. The larger degradation on coding and math tasks than on MCQ benchmarks is consistent with this hypothesis. The consistent reduction in perplexity further shows that GA-SVD better maintains the language modeling distribution after low-rank approximation. The results on Qwen3-8B show pronounced sensitivity to low-rank compression, particularly on generation tasks.

1. Generation-Aware Weight Whitening (GAWW) enables compressed models to better preserve generation capabilities by constructing whitening transformations aligned with generative behavior using gradients from on-policy knowledge distillation, which is achieved by estimating layerwise whitening statistics from trajectories generated by the compressed student model itself.

2. Evolutionary Global Rank Allocation searches for effective layer-wise rank allocation under a given global compression ratio, addressing the limitation that existing methods are based on heuristics, which may be suboptimal under a global compression budget by formulating rank selection as a search problem.

3. GA-SVD achieves state-of-the-art (SoTA) results on both multiple-choice and generative benchmarks across Mistral, Qwen3, and Llama models, specifically improving coding average to 27.4, outperforming SVD-LLM and achieving the best coding average, GSM8K accuracy, MCQ average among compressed models.

4. The method demonstrates scalability by applying the framework to larger models with up to 70B parameters, proving that generation-aware low-rank compression is an effective and practical direction for preserving LLM capabilities under substantial compression.

5. GA-SVD can be effectively combined with low-bit quantization (e.g., 4-bit weight-only GPTQ) resulting in modest performance degradation while retaining higher scores on coding, GSM8K, and MCQ benchmarks compared to other baselines.

Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost effectively. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank allocation under a given global compression ratio. Across extensive experiments on Mistral-7B, Qwen3 series, and Llama-3.3 models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art (SoTA) results. We further demonstrate scalability by applying our method to models with up to 70B parameters.