Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

The gist The central finding is that what matters most in large language models is the relative rank of weights rather than their exact numerical values, and this rank-based perspective offers a new lens on model compression and robustnessPage 1

Weight Clustering for Compression

The paper introduces weight clustering as a method to reduce the number of unique weight values by replacing every weight matrix with K shared values from K-meansPage 1 For Llama 3.1-8B-Instruct and SmolLM2-135M, reducing each matrix to only 16–64 distinct values preserves strong accuracy without retraining, providing a simple, training-free method to compress LLMs on diskPage 1 Optionally fine-tuning only the cluster means (centroids) recovers 30–40% of the remaining accuracy gap at minimal costPage 1 The method involves assigning all entries of a weight matrix W to K clusters and replacing each value with its cluster representative, resulting in a (clustered) matrix Wˆ given by Wˆ [d, o] = cL[d,o], where ck is the centroid (mean) of the kth clusterPage 3 This process allows for significant storage reduction; for example, with K = 32, a 58.7M-weight MLP projection is represented by 32 shared values—a reduction of nearly six orders of magnitudePage 3

The Importance of Rank Preservation

The study systematically modifies the cluster means while keeping the cluster assignments fixed to investigate what preserves accuracyPage 4 The results show that the exact numerical values of the cluster means are not what preserve accuracy; instead, What matters most is their relative rank across all layer types testedPage 4 Rank-preserving changes consistently cause far less damage than rank-breaking onesPage 4 For a mid-network layer, all three models are largely indifferent to the exact centroid values as long as the corresponding cluster ranks are preservedPage 4 The authors conclude that the global rank order matters more than sign structure because the model maintains the ranks of connection strengths — not merely their sign — and this is what must be preserved for the model to function correctlyPage 4

Safe Transformations and Scale Drift

The paper identifies a class of “safe” centroid transformations as affine maps f(w) = aw + b with a > 0, which preserve both rank order and the mean (µ) and variance (σ2) of the reconstructed weight distributionPage 5 These affine maps are particularly effective because for any linear layer W inside the network, an affine weight transform w → aw + b produces an output whose ranking across neurons is unchanged, and the resulting scale and offset are approximately absorbed by layer normalizationPage 5 Furthermore, when many layers are perturbed simultaneously, progressive layer-by-layer replacement reveals that scale drift—not rank distortion—is the dominant collapse mechanism in standard randomizationPage 6 However, an affine correction w′ = aw + b with a > 0 can substantially delay this driftPage 5 The paper demonstrates that scale drift is the dominant mechanism of cumulative failure when many layers are perturbed, and affine correction can substantially delay this collapsePage 6

Layer Depth Sensitivity

The dependence of randomization on layer depth shows that rank-breaking at early layers is catastrophic across all tested modelsPage 4 For the early layer (layer 1), even rank-preserving randomization gives a considerable hit to perplexity, and the sign-scramble pushes PPL to 117.39 on Llama-InstructPage 4 The results confirm that rank-breaking is the main failure mode at every depth; rank-preserving replacements can still hurt via cumulative scale drift, and the penalty grows as the randomized layer sits earlier in the networkPage 6

Relation to Existing Methods

Weight clustering provides a cleaner framework for this analysis because it stores cluster labels L and centroid values c as explicit separate objects, allowing them to be perturbed independentlyPage 4 This separation allows the researchers to directly test how changes in relative rank affect performance, unlike post-training quantization (PTQ) methods which do not readily admit this separationPage 4 The paper notes that codebook-based methods, such as AQLM [7], VPTQ [8], and SqueezeLLM [9], can achieve higher compression but typically require specialised kernels, whereas the proposed approach is applied post-training and requires no custom inference kernelsPage 4 The authors also show that weight clustering provides a clean framework for this analysis because cluster assignments and their representative values are stored separatelyPage 4

Practical Implications

As a practical compression method, weight clustering applied to the original Llama 3.1-8B-Instruct achieves PPL 9.32 at 6.9 GB disk without any fine-tuningPage 8 This is comparable in accuracy to INT4 quantization, and optional centroid-only fine-tuning recovers 30–40% of the accuracy gap at minimal costPage 5 The paper concludes that the two degradation mechanisms—scale drift and rank distortion—differ in their correctability, with scale drift being correctable by affine correction while rank distortion is notPage 6 Future work includes formalising the conjecture that centroid rank acts as an approximate sufficient statistic for model performancePage 8 The paper also confirms that the model maintains the ranks of connection strengths — not merely their sign — and this is what must be preserved for the model to function correctlyPage 4 The final practical recommendation is to use the rebuild-to-dense path, which is reliable, immediately deTABLE IXPage 8 The paper also shows that Clustered models use rebuild-to-dense execution, so GPU RAM matches the dense baselinePage 8 The compression and clustering approach achieves a storage reduction of 4.7× when combined with pre-compression and INT4 quantizationPage 8 The optional centroid-only fine-tuning recovers an additional 30–40% of the accuracy gap at minimal costPage 5

How it works

The method involves applying scalar K-means clustering to a weight matrix W ∈ R D×O to assign all the weights in W to K clusters of distinct values, resulting in a (clustered) matrix Wˆ given by Wˆ [d, o] = cL[d,o], where ck is the centroid (mean) of the kth clusterPage 3 The clustered weights can be stored compactly on disk: Cluster labels are stored as packed ⌈log2 K⌉-bit integers, while the typically short centroid vector adds negligible overheadPage 3 For instance, with K = 32, a 58.7M-weight MLP projection is represented by 32 shared values—a reduction of nearly six orders of magnitude in distinct valuesPage 3

Experimental Setup and Findings

Experiments were performed on three different models: (1) Llama 3.

1. Robust weight perturbation analysis for compression schemes can be significantly improved by focusing on rank preservation rather than exact magnitude preservation, as what matters most is the relative rank of weights—whether one connection is stronger or weaker than another—rather than precise magnitudes. This allows for effective model compression, as rank-preserving randomizations cause almost no loss at mid and late layers, which ensures performance stability during weight clustering.

2. Development of a systematic mechanism to correct scale drift during layer-wise perturbation can enhance model robustness; the paper suggests that "an affine correction w′ = aw + b with a > 0 (which preserves both rank order and overall weight distribution) can substantially delay this drift. This correction is vital because scale drift—not rank distortion—is the dominant collapse mechanism" when many layers are perturbed simultaneously.

3. Implementation of a conditional fine-tuning strategy can maximize compression gains; optional fine-tuning only the cluster means (centroids) recovers 30–40% of the remaining accuracy gap at minimal cost. This allows for highly compressed models to be optimized efficiently by only training a small subset of parameters, reducing the computational cost associated with full model retraining.

4. Design of compression methods that explicitly maintain both mean and variance during centroid transformation can yield more reliable results; a broad class of 'safe' transformations of the cluster means, namely, maps f: ck 7→ f(ck) are precisely those that preserve both ordering and the mean (µ) and variance (σ2) of the reconstructed weight distribution. This ensures that the overall weight distribution must also be preserved to avoid catastrophic performance degradation.

5. Creation of a layer-depth dependent sensitivity map can optimize compression for specific model architectures; Table III shows that rank-breaking conditions are consistently more damaging than order-preserving ones at every layer, allowing researchers to strategically choose which layers to prioritize for rank preservation based on their known sensitivity.