SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

The gist SERQ introduces a saliency-aware error reconstruction method that enables end-to-end 4-bit quantization of both weights and activations using a single low rank matrix. This method is significant because it outperforms prior matrix decomposition and distribution flattening methods, including state-of-the-art LoRA-based and rotation-based approaches, in terms of accuracy.

How it works

SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. This design avoids the overhead of an additional sequential low-rank branch during inference, while effectively mitigating quantization errors arising from both activation outliers and salient weights.

Saliency-Aware Low-Rank Adaptation

Prior error-reconstruction methods typically approximate the full-weight quantization error E = W − Q(W) using a truncated SVD of the weight matrix, E ≈ UrΣrV T. Building on the theoretical insight that quantization errors vary across weight rows in linear operations, AWQ (Lin et al. (2024b)) demonstrates that protecting only a small fraction (∼1%) of salient channels (i.e., their corresponding weight rows) using the activation distribution can substantially reduce the error. This method requires only a narrowed matrix from these salient rows, equal to the rank size, ensuring high-fidelity approximation while enabling a single matrix decomposition for the low-rank branch.

Static Activation Flattening

Activation quantization is notoriously fragile due to channel-wise outliers. Instead, since our method provides a residual path for error reconstruction at the linear layer, we avoid the online flattening process and revisit SmoothQuant (Xiao et al. (2024)), which flattens activation distributions using static per-channel scaling. Specifically, activations are scaled by a factor s, and the corresponding scale is folded into the weights. The scaling factors are obtained during calibration and merged into adjacent layers offline, incurring no runtime overhead.

Saliency-Aware Error Reconstruction

The per-channel static flattening process pushes the scale of activation outliers into the corresponding weight rows. Assuming that the original weights follow a normal distribution, the salient rows in the folded weights can be identified directly by their scales. To mitigate this, we introduce a low-rank compensator matrix R ∈ Rr×d that corrects the quantization errors in the r salient weight rows, denoted Wfs. The overall linear operation can be described as Y = (X · diag(s-1) · P-1)(P · diag(s) · W). Importantly, we quantize the low-rank matrix as well, enabling pure low-precision computation across the entire path.

Offline Weight Permutation

We have aforementioned that the weights and activations must be properly reordered based on their saliency (e.g., Xc = [Xfs Xfr], Wc = [Wfs;Wgr]). To address this, we propose a mergeable weight permutation scheme that eliminates latency overhead during inference. Both the rows and columns of the weight matrix are permuted offline according to the saliency order, enabling matrix multiplication to be executed directly on the appropriately reordered weights and activations.

Comparison with State-of-the-Art

SERQ consistently outperforms prior matrix decomposition and distribution flattening methods, including state-of-the-art LoRA-based and rotation-based approaches, in terms of accuracy. In Table 9, we present performance benchmarks alongside calibration costs for leading rotation-based techniques, including SpinQuant(Liu et al. (2025)), FlatQuant(Sun et al. (2025)), DuQuant(Lin et al. (2024a)), OSTQuant(Hu et al. (2025)), and QuaRot(Ashkboos et al., 16. The results indicate that methods employing extensive training or heavy iterative optimization generally yield the highest accuracy.

GPU Performance Analysis Details

We report the absolute latency obtained with NVIDIA Blackwell CUTLASS kernels when operating on linear layers of different sizes used in LLM models. SERQ is particularly effective for larger row-sized matrices. As shown in the left panel of Figure 3, SERQ outperforms both W4A4 and W4A8 settings of L2QER, while maintaining competitive perplexity. Compared directly with L2QER under W4A4, our single matrix error reconstruction path reduces latency overhead by up to 4.5× compared to the LoRA path, which requires two sequential low rank multiplications.

Overall LLM Accuracy Results

Table 11 reports accuracy results of low-rank matrix decomposition methods when tested with W4A8 settings. SERQ consistently outperforms LLM.int4 and L2QER across most tasks, while achieving the lowest effective bit-width. The accuracy gap is more pronounced under the W4A4 setting. SERQ maintains high accuracy across all tasks.

Conclusion

In this work, we introduced SERQ, a saliency-aware error reconstruction method that enables endto-end 4-bit quantization of both weights and activations using a single low rank matrix. By combining static activation flattening with mergeable weight permutation, SERQ identifies salient rows in the weight matrix without incurring additional latency overhead and reconstructs them through an auxiliary low rank branch. SERQ consistently outperforms prior matrix decomposition and distribution flattening methods, including state-of-the-art LoRA-based and rotation-based approaches, in terms of accuracy. Our implementation with the MXFP4 data format on NVIDIA Blackwell architecture further demonstrates that the auxiliary branch introduces minimal latency overhead, achieving significant speedups while preserving accuracy in low precision computation.

Ablation Study on Static Activation Flattening

In SERQ, static activation flattening (SAF) is combined with saliency aware low rank error reconstruction, effectively stabilizing the activation distribution during quantization. For the LLaMA-2-7B model, SERQ without SAF maintains accuracy well when only low error reconstruction is applied. However, for more error-sensitive small-scale models such as Qwen-2.5-3B, SAF provides a substantial accuracy improvement. This behavior arises because smaller models accumulate fewer partial sums per output element, making quantization errors from a single group disproportionately influential on the final output.

GPU Performance Comparison

SERQ is particularly effective for larger row-sized matrices. Compared to L2QER under W4A4, our single matrix error reconstruction path reduces latency overhead by up to 4.5× compared to the LoRA path, which requires two sequential low rank multiplications.

Overall LLM Accuracy Results

Table 13 shows the accuracy results of distribution flattening methods when tested with W4A4 settings. SERQ-MXFP4 achieves the highest perplexity score among all listed methods for LLaMA-2-7B in this table. SERQ consistently achieves superior performance with only minor accuracy degradation in the W4A8 setting.

1. No additional online layers are required during inference for low-rank error reconstruction, which is achieved by jointly addressing activation and weight saliency through a single low-rank decomposition. This allows SERQ to achieve a fully 4-bit end-to-end computation path without the latency overhead associated with sequential low-rank branches.

2. The system can perform high-precision matrix multiplication in linear layers using a single decomposition, as demonstrated by the formulation: Y = (X · diag(s−1) · P−1)(P · diag(s) · W) ≈ Xcq · Wdq + X[s,q] · Q(R). This enables pure low-precision computation across the entire path.

3. SERQ can be implemented with minimal overhead by leveraging offline operations: all other operations are performed offline, specifically through static activation flattening and offline weight permutation, which allows for the elimination of on-the-fly reordering latency during inference.

4. The system achieves superior accuracy in the challenging W4A4 setting compared to state-of-the-art rotation methods, as SERQ's performance is shown to be fully comparable to computationally intensive, training-based methods while offering a significant reduction in calibration complexity.

5. The model can be deployed efficiently on hardware like NVIDIA Blackwell GPUs by utilizing the MXFP4 data format kernels supported by CUTLASS, which enables speedups and reduced peak memory usage compared to FP16 baselines.

Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quantization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces a lightweight auxiliary computation path without requiring heavy optimization or additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conventional low-rank adaptations rely on two sequential factors, necessitating intermediate quantization during inference and thereby limiting low-precision efficiency. In this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically, SERQ outperforms prior error reconstruction methods under both W4A8 and W4A4 settings, and achieves higher accuracy than state-of-the-art rotation-based W4A4 approaches, while substantially reducing calibration complexity. Code is available at https://github.com/acalabys/SERQ