From Sweep to Seam: Interleaved Cross-Block Post-Training Quantization

Summary

This paper introduces Interleaved Cross-Block Quantization (ICBQ), a scheduling modification to block-wise post-training quantization (PTQ) for large language models (LLMs). The method is designed to mitigate the accumulation of quantization errors that occurs when quantizing Transformer blocks sequentially.

Problem and Motivation:

The paper addresses the challenge of compressing LLMs to very low bit-widths (e.g., two bits or fewer). While block-wise PTQ is a practical approach, the authors identify a key weakness in the standard sequential cross-block quantization (CBQ) baseline: the matched sequential baseline moves this window through the network once, so errors introduced early in the sweep are not revisited. This can lead to significant error accumulation with depth, sometimes causing severe degradation or divergence (e.g., perplexity > 1e4 for Qwen3-8B).

Proposed Method (ICBQ):

ICBQ is a schedule-level modification that revisits the boundary pair between consecutive chunks. The core idea is that Each seam pair is refined twice: first at the end of one chunk and again at the start of the next. This is achieved by interleaving a short CBQ sweep with progressive quantization. The method retains the local two-block objective and reuses the calibration inputs of existing block-wise PTQ pipelines. The key structural element is the seam, which is the boundary pair between chunks. The paper states, seams are exactly where block-wise errors are handed from one chunk to the next, so revisiting them is the core mechanism that suppresses depth-wise buildup.

Theoretical Contribution:

The paper provides a theoretical analysis of the method's benefits. Under stated local contraction and smoothness assumptions, the authors derive a depth-wise upper-bound comparison in which seam revisits multiply the propagated term while the residual remains bounded independently of depth. The main theorem (Theorem 4.1) shows that the error bound for ICBQ includes a factor of γ S(L,K), where S(L,K) is the number of chunk seams, applied to the propagated term of the sequential bound. This leads to a corollary about geometric decay in the number of seams, suggesting that the benefits compound with more seams (i.e., smaller chunk sizes).

Experimental Results:

The experiments evaluate ICBQ against a Sequential CBQ baseline using two inner quantizers: ternary DBF (1.58-bit) and GPTQ (W3 and W2g128).

1. Ternary DBF Results: ICBQ reduces ternary-quantization perplexity relative to the matched Sequential CBQ baseline on all seven tested models. It also yields finite perplexity in configurations where the baseline has severe degradation, specifically for Llama-3-8B and Qwen3-8B.

2. GPTQ Results: The method can also be used with 3-bit and 2-bit GPTQ. ICBQ attains the best or tied-best perplexity on five of seven models at both W3 and W2g128 precisions.

3. Ablations: The paper isolates the effect of chunk size (K) and the prefit step. It finds that PPL improves monotonically or near-monotonically as K decreases, which aligns with the theory that smaller chunks create more seams. The results also show that Interleaving remains better than Sequential CBQ in the reported no-prefit runs, indicating that the schedule effect is not limited to prefit-enabled quantization.

Conclusion:

The paper concludes that schedule design is a useful additional consideration for low-bit weight-only PTQ. ICBQ offers a way to improve the quality of quantized models without changing the inner objective or requiring new calibration data, only by changing when the refinement windows are applied.

Improvements to AI Systems:

1. Error-Compensating Quantization Scheduler: Implement ICBQ as a plug-in scheduling layer in any block-wise PTQ pipeline. The improved system can automatically revisit boundary pairs between quantization chunks, reducing depth-wise error accumulation. This enables LLMs to be quantized to 2-bit or lower without catastrophic perplexity spikes (e.g., preventing divergence in models like Qwen3-8B where sequential baselines fail).

2. Adaptive Chunk-Size Selection: Use the theoretical result (geometric decay in number of seams) to dynamically choose chunk size K based on model depth and target bit-width. The improved system can trade off computational cost (more seams = more refinement passes) against quality, selecting the smallest K that meets a perplexity threshold—yielding better compression ratios for memory-constrained deployment.

3. Robust Low-Bit Deployment for Edge Devices: Integrate ICBQ into mobile/edge inference stacks. The improved system can deploy 1.58-bit ternary or 2-bit GPTQ models on-device with finite, usable perplexity (e.g., for Llama-3-8B and Qwen3-8B) where prior methods degraded to >1e4 PPL, enabling on-device chatbots or code assistants with 4-8x memory reduction.

4. Calibration-Data-Reuse Refinement: Since ICBQ reuses existing calibration inputs, the improved system can enhance any existing PTQ tool (e.g., GPTQ, AWQ) without additional data collection. This allows rapid fine-tuning of quantized models for new domains (e.g., medical or legal text) with minimal overhead, improving task-specific accuracy post-quantization.

5. Theoretical Guarantee for Model Compression Pipelines: Leverage the depth-wise error bound (γ S(L,K) factor) to predict worst-case degradation before quantization. The improved system can pre-emptively reject quantization configurations that would violate quality constraints, saving compute and preventing silent model quality collapse in production.

Compressing large language models to two bits or fewer is increasingly feasible through block-wise post-training quantization; cross-block variants reconstruct neighboring Transformer blocks within a moving window. In the fixed two-block setting studied here, the matched sequential baseline moves this window through the network once, so errors introduced early in the sweep are not revisited. We propose Interleaved Cross-Block Quantization (ICBQ), a scheduling modification that revisits the boundary pair between consecutive chunks. Each seam pair is refined twice: first at the end of one chunk and again at the start of the next. The method retains the local two-block objective and reuses the calibration inputs of existing block-wise PTQ pipelines. Under stated local contraction and smoothness assumptions, we derive a depth-wise upper-bound comparison in which seam revisits multiply the propagated term while the residual remains bounded independently of depth. In the reported experiments, ICBQ reduces ternary-quantization perplexity relative to the matched Sequential CBQ baseline, yields finite perplexity in configurations where the baseline has severe degradation, and can also be used with 3-bit and 2-bit GPTQ.