SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
cs.CV, cs.AI, cs.CL
Submitted: 2025-08-05
Updated: 2026-09-21
Code: https://github.com/turboderp-org/exllamav2
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-language models (VLMs) for multimodal understanding.
Terminology
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities across diverse language tasks, motivating their extension to vision-language models (VLMs) for multimodal understanding. However, billion-parameter VLMs incur substantial memory and computational costs that hinder deployment in resource-constrained settings. Post-training quantization (PTQ) compresses models and accelerates inference without retraining, yet remains underexplored for VLMs. We identify two intrinsic VLM activation properties in PTQ: (1) visual over-representation, where vision tokens are excessive and often redundant, and (2) the modality gap separating text and vision tokens in the latent feature space. Prior methods largely overlook these properties, leading to quantization performance degradation. To address this mismatch, we propose SalQ-VLM, an importance-aware PTQ framework that prioritizes salient tokens and suppresses redundant vision tokens during calibration. We derive a gradient-driven importance factor that captures token-level importance variance and is theoretically grounded in the relationship among loss perturbation, activation errors, and output gradients. SalQ-VLM obtains this factor through a single lightweight block-wise gradient-caching pass and incorporates it into the layer-wise reconstruction objective. Because SalQ-VLM modifies only calibration, it adds no inference-time operations and remains compatible with existing high-performance kernels. Extensive evaluations across benchmarks and backbones show that SalQ-VLM consistently outperforms strong PTQ baselines, especially under ultra-low-bit quantization. Notably, it improves MME-RealWorld accuracy by 16.45% under INT2g128 quantization.
Sources
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- Qwen2.5-VL Technical Report
- Sparks of Artificial General Intelligence: Early experiments with GPT-4
- DB-LLM: Accurate Dual-Binarization for Efficient LLMs
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- CBQ: Cross-Block Quantization for Large Language Models
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
- LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
- OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
- QVGen: Pushing the Limit of Quantized Video Generative Models
- HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
- Mistral 7B
- BoA: Attention-aware Post-training Quantization without Backpropagation
- Critic-Guided Decoding for Controlled Text Generation
- LLaVA-OneVision: Easy Visual Task Transfer
- SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
- BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
- GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models