PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model 3
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model 3".
Jane: Here's the gist of the paper. Segment Anything Model 3 can segment objects from text prompts and track them across video, but its memory and computation costs make deployment difficult <ref:2611.06143#pg6>.
Tom: Semantics-Preserving Quantization (SPQ) combines three calibration steps to preserve prompt grounding and temporal memory while keeping the inference graph unchanged <ref:2611.06143#pg4>.
Jane: Controlled tests show that SPQ improves RefCOCO Pr@0.5 from 42.1 to 72.5 over PTQ4SAM while retaining 54.8 COCO mask AP <ref:2611.06143#pg8>.
Tom: Prompt Alignment Drift Identification The paper identifies prompt-alignment drift as an error where a quantized SAM 3 can select the wrong object or lose track of its target across frames <ref:2611.06143#pg3>.
Title and authors: Tom: So let’s talk about who’s behind this, and what the title actually suggests. The paper is PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model three and you see a lot of names listed there from Alabama at Birmingham to Oak Ridge National Laboratory <ref:2611.06143#pg1,PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model 3>.
Jane: It’s interesting how many different institutions are contributing to this work, suggesting it’s a collaborative effort across various research groups trying to solve this multimodal efficiency problem.
Lu: The title itself tells you the strategy: they are focusing on semantics preservation while doing post-training quantization for SAM three which is the specific model architecture they are working with <ref:2611.06143#pg1>.
Meng: I wonder what that means in practice, because traditionally, PTQ just shrinks the model and hoping things still work well enough. This paper seems to suggest a much more involved calibration process than just standard quantization.
Lalam: It suggests they’re not just throwing random bits at it; they’re trying to keep the meaning—the semantics—intact during that compression process, which is a big step for making these models reliable in real applications.
The paper's summary: Tom: So the summary says that SAM three can segment objects from text prompts and even track them across video, but because of its memory needs and computation costs, deploying it is difficult, which is the problem they’re trying to solve <ref:2611.06143#pg1,SAM 3 can segment objects from text prompts and>.
Jane: And the proposed solution, Semantics-Preserving Quantization or SPQ, combines three different calibration steps to make sure that when you quantize it for efficiency, you don't lose that prompt grounding or that temporal memory.
Lu: Specifically, this involves a Semantic-Anchor Calibration which uses a fixed concept codebook to give more weight to parameters affecting the text concepts scores.
Meng: That sounds like they are essentially teaching the model how to map its features back to those specific text concepts during quantization, rather than letting the quantization process scramble that mapping.
Lalam: And then they have Orthogonal Subspace Rectification, which is designed to preserve features along those residual directions by penalizing shifts at the feature level. That sounds like they are isolating and protecting the parts of the model that are most sensitive to small changes in precision.
The paper's improvements: Tom: The results are pretty compelling, because they show significant gains when you compare SPQ against standard post-training quantization methods like PTQ4SAM. They found that SPQ improves the RefCOCO Pr@zero point five score from forty-two point one up to seventy-two point five over PTQ4SAM, while still keeping a COCO mask AP of fifty-four point eight, which is a noticeable jump in accuracy for segmentation tasks under the W4A8 image encoder configuration.
Jane: That jump from forty-two point one to seventy-two point five on that specific metric shows that this method is much more effective at maintaining prompt grounding than what’s available now with standard quantization techniques.
Lu: The paper explains why this happens by looking at the subspace perturbation view, showing that matched perturbations in residual directions reduce grounding errors more than those in the dominant directions, which is a key insight into how to calibrate effectively.
Meng: I’m interested in that idea about residual directions; it implies they don't have to perfectly preserve every single feature across all dimensions of the model, but just the ones that matter for tracking and segmentation accuracy.
Lalam: And they also address temporal memory by including a Temporal Distribution Alignment step, which estimates and corrects the mean error in video memory by fusing its correction into an existing bias, which helps mitigate identity drift across frames.
Conclusion: Tom: So to wrap up, SPQ achieves better results across image segmentation, referring segmentation, and video tracking compared to standard PTQ baselines while keeping the inference graph exactly the same as standard PTQ does. This means you get performance gains without sacrificing the original model structure.
Jane: It’s a neat trick because it keeps your deployment footprint similar to what you're used to with standard post-training quantization, but it gets you much better performance on tasks that rely on tracking or precise object identification.
Lu: The main implication is that grounding is especially sensitive to errors in low-variance residual directions, and SPQ directly targets those directions with their orthogonal subspace rectification component.
Meng: For deployment at scale, the efficiency here is good because it matches the bit-width of standard PTQ and even shows up to three point five times compression on an RTX three thousand ninety for video frames. That’s concrete hardware data we can work with.
Lalam: And they also showed that this calibration strategy extends beyond SAM three to CLIP, recovering substantially more zero-shot image-text retrieval accuracy when applied to those models, which suggests the core idea is broadly applicable across different vision-language architectures <ref:2611.06143#pg1>.
Tom: So we're looking at PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model three which shows how careful calibration can significantly boost performance in video tasks while maintaining the same computational cost structure as standard quantization techniques <ref:2611.06143#pg1,PTQ4SAM3: Semantics-Preserving Post-Training Quantization for Segment Anything Model 3>.
Xi Xiao, Yunbei Zhang, Lin Zhao, Janet Wang, Xinru Jiang, Yanshu Li, Chenrui Ma, Pan Wang, Tianyang Wang
University of Alabama at Birmingham, USA · Tulane University, USA · Northeastern University, USA · University of Texas at Austin, USA · University of Virginia, USA · Oak Ridge National Laboratory, USA
cs.CV
Submitted: 2026-11-04
Updated: 2026-11-04
Comments: 15 pages, 4 figures. Accepted to WACV 2027. Includes supplementary material
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Segment Anything Model 3 can segment objects from text prompts and track them across video, but its memory and computation costs make deployment difficult <ref:2611.06143#pg6>.
Key concepts
- Prompt Alignment Drift
- This is an error where a quantized model, like SAM 3, selects the wrong object or loses track of its target across video frames. It happens even when segmentation scores look good because the model struggles to maintain consistent grounding due to small changes in feature directions.
- Semantic-Anchor Calibration
- This step calibrates the model by giving more importance to parameters that influence text-concept scores. It uses a fixed codebook, which includes synonyms and templates, allowing it to correctly identify concepts even if the exact training examples are missing.
- Orthogonal Subspace Rectification
- This technique preserves important features along residual directions by penalizing shifts at the feature level. It focuses on fixing errors in low-variance directions rather than dominant ones, which is key to stabilizing object tracking and grounding.
- Temporal Distribution Alignment
- This component estimates and corrects the average error in video memory. It fuses this correction into an existing bias using a closed-form method, ensuring that the model maintains accurate memory across consecutive frames without needing complex retraining.
Terminology
Summary
Segment Anything Model 3 can segment objects from text prompts and track them across video, but its memory and computation costs make deployment difficult <ref:2611. p. 006143#pg6>. Semantics-Preserving Quantization (SPQ) combines three calibration steps to preserve prompt grounding and temporal memory while keeping the inference graph unchanged <ref:2611. p. 106143#pg4>. Controlled tests show that SPQ improves RefCOCO Pr@0. p. 05 from 42.1 to 72.5 over PTQ4SAM while retaining 54. p. 08 COCO mask AP <ref:2611. p. 806143#pg8>.
Prompt Alignment Drift Identification
The paper identifies prompt-alignment drift as an error where a quantized SAM 3 can select the wrong object or lose track of its target across frames <ref:2611.06143#pg3>. This drift occurs even when feature reconstruction and standard segmentation scores remain strong, suggesting that grounding is especially sensitive to perturbations in low-variance residual directions <ref:2611. p. 106143#pg6>. The study uses a subspace perturbation view to organize the analysis, finding that matched perturbations in residual directions reduce grounding more than those in dominant directions <ref:2611. p. 106143#pg4>.
SPQ Calibration Components
Semantics-Preserving Quantization (SPQ) consists of three components:
-
Semantic-Anchor Calibration, which gives more weight to parameters that affect text-concept scores by using a fixed concept codebook <ref:2611. p. 106143#pg4>.
-
Orthogonal Subspace Rectification, which preserves features along the residual directions by penalizing shifts at the feature level <ref:2611. p. 106143#pg6>.
-
Temporal Distribution Alignment, which estimates and corrects the mean error in video memory by fusing its correction into an existing bias <ref:2611. p. 106143#pg6>.
Performance Gains Across Tasks
SPQ achieves superior results across image segmentation, referring segmentation, and video tracking compared to standard PTQ baselines <ref:2611.06143#pg6>. Specifically, with a W4A8 image encoder, SPQ reaches 54. p. 18 COCO mask AP and 72. p. 05 RefCOCO Pr@0. p. 05 <ref:2611. p. 006143#pg8>. The full model combines these gains and reaches 54. p. 48/72.5/81.2 in the factorial comparison <ref:2611. p. 906143#pg9>.
Mechanism and Scope
The analysis demonstrates that residual-basis changes affect grounding most strongly, while dominant-basis changes affect AP most strongly <ref:2611.06143#pg6>. The codebook calibration transfers to concepts beyond exact entries, showing that vocabulary coverage affects grounding accuracy <ref:2611. p. 906143#pg10>. SPQ recovers substantially more zero-shot image–text retrieval accuracy on both datasets when applied to CLIP <ref:2611. p. 406143#pg8>.
Efficiency and Deployment
SPQ matches the hardware footprint of standard PTQ at the same bit-width, preserving the inference cost of standard PTQ <ref:2611.06143#pg9>. On an RTX 3090 with 1024 × 1024 video frames, SPQ achieves up to 3. p. 75× compression and nearly 2.3× speedup over FP16 <ref:2611. p. 006143#pg9>. The calibration cost is paid once per model and deployment configuration, as the fused parameters preserve the inference graph <ref:2611. p. 706143#pg9>.
Conclusion
SPQ combines concept-sensitive calibration, residual-basis feature matching, and a correction for video memory <ref:2611.06143#pg6>. Controlled interventions show that grounding is especially sensitive to errors in low-variance residual directions <ref:2611. p. 706143#pg6>. Experiments show gains in image segmentation, referring segmentation, and video tracking with the inference graph unchanged <ref:2611. p. 706143#pg9>. Future work will extend SPQ to more architectures and lower activation bit widths <ref:2611. p. 706143#pg10>. The authors verified all text, codebook phrases, figure labels, citations, and analysis and take full responsibility for the manuscript <ref:2611.06143#pg10>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Semantic-Anchor Calibration allows for calibration without task annotations by using a fixed concept codebook generated offline from calibration images via Qwen2.5-VL-72B-Instruct, which includes synonyms and fixed templates to cover concepts beyond exact training examples.
-
Orthogonal Subspace Rectification preserves features along the residual directions by penalizing shifts at the feature level using a loss term that constrains feature error along selected residual coordinates, defined by the complement of the top k principal directions.
-
Temporal Distribution Alignment corrects mean error in video memory by estimating and fusing it into an existing bias using a closed-form correction, which preserves deployed operators and FLOPs while mitigating identity drift across frames.
-
The improved system can maintain prompt grounding and temporal memory while keeping the inference graph unchanged, leading to gains such as achieving 72.5 RefCOCO Pr@0.5 under W4A8 compared to 42.1 for PTQ4SAM on the same encoder architecture with a W4A8 image encoder.
-
The system can recover substantially more zero-shot image–text retrieval accuracy when applying Designs A and B to a separate vision-language model like CLIP, extending the calibration strategy beyond SAM 3 to broader architectures while maintaining high performance.
Sources
- SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
- Qwen2.5-VL Technical Report
- SAM 3: Segment Anything with Concepts
- BoA: Attention-aware Post-training Quantization without Backpropagation
- A White Paper on Neural Network Quantization
- SAM 2: Segment Anything in Images and Videos
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
- ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs