SegRAG: Retrieval Augmented Spatial Prompting for Open Vocabulary Semantic Segmentation
cs.CV
Submitted: 2026-05-17
Updated: 2026-09-30
Journal ref: Information Processing & Management, Volume 64, Issue 2, Part B, February 2027, 105190
DOI: 10.1016/j.ipm.2026.105190
Code: https://github.com/boudiafA/SegRAG
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Frozen segmentation foundation models often fail when the target class appears in a form that is weakly represented during pretraining.
Terminology
Abstract
Frozen segmentation foundation models often fail when the target class appears in a form that is weakly represented during pretraining. To address this problem, we introduce SegRAG, a retrieval-augmented inference-time spatial prompting pipeline for open-vocabulary semantic segmentation that uses frozen models without updating their weights. SegRAG builds a compact class-indexed memory from annotated references. When multiple references are available, Intra-Class Cohesion Distillation (ICCD) filters DINOv3 patch descriptors by cross-image foreground agreement. With one reference, foreground descriptors are retained directly without ICCD. Topographic Similarity Grounding (TSG) then turns high-similarity query regions into point prompts for SAM 3. In the matched five-shot comparison, SegRAG exceeds recent exemplar- and retrieval-based baselines on ADE20K-150, Cityscapes, and PC-59. It also improves over the SAM 3 text-only baseline by 1.15 to 3.92 mean Intersection over Union (mIoU) points. In the up-to-30-shot AgML agricultural domain-transfer evaluation, SegRAG raises mIoU from 25.27 to 59.24. It also recovers text-only failures, including cauliflower from 0.00 to 95.36 IoU and sugarbeet weed from 0.00 to 80.22 IoU. Controlled ablations show complementary contributions from ICCD, TSG point selection, and joint text-and-point prompting. SegRAG therefore formulates segmentation adaptation as an information organization and retrieval problem by maintaining annotated visual knowledge as an external, class-indexed memory that can guide a frozen segmentation model without weight updates. Code: https://github.com/boudiafA/SegRAG.
Sources
- SAM 2: Segment Anything in Images and Videos
- SAM 3: Segment Anything with Concepts
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- DINOv3
- Fully Convolutional Networks for Semantic Segmentation
- U-Net: Convolutional Networks for Biomedical Image Segmentation
- Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
- Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
- Panoptic Segmentation
- Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
- Segmenter: Transformer for Semantic Segmentation
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- Per-Pixel Classification is Not All You Need for Semantic Segmentation
- Emerging Properties in Self-Supervised Vision Transformers
- DINOv2: Learning Robust Visual Features without Supervision
- Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
- Revealing the Semantic Selection Gap in DINOv3 through Training-Free Few-Shot Segmentation
- Learning Transferable Visual Models From Natural Language Supervision
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models