Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement
cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: Accepted to NeurIPS 2026
Code: https://github.com/blue-531/pref-ovss
License: http://creativecommons.org/licenses/by/4.0/
The gist: Open-vocabulary semantic segmentation (OVSS) enables pixel-level prediction over arbitrary text-specified vocabularies and has shown strong generalization on common benchmarks.
Terminology
Abstract
Open-vocabulary semantic segmentation (OVSS) enables pixel-level prediction over arbitrary text-specified vocabularies and has shown strong generalization on common benchmarks. However, OVSS performance often degrades in specialized domains such as medical imaging, remote sensing, and industrial inspection, where dense pixel-level masks for adaptation are costly to obtain and require domain-specific expertise. We propose a preference-guided adaptation framework that replaces dense mask supervision with binary preferences. We observe that different prompt templates produce systematically different segmentations for the same image, a phenomenon we call prompt disagreement, and we repurpose it as a built-in source of preference supervision. Building on this, we mine localized preference queries from regions of high cross-template uncertainty, and adapt the OVSS model with Region-Localized Preference Optimization (RLPO) together with consistency regularization that stabilizes updates outside the queried region. Across extensive experiments on the MESS benchmark, the proposed method achieves consistent gains across diverse OVSS backbones without any pixel-level annotation, and remains effective under noisy preferences. Our code is available at https://github.com/blue-531/pref-ovss.
Sources
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation
- Language-driven Semantic Segmentation
- Open-Vocabulary Universal Image Segmentation with MaskCLIP
- OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation
- Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
- Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
- OV-COAST: Cost Aggregation with Optimal Transport for Open-Vocabulary Semantic Segmentation
- Personalized OVSS: Understanding Personal Concept in Open-Vocabulary Semantic Segmentation
- Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
- Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
- A Dense Reward View on Aligning Text-to-Image Diffusion with Preference
- DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution
- DP$^2$O-SR: Direct Perceptual Preference Optimization for Real-World Image Super-Resolution
- SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Multiple-Human Parsing in the Wild
- DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization
- Human Alignment of Large Language Models through Online Preference Optimisation
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models