Vision-Language Grounding as Bidirectional Concept Correspondence
Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna
cs.CV, cs.AI, cs.CL
Submitted: 2026-08-08
Updated: 2026-08-11
Project page: https://uwgzq.github.io/papers/ConCor-1
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category
Terminology
Abstract
Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as bidirectional concept correspondence over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce ConCor-1, a grounding model built on top of a pretrained vision-language model. It uses learnable bridge tokens to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that ConCor-1 consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.
Sources
- MolmoPoint: Better Pointing for VLMs with Grounding Tokens
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- Selective Visual Representations Improve Convergence and Generalization for Embodied AI
- Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
- TrajTok: Learning Trajectory Tokens enables better Video Understanding
- Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision
- An Open and Comprehensive Pipeline for Unified Object Grounding and Detection
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts
- Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
- LISA: Reasoning Segmentation via Large Language Model
- LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model
- COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation
- GREC: Generalized Referring Expression Comprehension
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models