Towards Visual Query Segmentation in the Wild
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Towards Visual Query Segmentation in the Wild".
Jane: Here's the gist of the paper. Visual query segmentation (VQS) is introduced as a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query.
Tom: This approach enables more comprehensive and precise localization by segmenting all object occurrences with pixel-level masks rather than just the last appearance using bounding boxes.
Jane: Pruning redundant reasoning tokens The paper proposes Visual Query Segmentation (VQS) as a new VQL paradigm that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query.
Tom: Compared to existing VQL locating only the last appearance of a target using bounding boxes, VQS enables more comprehensive (i.e., all object occurrences) and precise (i.e., pixel-level masks) localization, making it more practical for real-world scenarios.
Title and authors: Tom: Okay, so we know VQS aims for pixel-level masks for all occurrences in untrimmed videos. What’s the actual setup they use to train or test this idea? Jane, can you walk us through the training signal and what kind of challenge they set up for themselves?
Jane: They present a large-scale benchmark called VQS-4K <ref:2603.08898#pg1>. This dataset has four thousand one hundred eleven videos with over one point three million frames and covers two hundred twenty-two different object categories <ref:2603.08898#pg1,1.3 million frames and covers>. Each video comes with a visual query defined by a frame outside the search video and its target mask, and they have these spatial-temporal masklets annotated for the queried target.
Lu: That dataset size is substantial; it’s designed specifically for this VQS task, which is important because it sets a high bar for what we expect from this new approach.
Meng: four thousand videos with a million frames—that sounds like a lot of data to process and label manually; how much time or money does that dataset take to build? Is the manual labeling process scalable?
Lalam: They mention they ensured high quality by having all videos in VQS-4K manually labeled with meticulous inspection and iterative refinement, including re-labeling one hundred fifty videos with an intersection over union of zero point nine two.
The paper's summary: Tom: So, we have the goal and the data setup. Now, what actually makes their proposed method work better than what’s already out there? Lu, where does the actual technical innovation live in this VQS-4K paper <ref:2603.08898#pg1>?
Lu: The core of their improvement is a model they call VQ-SAM which extends something called SAM two <ref:2603.08898#pg2>. It uses a multi-stage architecture where it progressively evolves memory by leveraging both target-specific information and background distractor cues from the video, guided by an adaptive memory generation module.
Jane: That sounds complex; can you unpack that for us? How does this progressive memory evolution actually help the model find those missed occurrences, especially when there are lots of distractors in a video?
Lalam: They use features extracted from potential object masks and confidence scores in each stage to select top target and distractor regions, then use those features with the initial visual query memory to generate the new memory.
Meng: So it’s like they are learning what matters—is this feature important for finding the target or just noise—and they adjust based on that? That sounds like a lot of dynamic computation; what is that adaptive module actually doing mathematically?
The paper's improvements: Tom: We’ve covered the setup and the core mechanism, but before we wrap up, where does this all land in terms of performance compared to other methods they tested? Jane, what are the key numbers they gave us on how much better VQ-SAM is than existing approaches?
Jane: They show that VQ-SAM consistently achieves a stAP score of twenty-six point zero percent and a tAP score of twenty-six point eight percent, which is better than the second best method, SAM two Long, which got eighteen point six percent for stAP and twenty-four point four percent for tAP.
Lu: That performance gap over SAM two Long is what really shows the value of incorporating target-specific information and distractor cues into the memory evolution process as they described in their multi-stage framework <ref:2603.08898#pg2>.
Meng: From a deployment standpoint, if we look at the inference speed, they say it runs at twenty-nine frames per second on an Nvidia A6000 GPU for an eight frame video clip; that’s a concrete metric for how much throughput we can expect.
Lalam: And they also showed that incorporating things like TFG and DFG—which select target and distractor features based on divergence and predicted intersection over union scores—improves the scores, reaching twenty-six point zero percent stAP with twenty-nine point six percent tAP when both are used.
Tom: So, to wrap up the whole thing, this paper, "Towards Visual Query Segmentation in the Wild", presents VQ-SAM as a robust way to go from just finding the last object occurrence with bounding boxes to getting pixel-level masks for every single instance in an untrimmed video. It sets a new standard with their VQS-4K benchmark and shows that by intelligently evolving memory using target and distractor cues, you can get solid results, hitting twenty-six point zero percent stAP <ref:2603.08898#pg1>.
Jane: Exactly. It’s a significant contribution because it moves the goal from just localization to comprehensive segmentation across time in videos. We need to think about how this kind of precise understanding helps downstream tasks like automated video retrieval or surveillance systems.
Lu: It certainly opens up new avenues for research into memory evolution frameworks that can dynamically learn which features are most important at each step of a complex visual query task.
Meng: And the fact that they built VQS-4K means there’s a public resource now for anyone wanting to try and build something similar, even if it’s just running inference on smaller clips <ref:2603.08898#pg1>.
Lalam: The paper shows strong performance across different scales too, which suggests this approach isn't just good on small tests but is generally reliable when dealing with varied video content.
Conclusion: Tom: So we've talked about Visual Query Segmentation in the Wild, where they tackle finding every single instance of an object in an untrimmed video using a visual query and aiming for pixel-level masks instead of just bounding boxes.
Jane: Right, it’s really about moving beyond just finding the last thing you saw to actually mapping out everything that happened.
Lu: The core idea is using a multi-stage memory framework that progressively evolves by pulling in both what the query asked for and what's in the background, which they call Adaptive Memory Generation.
Meng: From a deployment standpoint, getting pixel masks across millions of frames sounds computationally heavy; I wonder how scalable this actually is for real-time use on actual video feeds.
Lalam: The results they show are pretty solid; VQ-SAM hits twenty-six point zero percent stAP and twenty-nine point six percent tAP, which is better than the other models they compared it to.
Tom: That twenty-six point zero percent stAP is what’s telling us this method actually performs well in the messy reality of untrimmed video footage.
Jane: It means for someone just listening, it changes how we think about searching through long videos; instead of just finding one thing, you get a detailed map.
Lu: They built a really big dataset called VQS-4K with over four thousand videos and two hundred and twenty different object categories to test this on <ref:2603.08898#pg1>.
Meng: A dataset that big is impressive, but I always wonder if those manual labels truly reflect what happens when you run it on unpredictable, messy production data.
Lalam: The construction pipeline they used involved five steps, from initial design to the final iterative refinement of the labels with an intersection over union of zero point nine two accuracy.
Tom: That level of manual effort suggests that getting high-quality training signals for these complex tasks is still a big hurdle in this area.
Jane: And even with that large scale, they show how incorporating target and distractor features through things like DFG helps the model distinguish the actual object from just background noise.
Lu: It shows that by having the memory evolve stage by stage, where each step mines both target and distractor information, you get a much more robust representation of what's going on in that video.
Meng: That dynamic learning of feature importance through the adaptive module sounds ambitious; I hope it doesn't just create a massive model that runs forever.
Lalam: The authors also noted that VQ-SAM can be used for visual query localization, achieving the best scores on all metrics on their validation set.
Tom: So they’re basically saying this approach is promising for real-world video segmentation, and it's available for others to check out.
Jane: It really lays a foundation here for more complex visual understanding tasks down the road.
Lu: This paper, "Towards Visual Query Segmentation in the Wild," gives us a solid direction for how to approach object occurrence localization in challenging video scenarios.
Department of Computer Science and Engineering, University of North Texas
cs.CV
Submitted: 2026-03-09
Updated: 2026-11-03
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 81/100
The gist: Visual query segmentation (VQS) is introduced as a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video…
Key concepts
- Visual Query Segmentation (VQS)
- This paradigm aims to segment all pixel-level instances of a specific object within an entire, untrimmed video. Unlike previous methods that only find the last appearance using bounding boxes, VQS provides detailed masks for every occurrence.
- Adaptive Memory Generation (AMG)
- This module dynamically learns which features—target or background distractor cues—are most important at each step of memory evolution. It uses global pooling and an MLP to assign importance weights, allowing the system to intelligently combine different feature sources into a richer, evolving memory.
- Progressive Memory Evolution
- The system operates in multiple stages (K stages), where in each stage, it mines features from the current memory. This process iteratively refines the representation by incorporating information about both the target and surrounding background distractors to build a more comprehensive understanding of the scene.
- Distractor Features (DFG)
- These are features extracted from background elements that are not the primary target of interest. By incorporating distractor features, the model can learn to better discriminate between the object and its surroundings, improving robustness against confusing background elements.
Terminology
Summary
Visual query segmentation (VQS) is introduced as a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query. p. 0 This approach enables more comprehensive and precise localization by segmenting all object occurrences with pixel-level masks rather than just the last appearance using bounding boxes. p. 0
Pruning redundant reasoning tokens
The paper proposes Visual Query Segmentation (VQS) as a new VQL paradigm that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query. Compared to existing VQL locating only the last appearance of a target using bounding boxes, VQS enables more comprehensive (i. p. 0e., all object occurrences) and precise (i. p. 0e., pixel-level masks) localization, making it more practical for real-world scenarios. p. 0 This paradigm differs from Video Object Segmentation (VOS) because the query in VQS is sourced from an image frame outside the search video, unlike VOS where the reference target is sampled from the first frame within the video. p. 1 Furthermore, VQS performs a video-level global search across an untrimmed video, which makes it a needle-in-the-haystack problem requiring a model to identify and segment sparse and intermittent target occurrences amidst substantial background distractors. p. 1
How the training signal is built
To foster research on this task, the paper presents VQS-4K, a large-scale benchmark dedicated to VQS. This dataset contains 4,111 videos with more than 1. p. 43 million frames and covers a diverse set of 222 object categories. p. 0 Each video is paired with a visual query defined by a frame outside search video and its target mask, and annotated with spatial-temporal masklets corresponding to the queried target. p. 0 To ensure high quality, all videos in VQS-4K are manually labeled with meticulous inspection and iterative refinement. p. 0 The dataset split includes 3,289 videos for training (VQS4KTra) and 822 sequences for testing (VQS4KTst). p. 6 Evaluation employs multiple metrics including spatial-temporal Average Precision (stAP), temporal Average Precision (tAP), Recovery (Rec), and Success (Succ) and stAP measures the accuracy of spatio-temporal prediction, while tAP is applied to evaluate the accuracy of temporal prediction. p. 6
Where the gains come from
The proposed method, VQ-SAM, extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS. The framework involves several key components:
-
Feature Extraction: A shared encoder Φ(·) is used to extract features for the visual query and video frames, generating initial memory Minit = MemEnc(fQ, mQ). p. 7
-
Progressive Memory Evolution: In each stage k (1 ≤ k < K), target and distractor features are mined using current memory, which are then fed into AMG to generate new memory Mk+1. p. 8
-
Adaptive Memory Generation (AMG): This module dynamically learns the relative importance of different feature sources by applying global pooling and an MLP, followed by a softmax normalization to learn adaptive importance weights w. p. 9
The ablation studies demonstrate the effectiveness of these components:
- Impact of TFG and DFG:
Without TFG and DFG, stAP and tAP scores are 23.8% and 24.9%. When applying TFG along, stAP and tAP are improved to 24. p. 68% and 25.4% with 1.0% and 0.5% gains (v.s.). When use DFG along, stAP and tAP are improved to 25. p. 64% and 27.3% with 1.6% and 2.4% gains (v.s.). When using both TFG and DFG, they achieve the best performance with 26. p. 00% stAP and 29. p. 66% tAP (v.s.).
- Impact of the STT block:
Without STT, stAP and tAP scores are 25.2% and 27.9%. When applying STT in VQ-SAM, stAP and tAP scores are improved to 26. p. 60% and 29.6%.
- Impact of the number of stages:
When setting K = 1, which means no target and distractor information is explored for memory evolution, the stAP and tAP are 21.5% and 13.9%. When adding a stage, the stAP and tAP are significantly improved to 26. p. 20% and 29.6%.
- Impact of AMG:
Using AMG achieves the best results (v.s /).
The final performance comparison shows that VQ-SAM consistently achieves the best stAP score with 26.0% and tAP score with 26. p. 68%, surpassing other approaches. p. 0 Specifically, VQ-SAM outperforms the second best SAM2Long with 18. p. 06% stAP and 24. p. 64% tAP by 7. p. 64% and 5.2%, clearly showing its superiority. p. 3 The method also outperforms the state-of-the-art VQL method REN with 16. p. 28% stAP and 23. p. 66% tAP by 9. p. 62% in stAP and 6. p. 60% in tAP.
The paper concludes that VQS, VQS-4K, and VQ-SAM provide a foundation for future research and practical applications on visual query segmentation. The benchmark VQS-4K is the first publicly available benchmark specifically designed for VQS. p. 2 The method VQ-SAM demonstrates promising performance and largely surpasses current methods, establishing a solid foundation for future research on VQS and its results are made publicly available. p. 2 The paper also shows that even VQ-SAM can be used for VQL, achieving the best performance on all metrics on the VQ2D validation set.
The supplementary material provides details on object categories, construction pipeline, evaluation metrics, and more. The VQS-4K dataset comprises 222 object classes organized hierarchically. p. 4 The construction pipeline involves five steps, including overall design, video collection from YouTube, video selection, initial labeling by experts, and iterative inspection and refinement. p. 5 The annotation accuracy analysis shows an IoU of 0. p. 692 when re-labeling 150 videos. p. 5 The memory requirement for running VQ-SAM is 9. p. 07G for an eight-frame video clip, and the inference speed on Nvidia A6000 GPU is 29 frames per second (fps). p. 4 The paper also discusses ethical statements and maintenance plans for VQS-4K. p. 0
The category organization of VQS-4K shows 19 coarse classes and 222 fine categories, with human
separated from mammal
. The dataset covers diverse contexts, including aircraft, aquatic animals, birds, clothing & accessory, electronic items, daily items, food, human subjects (separated), insects (7), kitchenware (15), mammals (6), personal care (22), reptiles (10), sport equipment (38). The paper emphasizes that VQ-SAM can accurately identify target-specific and background distractor regions in the videos, which are used to enhance the discriminability of evolved memory. p. 2 The visualization of attention maps shows that with memory evolution, the attention maps can better focus on queried target in the video, leading to better localization and segmentation performance. p. 7 The paper also notes that VQ-SAM's performance is robust across different scales over all metrics.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive Memory Generation (AMG) module can dynamically learn importance weights for different feature types, integrating target-specific and background distractor cues to form a new memory for VQS stages, as described by
AMG learns importance weights for different feature types and integrates them accordingly to form memory.
-
VQ-SAM can perform comprehensive localization by capturing all target occurrences throughout an untrimmed video and precise localization by representing objects with pixel-level masks, moving beyond the existing paradigm of locating only the last appearance using bounding boxes.
-
The system can handle target variations in videos better because it incorporates target-specific features, and more reliable discrimination between the target and the background by leveraging distractor features, as noted in
By incorporating the target-specific features, the newly evolved memory can better handle target variations in videos.
-
The model can resist distinguishing targets from background distractors by incorporating distractor information into memory through DFG, which selects alternative masks based on divergence and predicted IoU scores.
-
The system can improve localization and segmentation by progressively evolving memory through a multi-stage framework where each stage mines confident target and background distractor features using current memory.
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs