WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World-Model Features
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World-Model Features".
Paper summary: Tom: So, we're talking about this paper called WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World-Model Features. The big idea here is trying to locate one specific object instance in a messy scene using just one reference image and a short description.
Jane: It’s really interesting because large vision models usually handle finding an object category from text, but this paper asks if you can do the same thing for individual objects cheaply, using the knowledge already learned by a world model pretraining objective.
Lu: The core claim is that they present WALDO, which is a one-shot exemplar and language conditioned detection head with three point four million trainable parameters that uses frozen V-JEPA two point one features to predict both object localization and target presence at the same time, without any gradient on the backbone <ref:2608.28216#pg1,a one-shot exemplar and language conditioned detection head with>.
Meng: So it’s trying to solve this problem where open-vocabulary detectors answer a category question, but instance methods drop language conditioning and treat every query as answerable instead of just finding one specific thing.
Lalam: WALDO achieves a catalogue mean precision at fifty against zero point three zero six for a prompted Grounding DINO baseline when using the same scorer <ref:2608.28216#pg3>.
Tom: That’s a significant jump in catalogue performance, but what makes this approach different is how they handle absence determination. They treat absence as a trained output rather than just something that happens because the confidence is low.
Jane: And they build their training signal by synthesizing training episodes from instance annotations, mining exemplars from ground-truth boxes, and constructing specific absence cases that exclude the referenced instance while keeping other similar objects in view.
Lu: They synthesize three kinds of training examples: positive, within-category absent, and cross-category absent. They actually draw the region scale before choosing the label to stop shortcut learning from happening.
Meng: The augmentation they use is pretty standard but thorough—rotation up to twelve degrees, random re-cropping between seventy eight and one hundred percent, photometric jitter, Gaussian blur, and a JPEG round-trip at quality between forty and eighty eight.
Lalam: The performance comparison shows that world model features transfer to localization precision and absence detection but not to instance identity.
Tom: That’s a key distinction because it means they aren't claiming they can figure out *which* specific car it is, just where the target is and if it's there.
Jane: However, instance retrieval falls quite low at zero point one nine zero for V-JEPA two point one and DINOv3, which the authors say is statistically indistinguishable from a random draw among all the scene’s instances <ref:2608.28216#pg1>.
Lu: Instance retrieval fails in a lot of ways; success at one falls to zero point one nine zero for V-JEPA two point one, zero point one nine zero for DINOv3, and two point two four four for SigLIP2 <ref:2608.28216#pg1>.
Meng: The evidence is even blunter because seventy nine out of one hundred sixteen instance queries have a top-one box whose intersection over union with the correct object is exactly zero against sixty five point five percent for DINOv3 and sixty three point eight percent for SigLIP2.
Tom: So even when it finds the right category, it struggles to pick the specific one from a crowd, which means instance discrimination still has to come from just that exemplar crop alone.
Jane: But they do manage absence detection quite well; V-JEPA two point one reaches zero point eight eight zero against nine for hard negatives when it comes to determining if something is missing <ref:2608.28216#pg1>.
Lu: The scale and box tightness decomposition shows that by object scale, aircraft reach a catalogue mean precision at fifty of zero point eight seven nine, zero point eight seven seven, and zero point eight seven four while cars reach one point six six and one point five zero.
Meng: That gap is attributed to the input resolution and object density rather than the representation itself.
Tom: And when you look at box tightness, V-JEPA falls from that catalogue mean precision of zero point four six one down to zero point one six eight, SigLIP2 falls from zero point four eight one to zero point zero six three, and DINOv3 falls from zero point four two eight down to just two point one.
Jane: It seems like this supports the idea that world model pretraining transfers here to localization precision and presence estimation, but not necessarily to retrieval accuracy.
Lu: The shared box-tightness limit is plausibly architectural, constraining each patch's predicted center to be within plus or minus one patch width caps precision for objects a few patches across.
Tom: So the main conclusion of this paper is that WALDO localizes a referenced instance from one exemplar and a description over frozen representations, treating absence as a trained output rather than just a confidence threshold.
Jane: The catalogue mean precision at fifty reaches zero point four six one against zero point three zero six for the strongest open-vocabulary baseline when using one scorer.
Meng: But the instance identity part is where it doesn't follow, because success at one sits right at its category chance floor of zero point one nine zero.
Lu: The model finds the category, then it picks wrongly inside that category, which is what the results show on page six of this work.
Tom: So to wrap up WALDO, it’s a method that uses a world model to do one-shot detection and presence checking in cluttered scenes by explicitly training for absence detection.
Jane: That was the whole picture on how they set up the training signal and what they found about the performance trade-offs between localization and retrieval.
Conclusion: Tom: So, we're wrapping up WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World Model Features. The main point here is that they used a world model to do one-shot object detection and presence checking in crowded scenes.
Jane: Right, so if you had a picture and just one description, you can find one specific thing and know if it's there or not, without needing tons of training data for every single object. That’s what this paper is about.
Lu: The authors use a detection head that conditions on an exemplar—that's the reference image—and the language description, and they fuse that with frozen features from a world model. They call it WALDO.
Meng: So it's not training a whole new detector from scratch; you're just adding this small head on top of something already trained, but you’re using that existing knowledge to do a very specific task: finding one thing.
Tom: Exactly, and the key is how they treat absence. They train the model to recognize when something is missing, treating that absence as a learned output instead of just guessing because it's low confidence.
Jane: And they show some solid numbers here too—the catalogue mean precision at fifty hits zero point four six one, which beats a baseline by using the same scorer.
Lu: But there’s a catch on the instance part, though; they find that while localization is good, finding the exact same object in a crowd is still pretty hard. Success at one falls to zero point one nine zero for models like V-JEPA two point one and DINOv3.
Meng: That means you can find the general category, but when you ask for *that specific* car from a picture with fifty cars, it's like picking randomly from the group.
Tom: So the implication is that this world model helps localize things precisely and figure out if they're present in a scene, but it doesn't magically solve the problem of telling one object apart when they look similar.
Jane: It seems like WALDO’s most powerful feature right now is its ability to confirm presence—knowing something *isn't* there—which is huge for safety applications.
Lu: Exactly, and that absence detection success against nine hard negatives shows that the world model features are really good at spatial reasoning about what should be there.
Meng: So it’s a strong tool for presence checking in complex environments, but if you need to identify every single specific thing, you still have a lot of work to do on that retrieval part.
Tom: The authors are clear here: the model finds the category, then it picks wrongly inside that category. That's the real limitation they’re admitting upfront.
Jane: So what does this mean for people who only listen to this show? It means if you need to know *if* a specific hazard is in your field of view, WALDO is getting really good at telling you yes or no.
Lu: The next thing we're looking at is how they built that training signal from those instance annotations and exemplars, which explains why the absence detection works so well.
Department of Cyber Physical Systems, Clark Atlanta University · Department of Computer Science and Engineering, United International University
cs.CV
Submitted: 2026-08-28
Updated: 2026-11-17
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: Locating one specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent…
Key concepts
- Exemplar-Conditioned Detection
- This is the core method where the model uses one specific example (an exemplar) and a text description to guide its search for an object. Instead of just classifying categories, it learns to pinpoint exactly where that specific instance is located within a complex scene based on the provided reference.
- Absence Determination
- WALDO is trained not only to find objects but also explicitly to determine when an object is *not* present in the scene. This means the model learns a dedicated output for 'absence,' which helps it distinguish between different objects and makes its localization more robust, even when an instance is missing.
- Instance Retrieval Failure
- While WALDO excels at finding the correct category (catalogue detection), it fails to accurately retrieve the specific identity of an instance. The model often locates a plausible object but guesses incorrectly among similar instances, indicating that its ability to discriminate between individual objects is weak.
Terminology
Summary
Locating one specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent, large vision-language models usually address this task <ref:2608. p. 0 Mahfuzur Rahman1
, Md. Sadman Haque2
and Mohd Ariful Haque1
1Department of Cyber Physical Systems, Clark Atlanta University, Atlanta, GA, USA
2Department of Computer Science and Engineering, United International University, Dhaka, Bangladesh
August 31, 2026
ABSTRACT
Locating a specific object instance in a cluttered scene using a single reference image and a short
description, and reporting when that instance is absent, large vision-language models usually address this task. We ask whether the same capability is available far more cheaply, from representations already learned by a world-model pretraining objective. We present WALDO, a one-shot
exemplar- and language-conditioned detection head with 3. We present WALDO, a one-shot
exemplar- and language-conditioned detection head with 3. 4M trainable parameters that reads frozen
V-JEPA 2. 4M trainable parameters that reads frozen
V-JEPA 2. 1 features to jointly predict object localization and target presence, with no gradient on
the backbone. Referring expression comprehension [9, 23] disambiguates one object through language alone, but every benchmark guarantees the referent is present,
so these methods are never trained to abstain. One-shot detectors such as CoAE [5] and OS2D [15] replace the category label with an exemplar image and instance-level correspondence, but drop language conditioning and treat every
query as answerable. 9998 absence AUROC without ever consulting the exemplar, and we report
the negative controls that close the shortcut. On 35 held-out cluttered scenes, WALDO achieves a
0. On 35 held-out cluttered scenes, WALDO achieves a
0. 461 catalogue AP@50, compared to 0. 461 catalogue AP@50, compared to 0. 306 for a prompted Grounding DINO baseline under an
identical scorer.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The core contribution of the work is formulating exemplar-conditioned detection with explicit presence determination, distinct from category-level open-vocabulary detection <ref:2608.28216#pg6> WALDO proposes a lightweight, backbone-agnostic head fusing exemplar correlation with FiLM conditioning to predict localization and presence jointly <ref:2608. p. 1 (i) We formulate exemplar-conditioned detection with explicit presence determination, distinct from
category-level open-vocabulary detection. (ii) We propose WALDO, a lightweight, backbone-agnostic head fusing exemplar correlation with FiLM conditioning to predict localization and presence jointly. 1 [14], compared under an identical recipe to DINOv3 [19] and SigLIP2 [20]. It fuses a
patch-wise correlation score against the exemplar, the strongest known inductive bias for one-shot correspondence [5,
15], with FiLM conditioning [16] from the joint exemplar-text embedding, and jointly predicts per-patch localization
and scene-level presence, so absence is a trained output rather than a byproduct of low confidence. 1 features to jointly predict object localization and target presence, with no gradient on
the backbone. Because exemplar-conditioned supervision is scarce, we synthesize training episodes
from instance annotations, mining exemplars from ground-truth boxes and constructing absence
cases that exclude the referenced instance while leaving same-category distractors in view.Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The authors synthesize exemplar-conditioned episodes by mining exemplars from ground-truth boxes, creating three kinds of training examples: Positive, Within-category absent, and Cross-category absent <ref:2608.28216#pg4> The construction of these episodes involves drawing the region scale before the label is chosen to prevent shortcut learning <ref:2608. p. 2 9998 absence AUROC
by epoch six without consulting the exemplar, having learned to measure the crop rather than look for the object. We
break the correlation by drawing the region scale from one distribution shared by all three episode kinds before the
label is chosen, routing whole-scene absences to the cross-category kind, the only kind that can express one. 9998 absence AUROC
by epoch six without consulting the exemplar, having learned to measure the crop rather than look for the object. We
break the correlation by drawing the region scale from one distribution shared by all three episode kinds before the
label is chosen, routing whole-scene absences to the cross-category kind, the only kind that can express one. A mined exemplar is a crop of the very image in which it must be found, so a model could
in principle match exact pixels; augmentation is the only defence. We apply rotation (±12◦
), random-scale re-crop to
78 to 100%, photometric jitter, Gaussian blur and sensor noise, and a JPEG round-trip at quality 40 to 88, the last two
attacking exact-pixel matching directly, re-rolled every epoch, since exemplars are not cached.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The performance comparison shows that world-model features transfer to localization precision and absence detection but not to instance identity <ref:2608.28216#pg4> WALDO substantially outperforms the strongest open-vocabulary baseline on catalogue detection, while instance identity remains unresolved <ref:2608. p. 1 (iii) We introduce an episodemining strategy generating instance-level and hard/easy absence supervision from sparse annotations, with region
geometry decoupled from the label to prevent shortcut learning. (iv) We show WALDO substantially outperforms the
strongest open-vocabulary baseline on catalogue detection, while instance identity remains unresolved. 5) and one detection cap, so every number is on a single scale. 1 Category retrieval succeeds, instance retrieval fails
At the catalogue level all three backbones retrieve the target category reliably (Table 2): SigLIP2 attains the highest
Success@1 at 0. Success@1 falls to
0. 190 (V-JEPA 2. p. 6 Success@1 falls to
0. 190 (V-JEPA 2. 0 against observed 22, 22 and 26. Instance selection
is statistically indistinguishable from a fair draw among the scene’s instances: the model locates a Highlander,
then guesses which one was meant. It is one query, not a representative outcome: over
twelve paired McNemar tests the multi-look variant never differs significantly from single-pass inference, so we report
single-pass throughout and read panel (a) as an illustration of the region-callable interface, not evidence of a gain. 2 Target absence
Where instance discrimination fails, absence detection succeeds (Table 4). 1) are
scored once per scene and once per instance; since they apply different acceptance rules to the same predictions, their
difference is reported only as the transfer gap, never as a performance gap. A third absence setting asks whether the
model withholds a detection when the exemplar’s object is not in view, reporting hard negatives (a region excluding
that object) and easy ones (an exemplar against a scene of the other category) separately: 36 of the 46 hard negatives
(78.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Instance retrieval fails
The results indicate that instance retrieval tells a sharply different story, as Success@1 falls to 0.190 (V-JEPA 2.1), 0.190 (DINOv3) and 0. p. 6 190 (V-JEPA 2. 190 (DINOv3) and 0. 190 (DINOv3) and 0. 224 (SigLIP2), and must be read against the floor column of Table 4, the
score of a model that solves catalogue retrieval and then guesses uniformly among that scene’s instances, for ratios of
1. 0 against observed 22, 22 and 26. Instance selection
is statistically indistinguishable from a fair draw among the scene’s instances: the model locates a Highlander,
then guesses which one was meant. In the V-JEPA arm 79 of 116 instance queries (68. 1%) have a top-1 box whose
IoU with the correct object is exactly zero, against 65. 306 for a prompted Grounding DINO baseline under an
identical scorer. Substituting DINOv3 for V-JEPA under a matched 576-token grid drops withincategory absence AUROC from 0. The target beat every distractor in only 15. 9% of
arXiv:2608. A model using the exemplar to discriminate among
same-category candidates would close this gap; none does. 1’s bound applies: discrimination must come from
the exemplar crop alone, and that crop is small, a median 87 × 108 px needing 3.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Scale and box tightness
The catalogue AP@50 decomposition shows that by object scale, aircraft reach AP@50 of 0.879, 0.877 and 0.874 while cars reach 0. p. 8 By object scale,
aircraft reach AP@50 of 0. 874 while cars reach 0. The target beat every distractor in only 15. 9% of
arXiv:2608. The median car box measures 67 × 83 px, only 3 to 4 patches at 384 px input, so a one-patch error consumes a
large fraction of it, and cars appear at up to ten instances per scene, with a median of 3. We attribute the gap to input
resolution and object density, not to the representation. Splitting the same number by IoU strictness is where the backbones separate. From AP@50 to AP@75, V-JEPA
falls from 0. From AP@50 to AP@75, V-JEPA
falls from 0. 168, SigLIP2 from 0. 168, SigLIP2 from 0. 063 and DINOv3 from 0. 063 and DINOv3 from 0. 021, reversing the AP@50
ordering. This supports a narrower claim than V-JEPA is the better backbone: SigLIP2 finds objects slightly more often, while
V-JEPA places boxes more precisely and abstains more reliably. Since the catalogue Success@1 difference is inside
the noise floor while the AP@75 and hard-absence differences are not, world-model pretraining transfers here to
localization precision and presence estimation, not to retrieval accuracy. Since the catalogue Success@1 difference is inside
the noise floor while the AP@75 and hard-absence differences are not, world-model pretraining transfers here to
localization precision and presence estimation, not to retrieval accuracy. The shared box-tightness limit is plausibly
architectural: constraining each patch’s predicted center to within ±1 patch width caps precision for objects a few
patches across.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
WALDO localizes a referenced instance from one exemplar and a description over frozen representations, treating absence as a trained output rather than a confidence threshold <ref:2608.28216#pg6> Catalogue AP@50 reaches 0. p. 8 6 Conclusion
WALDO localizes a referenced instance from one exemplar and a description over frozen representations, treating
absence as a trained output rather than a confidence threshold. Catalogue AP@50 reaches 0. Catalogue AP@50 reaches 0. 461 against 0. 461 against 0. 306 for
the strongest open-vocabulary baseline under one scorer, and within-category absence AUROC reaches 0. 726 for the image-pretrained controls, the one measurement on which the three representations separate. Instance identity does not follow: Success@1 of 0. Instance identity does not follow: Success@1 of 0. 190 sits at its category-chance floor, and 68. 1% of instance queries return a box overlapping nothing that counts. The model finds the category, then picks wrongly inside
it.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Exemplar-conditioned detection with explicit presence determination allows AI systems to
localize that object and state plainly that it is absent,
enabling robotic fetch-and-place or retail auditing tasks by distinguishing between instance identity and category membership. -
WALDO can perform one-shot object detection using a single reference image and a short description, predicting both the object's localization and its presence in the scene simultaneously, which is distinct from open-vocabulary detectors that only answer category-level questions.
-
The system uses a novel training strategy called episodemining to synthesize instance-level and hard/easy absence supervision from sparse annotations, which helps prevent shortcut learning where region size alone predicts the label.
-
By fusing a patch-wise correlation score against the exemplar with FiLM conditioning from the joint exemplar-text embedding, WALDO is designed to
jointly predict per-patch localization and scene-level presence,
ensuring absence is a trained output rather than a byproduct of low confidence. -
The architecture incorporates a correlation channel that computes the cosine similarity between each patch and the exemplar vector in the backbone's own space before any learned projection can distort it, allowing
the signal available to every downstream layer.
Abstract
Finding a particular object instance in a cluttered scene from a single reference image and saying when it is not there is typically given to large vision-language models. We ask whether the representations of a world model add something that the strongest self-supervised image features do not have. We introduce WALDO, a one-shot detector that finds a specific object instance in a cluttered real scene from a single reference image. Instance detection has usually needed per-object training. The alternative is a category-level open-vocabulary detector or vision-language model, which can find a mug but not this mug. We show that a lightweight conditioned head on frozen self-supervised features is enough to localize unseen instances without any per-object adaptation. We also find that image features and world-model features are complementary: DINOv3 and the video world model V-JEPA 2.1 succeed on different queries. There is one catch. Used the way video models are usually applied to still images, V-JEPA's dense features underperform. Reading them through the model's image pathway improves their localization. Combining the two backbones with a validation-tuned late fusion modestly improves DINOv3's localization of unseen instances without hurting its ability to tell whether the object is in the scene. A strong vision-language model and an image-guided open-vocabulary detector almost never localize the specific instance, and take far longer per query. Our fusion with world-model features strengthens localization from image features alone.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
- DINOv3
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- Frustratingly Simple Few-Shot Object Detection
- Detect Everything with Few Examples
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs