WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World-Model Features

arXiv:2608.28216 · cs.CV · Submitted 2026-08-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World-Model Features".

Paper summary: Tom: So, we're talking about this paper called WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World-Model Features. The big idea here is trying to locate one specific object instance in a messy scene using just one reference image and a short description.

Jane: It’s really interesting because large vision models usually handle finding an object category from text, but this paper asks if you can do the same thing for individual objects cheaply, using the knowledge already learned by a world model pretraining objective.

Lu: The core claim is that they present WALDO, which is a one-shot exemplar and language conditioned detection head with three point four million trainable parameters that uses frozen V-JEPA two point one features to predict both object localization and target presence at the same time, without any gradient on the backbone <ref:2608.28216#pg1,a one-shot exemplar and language conditioned detection head with>.

Meng: So it’s trying to solve this problem where open-vocabulary detectors answer a category question, but instance methods drop language conditioning and treat every query as answerable instead of just finding one specific thing.

Lalam: WALDO achieves a catalogue mean precision at fifty against zero point three zero six for a prompted Grounding DINO baseline when using the same scorer <ref:2608.28216#pg3>.

Tom: That’s a significant jump in catalogue performance, but what makes this approach different is how they handle absence determination. They treat absence as a trained output rather than just something that happens because the confidence is low.

Jane: And they build their training signal by synthesizing training episodes from instance annotations, mining exemplars from ground-truth boxes, and constructing specific absence cases that exclude the referenced instance while keeping other similar objects in view.

Lu: They synthesize three kinds of training examples: positive, within-category absent, and cross-category absent. They actually draw the region scale before choosing the label to stop shortcut learning from happening.

Meng: The augmentation they use is pretty standard but thorough—rotation up to twelve degrees, random re-cropping between seventy eight and one hundred percent, photometric jitter, Gaussian blur, and a JPEG round-trip at quality between forty and eighty eight.

Lalam: The performance comparison shows that world model features transfer to localization precision and absence detection but not to instance identity.

Tom: That’s a key distinction because it means they aren't claiming they can figure out *which* specific car it is, just where the target is and if it's there.

Jane: However, instance retrieval falls quite low at zero point one nine zero for V-JEPA two point one and DINOv3, which the authors say is statistically indistinguishable from a random draw among all the scene’s instances <ref:2608.28216#pg1>.

Lu: Instance retrieval fails in a lot of ways; success at one falls to zero point one nine zero for V-JEPA two point one, zero point one nine zero for DINOv3, and two point two four four for SigLIP2 <ref:2608.28216#pg1>.

Meng: The evidence is even blunter because seventy nine out of one hundred sixteen instance queries have a top-one box whose intersection over union with the correct object is exactly zero against sixty five point five percent for DINOv3 and sixty three point eight percent for SigLIP2.

Tom: So even when it finds the right category, it struggles to pick the specific one from a crowd, which means instance discrimination still has to come from just that exemplar crop alone.

Jane: But they do manage absence detection quite well; V-JEPA two point one reaches zero point eight eight zero against nine for hard negatives when it comes to determining if something is missing <ref:2608.28216#pg1>.

Lu: The scale and box tightness decomposition shows that by object scale, aircraft reach a catalogue mean precision at fifty of zero point eight seven nine, zero point eight seven seven, and zero point eight seven four while cars reach one point six six and one point five zero.

Meng: That gap is attributed to the input resolution and object density rather than the representation itself.

Tom: And when you look at box tightness, V-JEPA falls from that catalogue mean precision of zero point four six one down to zero point one six eight, SigLIP2 falls from zero point four eight one to zero point zero six three, and DINOv3 falls from zero point four two eight down to just two point one.

Jane: It seems like this supports the idea that world model pretraining transfers here to localization precision and presence estimation, but not necessarily to retrieval accuracy.

Lu: The shared box-tightness limit is plausibly architectural, constraining each patch's predicted center to be within plus or minus one patch width caps precision for objects a few patches across.

Tom: So the main conclusion of this paper is that WALDO localizes a referenced instance from one exemplar and a description over frozen representations, treating absence as a trained output rather than just a confidence threshold.

Jane: The catalogue mean precision at fifty reaches zero point four six one against zero point three zero six for the strongest open-vocabulary baseline when using one scorer.

Meng: But the instance identity part is where it doesn't follow, because success at one sits right at its category chance floor of zero point one nine zero.

Lu: The model finds the category, then it picks wrongly inside that category, which is what the results show on page six of this work.

Tom: So to wrap up WALDO, it’s a method that uses a world model to do one-shot detection and presence checking in cluttered scenes by explicitly training for absence detection.

Jane: That was the whole picture on how they set up the training signal and what they found about the performance trade-offs between localization and retrieval.

Conclusion: Tom: So, we're wrapping up WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes with World Model Features. The main point here is that they used a world model to do one-shot object detection and presence checking in crowded scenes.

Jane: Right, so if you had a picture and just one description, you can find one specific thing and know if it's there or not, without needing tons of training data for every single object. That’s what this paper is about.

Lu: The authors use a detection head that conditions on an exemplar—that's the reference image—and the language description, and they fuse that with frozen features from a world model. They call it WALDO.

Meng: So it's not training a whole new detector from scratch; you're just adding this small head on top of something already trained, but you’re using that existing knowledge to do a very specific task: finding one thing.

Tom: Exactly, and the key is how they treat absence. They train the model to recognize when something is missing, treating that absence as a learned output instead of just guessing because it's low confidence.

Jane: And they show some solid numbers here too—the catalogue mean precision at fifty hits zero point four six one, which beats a baseline by using the same scorer.

Lu: But there’s a catch on the instance part, though; they find that while localization is good, finding the exact same object in a crowd is still pretty hard. Success at one falls to zero point one nine zero for models like V-JEPA two point one and DINOv3.

Meng: That means you can find the general category, but when you ask for *that specific* car from a picture with fifty cars, it's like picking randomly from the group.

Tom: So the implication is that this world model helps localize things precisely and figure out if they're present in a scene, but it doesn't magically solve the problem of telling one object apart when they look similar.

Jane: It seems like WALDO’s most powerful feature right now is its ability to confirm presence—knowing something *isn't* there—which is huge for safety applications.

Lu: Exactly, and that absence detection success against nine hard negatives shows that the world model features are really good at spatial reasoning about what should be there.

Meng: So it’s a strong tool for presence checking in complex environments, but if you need to identify every single specific thing, you still have a lot of work to do on that retrieval part.

Tom: The authors are clear here: the model finds the category, then it picks wrongly inside that category. That's the real limitation they’re admitting upfront.

Jane: So what does this mean for people who only listen to this show? It means if you need to know *if* a specific hazard is in your field of view, WALDO is getting really good at telling you yes or no.

Lu: The next thing we're looking at is how they built that training signal from those instance annotations and exemplars, which explains why the absence detection works so well.

Department of Cyber Physical Systems, Clark Atlanta University · Department of Computer Science and Engineering, United International University

cs.CV

Submitted: 2026-08-28

Updated: 2026-11-17

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: Locating one specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent…

Key concepts

Exemplar-Conditioned Detection
This is the core method where the model uses one specific example (an exemplar) and a text description to guide its search for an object. Instead of just classifying categories, it learns to pinpoint exactly where that specific instance is located within a complex scene based on the provided reference.
Absence Determination
WALDO is trained not only to find objects but also explicitly to determine when an object is *not* present in the scene. This means the model learns a dedicated output for 'absence,' which helps it distinguish between different objects and makes its localization more robust, even when an instance is missing.
Instance Retrieval Failure
While WALDO excels at finding the correct category (catalogue detection), it fails to accurately retrieve the specific identity of an instance. The model often locates a plausible object but guesses incorrectly among similar instances, indicating that its ability to discriminate between individual objects is weak.

Terminology

Summary

Locating one specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent, large vision-language models usually address this task <ref:2608. p. 0

Where the paper says it

Mahfuzur Rahman1 , Md. Sadman Haque2 and Mohd Ariful Haque1 1Department of Cyber Physical Systems, Clark Atlanta University, Atlanta, GA, USA 2Department of Computer Science and Engineering, United International University, Dhaka, Bangladesh August 31, 2026 ABSTRACT Locating a specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent, large vision-language models usually address this task.

28216#pg2> WALDO presents a one-shot exemplar- and language-conditioned detection head with 3. p. 0
Where the paper says it

We ask whether the same capability is available far more cheaply, from representations already learned by a world-model pretraining objective. We present WALDO, a one-shot exemplar- and language-conditioned detection head with 3.

4M trainable parameters that reads frozen V-JEPA 2. p. 0
Where the paper says it

We present WALDO, a one-shot exemplar- and language-conditioned detection head with 3. 4M trainable parameters that reads frozen V-JEPA 2.

1 features to jointly predict object localization and target presence, with no gradient on the backbone <ref:2608. p. 0
Where the paper says it

4M trainable parameters that reads frozen V-JEPA 2. 1 features to jointly predict object localization and target presence, with no gradient on the backbone.

28216#pg4> This approach addresses the gap where open-vocabulary detectors answer a category-level question, while instance-level methods drop language conditioning and treat every query as answerable <ref:2608. p. 0
Where the paper says it

Referring expression comprehension [9, 23] disambiguates one object through language alone, but every benchmark guarantees the referent is present, so these methods are never trained to abstain. One-shot detectors such as CoAE [5] and OS2D [15] replace the category label with an exemplar image and instance-level correspondence, but drop language conditioning and treat every query as answerable.

28216#pg5> WALDO achieves a 0. p. 0
Where the paper says it

9998 absence AUROC without ever consulting the exemplar, and we report the negative controls that close the shortcut. On 35 held-out cluttered scenes, WALDO achieves a 0.

461 catalogue AP@50, compared to 0. p. 0
Where the paper says it

On 35 held-out cluttered scenes, WALDO achieves a 0. 461 catalogue AP@50, compared to 0.

306 for a prompted Grounding DINO baseline under an identical scorer <ref:2608. p. 0
Where the paper says it

461 catalogue AP@50, compared to 0. 306 for a prompted Grounding DINO baseline under an identical scorer.

28216#pg10>

Pruning redundant reasoning tokens

The core contribution of the work is formulating exemplar-conditioned detection with explicit presence determination, distinct from category-level open-vocabulary detection <ref:2608.28216#pg6> WALDO proposes a lightweight, backbone-agnostic head fusing exemplar correlation with FiLM conditioning to predict localization and presence jointly <ref:2608. p. 1

Where the paper says it

(i) We formulate exemplar-conditioned detection with explicit presence determination, distinct from category-level open-vocabulary detection. (ii) We propose WALDO, a lightweight, backbone-agnostic head fusing exemplar correlation with FiLM conditioning to predict localization and presence jointly.

28216#pg8> This method is distinct because it treats absence as a trained output rather than a byproduct of low confidence <ref:2608. p. 1
Where the paper says it

1 [14], compared under an identical recipe to DINOv3 [19] and SigLIP2 [20]. It fuses a patch-wise correlation score against the exemplar, the strongest known inductive bias for one-shot correspondence [5, 15], with FiLM conditioning [16] from the joint exemplar-text embedding, and jointly predicts per-patch localization and scene-level presence, so absence is a trained output rather than a byproduct of low confidence.

28216#pg8> The training strategy involves synthesizing training episodes from instance annotations, mining exemplars from ground-truth boxes and constructing absence cases that exclude the referenced instance while leaving same-category distractors in view <ref:2608. p. 0
Where the paper says it

1 features to jointly predict object localization and target presence, with no gradient on the backbone. Because exemplar-conditioned supervision is scarce, we synthesize training episodes from instance annotations, mining exemplars from ground-truth boxes and constructing absence cases that exclude the referenced instance while leaving same-category distractors in view.

28216#pg4>

How the training signal is built

The authors synthesize exemplar-conditioned episodes by mining exemplars from ground-truth boxes, creating three kinds of training examples: Positive, Within-category absent, and Cross-category absent <ref:2608.28216#pg4> The construction of these episodes involves drawing the region scale before the label is chosen to prevent shortcut learning <ref:2608. p. 2

Where the paper says it

9998 absence AUROC by epoch six without consulting the exemplar, having learned to measure the crop rather than look for the object. We break the correlation by drawing the region scale from one distribution shared by all three episode kinds before the label is chosen, routing whole-scene absences to the cross-category kind, the only kind that can express one.

28216#pg4> Specifically, a natural first implementation makes positives whole-scene and absences sub-crops, but they break this correlation by drawing the region scale from one distribution shared by all three episode kinds before the label is chosen <ref:2608. p. 2
Where the paper says it

9998 absence AUROC by epoch six without consulting the exemplar, having learned to measure the crop rather than look for the object. We break the correlation by drawing the region scale from one distribution shared by all three episode kinds before the label is chosen, routing whole-scene absences to the cross-category kind, the only kind that can express one.

28216#pg4> Exemplar augmentation includes rotation (±12◦), random-scale re-crop to 78 to 100%, photometric jitter, Gaussian blur and sensor noise, and a JPEG round-trip at quality 40 to 88 <ref:2608. p. 2
Where the paper says it

A mined exemplar is a crop of the very image in which it must be found, so a model could in principle match exact pixels; augmentation is the only defence. We apply rotation (±12◦ ), random-scale re-crop to 78 to 100%, photometric jitter, Gaussian blur and sensor noise, and a JPEG round-trip at quality 40 to 88, the last two attacking exact-pixel matching directly, re-rolled every epoch, since exemplars are not cached.

28216#pg4>

Where the gains come from

The performance comparison shows that world-model features transfer to localization precision and absence detection but not to instance identity <ref:2608.28216#pg4> WALDO substantially outperforms the strongest open-vocabulary baseline on catalogue detection, while instance identity remains unresolved <ref:2608. p. 1

Where the paper says it

(iii) We introduce an episodemining strategy generating instance-level and hard/easy absence supervision from sparse annotations, with region geometry decoupled from the label to prevent shortcut learning. (iv) We show WALDO substantially outperforms the strongest open-vocabulary baseline on catalogue detection, while instance identity remains unresolved.

28216#pg4> While category retrieval succeeds across all backbones, instance retrieval falls to 0. p. 6
Where the paper says it

5) and one detection cap, so every number is on a single scale. 1 Category retrieval succeeds, instance retrieval fails At the catalogue level all three backbones retrieve the target category reliably (Table 2): SigLIP2 attains the highest Success@1 at 0.

190 for V-JEPA 2. p. 6
Where the paper says it

Success@1 falls to 0. 190 (V-JEPA 2. p. 6

Where the paper says it

Success@1 falls to 0. 190 (V-JEPA 2.

1 and DINOv3, which is statistically indistinguishable from a fair draw among the scene’s instances <ref:2608. p. 6
Where the paper says it

0 against observed 22, 22 and 26. Instance selection is statistically indistinguishable from a fair draw among the scene’s instances: the model locates a Highlander, then guesses which one was meant.

28216#pg6> Absence detection succeeds where instance discrimination fails, with V-JEPA 2. p. 7
Where the paper says it

It is one query, not a representative outcome: over twelve paired McNemar tests the multi-look variant never differs significantly from single-pass inference, so we report single-pass throughout and read panel (a) as an illustration of the region-callable interface, not evidence of a gain. 2 Target absence Where instance discrimination fails, absence detection succeeds (Table 4).

1 reaching 0.880 against 9 for hard negatives <ref:2608. p. 6
Where the paper says it

1) are scored once per scene and once per instance; since they apply different acceptance rules to the same predictions, their difference is reported only as the transfer gap, never as a performance gap. A third absence setting asks whether the model withholds a detection when the exemplar’s object is not in view, reporting hard negatives (a region excluding that object) and easy ones (an exemplar against a scene of the other category) separately: 36 of the 46 hard negatives (78.

28216#pg6>

Instance retrieval fails

The results indicate that instance retrieval tells a sharply different story, as Success@1 falls to 0.190 (V-JEPA 2.1), 0.190 (DINOv3) and 0. p. 6

Where the paper says it

190 (V-JEPA 2. 190 (DINOv3) and 0.

224 (SigLIP2) <ref:2608. p. 6
Where the paper says it

190 (DINOv3) and 0. 224 (SigLIP2), and must be read against the floor column of Table 4, the score of a model that solves catalogue retrieval and then guesses uniformly among that scene’s instances, for ratios of 1.

28216#pg6> Instance selection is statistically indistinguishable from a fair draw among the scene’s instances, as the model locates a Highlander, then guesses which one was meant <ref:2608. p. 6
Where the paper says it

0 against observed 22, 22 and 26. Instance selection is statistically indistinguishable from a fair draw among the scene’s instances: the model locates a Highlander, then guesses which one was meant.

28216#pg6> The per-query evidence is blunter, with 79 of 116 instance queries having a top-1 box whose IoU with the correct object is exactly zero against 65. p. 6
Where the paper says it

In the V-JEPA arm 79 of 116 instance queries (68. 1%) have a top-1 box whose IoU with the correct object is exactly zero, against 65.

5% for DINOv3 and 63. p. 0
Where the paper says it

306 for a prompted Grounding DINO baseline under an identical scorer. Substituting DINOv3 for V-JEPA under a matched 576-token grid drops withincategory absence AUROC from 0.

8% for SigLIP2 <ref:2608. p. 0
Where the paper says it

The target beat every distractor in only 15. 9% of arXiv:2608.

28216#pg6> The discrimination gap between WALDO and baselines is not closed, as instance discrimination must come from the exemplar crop alone <ref:2608. p. 7
Where the paper says it

A model using the exemplar to discriminate among same-category candidates would close this gap; none does. 1’s bound applies: discrimination must come from the exemplar crop alone, and that crop is small, a median 87 × 108 px needing 3.

28216#pg6>

Scale and box tightness

The catalogue AP@50 decomposition shows that by object scale, aircraft reach AP@50 of 0.879, 0.877 and 0.874 while cars reach 0. p. 8

Where the paper says it

By object scale, aircraft reach AP@50 of 0. 874 while cars reach 0.

166, 0.150 and 0.210 <ref:2608. p. 0
Where the paper says it

The target beat every distractor in only 15. 9% of arXiv:2608.

28216#pg6> This gap is attributed to input resolution and object density rather than the representation <ref:2608. p. 8
Where the paper says it

The median car box measures 67 × 83 px, only 3 to 4 patches at 384 px input, so a one-patch error consumes a large fraction of it, and cars appear at up to ten instances per scene, with a median of 3. We attribute the gap to input resolution and object density, not to the representation.

28216#pg6> Splitting the same number by IoU strictness shows that V-JEPA falls from 0. p. 8
Where the paper says it

Splitting the same number by IoU strictness is where the backbones separate. From AP@50 to AP@75, V-JEPA falls from 0.

461 to 0.168, SigLIP2 from 0. p. 8
Where the paper says it

From AP@50 to AP@75, V-JEPA falls from 0. 168, SigLIP2 from 0.

481 to 0.063 and DINOv3 from 0. p. 8
Where the paper says it

168, SigLIP2 from 0. 063 and DINOv3 from 0.

428 to 0.021, reversing the AP@50 ordering <ref:2608. p. 8
Where the paper says it

063 and DINOv3 from 0. 021, reversing the AP@50 ordering.

28216#pg6> This supports a claim that world-model pretraining transfers here to localization precision and presence estimation, not to retrieval accuracy <ref:2608. p. 8
Where the paper says it

This supports a narrower claim than V-JEPA is the better backbone: SigLIP2 finds objects slightly more often, while V-JEPA places boxes more precisely and abstains more reliably. Since the catalogue Success@1 difference is inside the noise floor while the AP@75 and hard-absence differences are not, world-model pretraining transfers here to localization precision and presence estimation, not to retrieval accuracy.

28216#pg6> The shared box-tightness limit is plausibly architectural, constraining each patch’s predicted center to within ±1 patch width caps precision for objects a few patches across <ref:2608. p. 8
Where the paper says it

Since the catalogue Success@1 difference is inside the noise floor while the AP@75 and hard-absence differences are not, world-model pretraining transfers here to localization precision and presence estimation, not to retrieval accuracy. The shared box-tightness limit is plausibly architectural: constraining each patch’s predicted center to within ±1 patch width caps precision for objects a few patches across.

28216#pg6>

Conclusion

WALDO localizes a referenced instance from one exemplar and a description over frozen representations, treating absence as a trained output rather than a confidence threshold <ref:2608.28216#pg6> Catalogue AP@50 reaches 0. p. 8

Where the paper says it

6 Conclusion WALDO localizes a referenced instance from one exemplar and a description over frozen representations, treating absence as a trained output rather than a confidence threshold. Catalogue AP@50 reaches 0.

461 against 0. p. 8
Where the paper says it

Catalogue AP@50 reaches 0. 461 against 0.

306 for the strongest open-vocabulary baseline under one scorer <ref:2608. p. 8
Where the paper says it

461 against 0. 306 for the strongest open-vocabulary baseline under one scorer, and within-category absence AUROC reaches 0.

28216#pg6> Instance identity does not follow, as Success@1 of 0. p. 9
Where the paper says it

726 for the image-pretrained controls, the one measurement on which the three representations separate. Instance identity does not follow: Success@1 of 0.

190 sits at its category-chance floor <ref:2608. p. 9
Where the paper says it

Instance identity does not follow: Success@1 of 0. 190 sits at its category-chance floor, and 68.

28216#pg6> The model finds the category, then picks wrongly inside it <ref:2608. p. 9
Where the paper says it

1% of instance queries return a box overlapping nothing that counts. The model finds the category, then picks wrongly inside it.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Exemplar-conditioned detection with explicit presence determination allows AI systems to localize that object and state plainly that it is absent, enabling robotic fetch-and-place or retail auditing tasks by distinguishing between instance identity and category membership.

  2. WALDO can perform one-shot object detection using a single reference image and a short description, predicting both the object's localization and its presence in the scene simultaneously, which is distinct from open-vocabulary detectors that only answer category-level questions.

  3. The system uses a novel training strategy called episodemining to synthesize instance-level and hard/easy absence supervision from sparse annotations, which helps prevent shortcut learning where region size alone predicts the label.

  4. By fusing a patch-wise correlation score against the exemplar with FiLM conditioning from the joint exemplar-text embedding, WALDO is designed to jointly predict per-patch localization and scene-level presence, ensuring absence is a trained output rather than a byproduct of low confidence.

  5. The architecture incorporates a correlation channel that computes the cosine similarity between each patch and the exemplar vector in the backbone's own space before any learned projection can distort it, allowing the signal available to every downstream layer.

Abstract

Finding a particular object instance in a cluttered scene from a single reference image and saying when it is not there is typically given to large vision-language models. We ask whether the representations of a world model add something that the strongest self-supervised image features do not have. We introduce WALDO, a one-shot detector that finds a specific object instance in a cluttered real scene from a single reference image. Instance detection has usually needed per-object training. The alternative is a category-level open-vocabulary detector or vision-language model, which can find a mug but not this mug. We show that a lightweight conditioned head on frozen self-supervised features is enough to localize unseen instances without any per-object adaptation. We also find that image features and world-model features are complementary: DINOv3 and the video world model V-JEPA 2.1 succeed on different queries. There is one catch. Used the way video models are usually applied to still images, V-JEPA's dense features underperform. Reading them through the model's image pathway improves their localization. Combining the two backbones with a validation-tuned late fusion modestly improves DINOv3's localization of unseen instances without hurting its ability to tell whether the object is in the scene. A strong vision-language model and an image-guided open-vocabulary detector almost never localize the specific instance, and take far longer per query. Our fusion with world-model features strengthens localization from image features alone.

Sources

Related papers