STORM: Segment, Track, and Object Re-Localization from a Single Image

arXiv:2511.09771 · cs.CV · Submitted 2025-11-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "STORM: Segment, Track, and Object Re-Localization from a Single Image".

Jane: The gist: STORM, a unified framework for reference-conditioned 6D pose tracking,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper called "STORM: Segment, Track, and Object Re-Localization from a Single Image". It's tackling the big problem of 6D pose estimation and tracking for physical AI systems <ref:2511.09771#pg1,6D pose estimation and tracking>.

Jane: Exactly. The main idea is that real-world deployment of these systems is really brittle and takes too much manual work to do right. They propose STORM as a unified framework that lets you track an object using just one reference image, with way less manual input and it's supposed to be much more robust than what we have now.

Lu: It claims that this framework combines two main things: Hierarchical Spatial Fusion Attention, or HSFA, which is like a task-driven architecture for mixing reference and query views, and a tracking verifier that uses BCE training to check for drift and automatically re-initializes the tracking.

Meng: That sounds complex. So what does it actually do on the ground? Does it solve the problem of objects disappearing behind something else or moving too fast?

Tom: It directly addresses those issues because it formulates 6D pose estimation as a joint problem involving reference-query feature fusion and tracking-loss verification, which is a couple of stages <ref:2511.09771#pg1>. The first stage is the Segmenting Object Module, or SOM, which does coarse-to-fine feature alignment to recover object masks and three dee geometry from those reference images <ref:2511.09771#pg1>.

Jane: And then the second stage, the Tracking Object Module, or TOM, acts like a learned compatibility verifier that watches how well the tracking is doing and triggers a re-initialization when drift keeps happening.

Lu: The SOM uses Hierarchical Spatial Fusion Attention to learn a mapping function F that aligns your query image with one or more reference views. This involves aggregating reference evidence, doing query-reference cross-attention, and optionally using vision-language semantic conditioning from Vision-Language Models to sort out things that look similar but are different instances.

Meng: So the attention mechanism is key here. How does it use that VLM part if we don't want to add another model running constantly?

Lu: They feed the reference image and a generic prompt into a VLM, which makes an object descriptor, and then CLIP encodes that into a semantic embedding. This embedding modulates the visual tokens using something called AdaLN or FiLM style conditioning to influence how the alignment happens.

Jane: And this alignment comes out implicitly through mask supervision because the cross-attention layer computes attention weights that are interpreted as an alignment matrix, which then propagates a reference objectness prior through that matrix to get the query mask.

Paper summary: Tom: That’s a lot of machinery for one step, but it seems they are trying to build something that is much more flexible than relying on traditional template matching or per-object adaptation.

Jane: And they're not just focusing on segmentation; they're also using SAMthree dee to generate a canonical mesh from the reference images, which then serves as a rigid geometric anchor for lifting those 2D semantics into three dee and enforcing geometric consistency as a soft latent constraint <ref:2511.09771#pg1>.

Meng: So you’ve got the semantic understanding from the VLM side and the hard geometry from SAMthree dee side working together to guide the pose estimation part?

Lu: Right. The framework explicitly separates inherited frozen components, like DINOv3 for dense visual features, and trainable modules. For example, SAMthree dee is a frozen component that generates a canonical mesh Pref which is then used by the downstream pose estimator to map predicted masks onto metric three dee coordinates <ref:2511.09771#pg1>.

Tom: Now we get into the tracking part, which they call Tracking-Loss Verification. The TOM is trained as a binary compatibility classifier over current observations and a memory pool of successful tracks.

Jane: At inference time, they convert that logit from the classifier into an energy-like continuous score which smooths it temporally and triggers re-localization only after persistent evidence of drift.

Lu: They define this energy-like score E(xt,M) as the negative compatibility logit, E(xt,M) is defined as - g theta (xt,M). A high compatibility logit usually means a stable match resulting in low energy, and conversely drift yields low compatibility and high energy.

Meng: So how do they decide when it's actually time to re-localize? Is it just when the score hits some arbitrary number?

Jane: No, they use a temporal smoothing technique for that score E(xt,M) into an EMA smoothed score E t equals alphaE t-one plus (one minus alpha)Et <ref:2511.09771#pg1>. Then tracking loss is declared if this smoothed energy exceeds a calibrated threshold for L consecutive frames.

Tom: They use L equals three for all their experiments, and the threshold tau is calibrated once on a held-out validation split as the ninety-fifth percentile of compatible-pair scores to make sure there's only a five percent chance of drift when you don't have drift <ref:2511.09771#pg1>.

Lu: This persistence constraint prevents those oscillations between tracked and lost states that you see in other trackers, which is really important for stability.

Jane: So what does this mean for someone who just listens to the show? It means the system doesn't just guess a pose if it gets confused; it actively checks its own work against its memory of what worked before.

Paper summary: Tom: And when you look at the experimental results, they tested STORM on LM-O and YCB-Video. They found that STORM improves annotation-free pose tracking accuracy over strong baselines and reliably recovers from severe occlusions and rapid viewpoint changes with minimal overhead.

Meng: They also showed that the Tracking Object Module provides more reliable tracking-loss detection than fixed metric learning baselines, and the attention mechanisms in TOM significantly separate failure states, increasing accuracy from eighty-seven point five five percent to ninety-eight point thirty six percent with only one point three milliseconds additional latency.

Jane: That latency is small enough that it doesn't really slow down a real-time application much, which is a big deal for practical engineering.

Lu: They also mention that the framework operates in an annotation-free setting, meaning no manual masks or object-specific fine-tuning are done at test time.

Tom: The authors point out some limitations on the STORM framework. They say it inherits boundaries from reference-conditioned perception, like needing a single reference image to operate and depending on the coordinate consistency of the SAMthree dee reconstructions.

Jane: And they also flag that the full pipeline still depends on whatever frozen downstream FoundationPose registration and tracking backend you choose to use for the final pose estimation step.

Meng: So, even with STORM, you still have some other parts of a pipeline that need to be perfectly calibrated for it to work well in practice?

Lu: Exactly. Future work they plan involves making STORM adaptive, maybe online calibration of the TOM's failure threshold, and stronger reconstruction for objects that are transparent or reflective.

Tom: It sounds like this paper really pushes the idea that segmentation, geometry, and failure awareness should be considered together when you're doing reference-conditioned pose tracking.

Jane: It’s a solid framework because it moves away from having to manually mask everything or adapt models for every single object in a scene.

Lu: The zero-shot setting they use means the system is annotation-free and adaptation-free inference without test time masks or object specific updates, which is a big deal for deployment.

Meng: For me, the practical impact is that if we can deploy this with minimal manual labeling, we can get physical AI systems into logistics and manufacturing environments much faster than before.

Tom: It sounds like STORM provides a more reliable and safer way to handle dynamic scenes like warehouses or factories by actively detecting when it's lost its footing.

Jane: It’s about moving from just predicting a pose to building a system that knows when it should stop trying and start over reliably in complex, changing environments.

Conclusion: Tom: So, we’re wrapping up STORM now by looking at what this paper actually says about its title and who wrote it.

Jane: It’s called "STORM: Segment, Track, and Object Re-Localization from a Single Image." That sounds like a lot to do in one go.

Lu: It is. The authors are trying to solve the problem of tracking physical objects using just one reference picture, without needing tons of manual labeling for every single object they want to follow.

Meng: So, the core idea is that you can get a 6D pose—that's position and orientation in three dimensions—from just one picture <ref:2511.09771#pg1>. That sounds like it could cut down on how much data you have to collect for training models.

Lalam: From my side, I see this as really important because if we can do this annotation-free, it means physical AI systems can be deployed way faster in real life without needing massive datasets for every single thing they interact with.

Tom: Exactly. It moves us away from having to manually create masks or fine-tune models for every single object we want an AI to track in a warehouse or on a factory floor.

Jane: It really boils down to taking the hard problem of 6D tracking and making it less labor-intensive by using this reference image as a starting point <ref:2511.09771#pg1>.

Lu: They achieve that through this whole pipeline, combining attention for feature mixing and a built-in check to see if the tracking is actually drifting off course.

Meng: That drift detection part, the Tracking Object Module—that’s what I find most practical. It means if the AI gets confused by occlusion or fast motion, it doesn't just keep guessing; it knows when its memory has failed and can trigger a reset.

Tom: Right. And that reset mechanism is what makes it robust in messy, real-world settings where things move quickly or get hidden behind other objects.

Jane: It suggests we don’t always need perfect, labeled data to build reliable tracking systems for robots or autonomous agents.

Lu: It shows that you can combine geometry from three dee reconstruction with semantic understanding from models like Vision-Language Models in a unified way to guide the pose estimation.

Meng: So, the numbers show it does better than strong baselines on tests like LM-O and YCB-Video, and it does this without needing test time masks or object specific fine-tuning.

Tom: It’s about showing that you can get high accuracy in annotation-free settings while keeping the system relatively light in terms of extra processing time.

Jane: It’s a big step toward making these physical AI systems much more practical for everyday use in dynamic environments.

Lu: And they pointed out that it still relies on the quality of three dee geometry reconstruction from those reference images, which is a real technical dependency we have to keep an eye on.

Meng: So, while the tracking part is smart, you still need a solid geometric anchor from the reference image itself for it to work well.

Tom: Right. It’s not magic; it’s a combination of attention mechanisms and geometric constraints that works together in this closed-loop system.

Jane: It's an interesting direction because it really pushes the idea that we should consider segmentation, three dee geometry, and failure awareness all at the same time when tracking something from a reference.

Lu: And they are already looking at making it adaptive, which means it could potentially adjust its own failure detection thresholds online based on what it’s seeing.

Meng: That would make it even more useful because you wouldn't have to manually set those drift thresholds for every new type of object you introduce into your system.

Tom: So, we’ve seen the mechanics and the results, and now we understand that STORM is a unified approach to tackling this problem from a single image.

Jane: It really changes how we think about setting up systems that need to perceive and follow things in real-time without constant manual intervention.

Department of Computer Science, Technical University of Darmstadt · Hessian Center for Artificial Intelligence (hessian.AI) · Google Intrinsic AI Research

cs.CV

Submitted: 2025-11-12

Updated: 2026-05-13

Comments: 21 pages. Accepted at the 43rd International Conference on Machine Learning (ICML 2026); camera-ready version

Journal ref: Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306:23759-23779, 2026

Code: https://github.com/YuDeng321/STORM

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: The gist: STORM, a unified framework for reference-conditioned 6D pose tracking, improves annotation-free pose tracking accuracy over strong baselines by combining Hierarchical Spatial Fusion

Key concepts

Hierarchical Spatial Fusion Attention (HSFA): This is an attention architecture that fuses information from a single reference image with the current query image. It works by building a coherent canonical representation of the reference, then using cross-attention to align the query features with this reference. Optionally, it incorporates vision-language semantic conditioning to resolve ambiguities in object instances.
Tracking Object Module (TOM): The TOM acts as a learned verifier that checks if the current observation is compatible with previously tracked objects. It is trained using binary classification and outputs a compatibility logit, which is converted into an energy-like score. A high score indicates stable tracking, while a low score signals drift, triggering automatic re-initialization.
Segmenting Object Module (SOM): The SOM performs the initial alignment task. It uses HSFA to map the query image features to the reference views, effectively recovering object masks and 3D geometry from the single reference input. This process ensures that semantic understanding and geometric consistency are established before tracking continues.
Energy-like Score E(xt,M): This is a continuous score derived from the TOM's compatibility logit, defined as the negative logit of compatibility. It serves as an energy measure where low energy corresponds to a stable match between the current observation and tracked memory. Tracking loss is declared when this smoothed score exceeds a calibrated threshold for several consecutive frames.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Adaptation — 1× in this paper · explained in 1 episode
Core — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes
Unified Framework — 1× in this paper · explained in 5 episodes
Verifier — 1× in this paper · explained in 2 episodes

Summary

The gist: STORM, a unified framework for reference-conditioned 6D pose tracking, improves annotation-free pose tracking accuracy over strong baselines by combining Hierarchical Spatial Fusion Attention for reference-query fusion and a BCE-trained Tracking Object Module to detect drift and trigger automatic re-initialization

How it works

STORM addresses the distributional shift between canonical reference data and dynamic observational queries by formulating the task as a joint problem of reference-query feature fusion and tracking-loss verification The framework operates in two coupled stages, where the Segmenting Object Module (SOM) performs coarse-to-fine feature alignment to recover object masks and 3D geometry from reference image(s), and the Tracking Object Module (TOM) functions as a learned compatibility verifier that monitors tracking fidelity and triggers re-initialization when drift persists.

The Segmenting Object Module (SOM) employs Hierarchical Spatial Fusion Attention (HSFA) to learn a mapping function F that aligns a query image Iq with one or more reference views Iref, which is achieved through task-driven composition of single-/multiview reference aggregation, query-reference cross-attention, and optional VLM-derived semantic conditioning. This process involves Latent Reference Construction via self-attention to build a coherent canonical representation Zref, followed by Query-Reference Alignment where the query features Zquery attend to Zref via cross-attention. Furthermore, semantic prior injection is achieved by feeding the reference Iref and a generic prompt p into a VLM to generate an object descriptor T, which is then encoded by CLIP into a semantic embedding et that modulates the normalized visual tokens via an AdaLN/FiLM-style conditioning layer. The alignment emerges implicitly through mask supervision, as the cross-attention layer computes attention weights interpreted as an alignment matrix W, which propagates a reference objectness prior through W to obtain the query mask.

Tracking-Loss Verification

The Tracking Object Module (TOM) is trained as a binary compatibility classifier over current observations and a memory pool of successful tracks, and at inference time, its logit is converted into an energy-like continuous score that detects drift. This energy-like score E(xt,M) is defined as the negative compatibility logit, E(xt,M) ≜ − gθ(xt,M), where a high compatibility logit generally implies a stable match resulting in low energy, and conversely drift yields low compatibility and high energy.

The decision rule for tracking loss involves temporal smoothing of the energy-like score E(xt,M) into an EMA-smoothed score E˜t = αE˜t−1 + (1 − α)Et, where α ∈ [0, 1), and tracking loss is declared if the smoothed energy exceeds a calibrated threshold for L consecutive frames. This persistence constraint prevents oscillations between tracked and lost states, with L=3 used for all experiments. The threshold τ is calibrated once on a held-out validation split as the 95th percentile of compatible-pair scores to satisfy Pr(E > τ Din) ≤ 0.05, while maximizing recall on Dout.

Component Roles and Geometry

STORM explicitly separates inherited and trainable components, utilizing frozen foundation components such as DINOv3 for dense visual features, SAM3D to generate a canonical mesh from reference images, and FoundationPose for downstream pose estimation. The SAM3D reconstruction serves as a rigid geometric anchor that supports lifting 2D semantics into 3D and enforcing geometric consistency as a soft latent constraint. Specifically, the framework leverages SAM3D to generate a canonical mesh Pref from the reference image(s), which is then used by the downstream pose estimator to map predicted masks onto metric 3D coordinates.

Experimental Evaluation

Experiments evaluate STORM under an annotation-free inference setting, where no manual masks or object-specific fine-tuning are performed at test time. The framework is tested on challenging benchmarks like LM-O and YCBVideo, showing that it improves annotation-free pose tracking accuracy over strong baselines and recovers reliably from severe occlusions and rapid viewpoint changes with minimal overhead. Key results show that STORM achieves state-of-the-art performance on LM-O and YCBVideo datasets, substantially narrowing the gap to the manual-mask upper bound. The Tracking Object Module (TOM) is shown to provide more reliable tracking-loss detection than fixed metric learning baselines, and attention mechanisms in TOM significantly improve failure-state separation, increasing accuracy from 87.55% to 98.36% with only 1.3ms additional latency.

Limitations and Future Directions

STORM inherits several boundaries from reference-conditioned perception and tracking, including the minimum input of a single reference image for operation. The full pipeline also depends on the fidelity and coordinate consistency of SAM3D reconstructions and on the frozen downstream FoundationPose registration/tracking backend. Future work will focus on making STORM adaptive, including online calibration of TOM’s failure threshold, stronger reconstruction for transparent or reflective objects, and evaluation on category-disjoint object splits. The zero-shot setting denotes annotation-free and adaptation-free inference without test-time masks or object-specific updates.

The work aims to advance computer vision by providing an annotation-free inference framework for 6D pose tracking, significantly reducing the labor required to deploy physical AI systems. STORM introduces Hierarchical Spatial Fusion Attention and a BCE-trained Tracking Object Module to bridge semantic understanding and geometric consistency, allowing the system to segment objects under occlusion and trigger re-initialization when tracking loss is detected. This closed-loop design is particularly useful in dynamic scenes where occlusion, fast motion, and viewpoint changes are common.

The final SOM release is trained once offline from BOP training splits, and the system runs FoundationPose tracking and TOM drift detection per frame in the steady-state tracking loop. In this steady-state operation, STORM runs FoundationPose tracking and TOM drift detection per frame; TOM adds 9.10ms and the combined loop is approximately 25–33 FPS in our measurements. This supports a key insight of our framework: once the observation becomes incompatible with the tracked-object memory, drift is often irrecoverable for standard temporal trackers, especially at longer sampling intervals in practice. Overall, STORM shows that segmentation, geometry, and failure awareness should be considered jointly in reference-conditioned pose tracking.

The SAM3D analysis further suggests that reconstructed geometry can be sufficient for practical pose registration when coordinate consistency is handled carefully. The CAD-based alignment is only used to report under the BOP coordinate convention; deployment requires a self-consistent reconstructed object frame rather than curated benchmark CAD assets. This work aims to advance computer vision by providing an annotation-free inference framework for 6D pose tracking, significantly reducing the labor required to deploy physical AI systems. By integrating explicit tracking-loss detection, STORM improves the reliability and safety of autonomous agents in dynamic environments like logistics and manufacturing. While these advancements facilitate broader robotic accessibility, we do not foresee specific negative societal consequences warranting individual highlighting here.

The work was supported by public research grants from the European Union, the Federal Ministry of Research, Technology and Space within the JUPITER AI Factory, and the Federal Ministry of Food and Agriculture within the FarmerSpaceAI project. This work aims to advance computer vision by providing an annotation-free inference framework for 6D pose tracking, significantly reducing the labor required to deploy physical AI systems. By integrating explicit tracking-loss detection, STORM improves the reliability and safety of autonomous agents in dynamic environments like logistics and manufacturing. While these advancements facilitate broader robotic accessibility, we do not foresee specific negative societal consequences warranting individual highlighting here.

The work was supported by public research grants from the European Union (Grant No. 57100467), the Federal Ministry of Research, Technology and Space within the JUPITER AI Factory, and the Federal Ministry of Food and Agriculture within the FarmerSpaceAI project (Grant No. 28DE401E23). This work aims to advance computer vision by providing an annotation-free inference framework for 6D pose tracking, significantly reducing the labor required to deploy physical AI systems. By integrating explicit tracking-loss detection, STORM improves the reliability and safety of autonomous agents in dynamic environments like logistics and manufacturing. While these advancements facilitate broader robotic accessibility, we do not foresee specific negative societal consequences warranting individual highlighting here.

Improvements for AI systems

Improved 6D pose tracking robustness through explicit failure detection: The system can now distinguish between a difficult but recoverable frame and a complete tracking failure by using a BCE-trained tracking verifier whose continuous compatibility logit is used as an energy-like score to detect drift and trigger automatic re-initialization.
Enhanced zero-shot segmentation accuracy via multimodal conditioning: The system can resolve instance ambiguities by optionally using vision-language semantic conditioning injected via a VLM descriptor, allowing it to perform reference-conditioned segmentation even when visual evidence is insufficient.
Improved geometric consistency through robust 3D model utilization: The system can leverage reconstructed geometry by using the SAM3D output as a rigid geometric anchor, ensuring that downstream pose estimation maps the SOM-predicted masks onto metric 3D coordinates even when relying on reconstructions rather than curated CAD models.
Increased tracking reliability under dynamic conditions: The system can maintain steady-state tracking loops in real-time by using a temporal policy that suppresses transient noise, as demonstrated by the fact that a per-frame binary decision is brittle in a tracking loop because a single noisy frame can trigger re-localization.

Abstract

Accurate 6D pose estimation and tracking are core capabilities for physical AI systems, yet real-world deployment remains brittle and labor-intensive. Many pipelines rely on CAD models, manual masking, or per-object adaptation, and still fail under occlusion or fast motion without a principled way to recognize failure. We propose STORM, a unified framework for reference-conditioned 6D tracking that can operate from a single reference image, with minimal manual input and improved robustness. STORM combines: (i) Hierarchical Spatial Fusion Attention (HSFA), a task-driven reference-query fusion architecture that supports both single-reference and multi-reference conditioning and can optionally use vision-language semantic conditioning to resolve instance ambiguities; and (ii) a BCE-trained tracking verifier whose continuous compatibility logit is used as an energy-like score to detect drift and trigger automatic re-initialization. Experiments on LM-O and YCB-Video show that STORM improves annotation-free pose tracking accuracy over strong baselines and recovers reliably from severe occlusions and rapid viewpoint changes with minimal overhead.