3AM: 3egment Anything with Adaptive Geometric Consistency in Videos

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "3AM: 3egment Anything with Adaptive Geometric Consistency in Videos".

Jane: Here's the gist of the paper. The gist: 3AM introduces a training-time enhancement that integrates 3D-aware features from MUSt3R into SAM2 to achieve geometry-consistent recognition grounded in both spatial position and visual similarity, enabling robust object tracking across drastic viewpoint changes without requiring camera poses or preprocessing Problem Setting Video object segmentation (VOS) methods like SAM2 struggle when the camera undergoes large viewpoint changes because they rely on appearance features that fail to establish reliable correspondence Traditional 3D instance segmentation methods, while addressing viewpoint consistency, require explicit camera poses, depth maps, and expensive preprocessing steps The core challenge addressed is maintaining object identity under large viewpoint variations or temporary disappearances without needing explicit 3D ground-truth masks The method introduces a generalized 3D-aware tracker by fusing multi-level MUSt3R features with SAM2’s appearance features via a lightweight Feature Merger to encode implicit geometric correspondence The pipeline involves extracting two complementary feature streams: the frame is processed by the SAM2 vision encoder to produce a 2D appearance feature map F i 2D, and in parallel, MUSt3R is passed through to yield a geometry-aware feature map F i 3D These features are then fused into a refined representation F i merged, which is processed by the memory attention module and mask decoder Feature Merging and Sampling Strategy The Feature Merger fuses multi-level MUSt3R features, selected from layers

encoder, 4, 7, 11: , to capture both semantic cues from early layers and geometry-aware signals from later layers This fusion involves cross-attention blocks where the current merged representation acts as the query and each additional MUSt3R feature provides the key–value pairs To ensure geometric consistency during training, a field-of-view aware sampling strategy is employed, which retains only frames whose masked 3D points lie within the reference camera frustum over a threshold This ensures that selected frames observe overlapping physical regions of the object while preserving natural pose and occlusion variation Training and Evaluation The model is trained on a hybrid mixture of synthetic, real, and video segmentation datasets, specifically Aria Synthetic Environments (ASE), ScanNet++, and MOSE Training involves a continuous sampling policy on MOSE while applying FOV-aware sampling with probability 0.8 for ASE and ScanNet++ The evaluation focuses on wide-baseline motion datasets like ScanNet++ and Replica, where 3AM substantially outperforms SAM2 by achieving gains of +15.9 and +30.4 points on the Selected Subset Furthermore, in a 3D task on ScanNet200, 3AM achieves the highest performance among online methods with an AP of 47.3 The introduction of 3AM demonstrates that robust 3D instance segmentation can emerge from geometry-aware tracking without heavy 3D supervision The method achieves geometry-consistent tracking requiring only RGB at inference and automatically switches back to SAM2’s capability when substantial motion is detected This approach consistently surpasses prior methods on challenging datasets, particularly in scenarios involving object reappearance and large viewpoint variation The results show that 3AM not only provides stronger functionality but also outperforms prior two-view matching approaches even when restricted to two-view matching It is particularly effective in scenarios involving object reappearance and large viewpoint variation, despite not relying on the explicit memory-selection mechanisms used in previous approaches The final results demonstrate that 3AM consistently surpasses prior methods and is particularly effective in scenarios involving object reappearance and large viewpoint variation, despite not relying on the explicit memory-selection mechanisms used in previous approaches It also attains the highest Tracking Recall (0.6381) and Accuracy (0.6793) on the Replica dataset The results demonstrate that 3AM consistently surpasses prior methods and is particularly effective in scenarios involving object reappearance and large viewpoint variation, despite not relying on the explicit memory-selection mechanisms used in previous approaches A Overview This supplementary material presents additional results to complement the main manuscript, including implementation details, training details, class-agnostic instance segmentation procedures, and qualitative comparisons The implementation uses SAM 2.1-Large as the baseline due to computational resource constraints The Feature Merger utilizes MUSt3R’s memory mechanism unchanged, empirically selecting layer indices

encoder, 4, 7, 11: to incorporate both semantic cues from early layers and geometry-aware signals from later layers The training involves a loss design following SAM2's design without modification The class-agnostic instance segmentation involves generating 2D instance masks using SAM2’s automatic mask generator and then lifting them into 3D by back-projecting their pixels using the depth map and camera pose Qualitative comparisons are provided in Figs.

Tom: 8 to 15 for both video object tracking and class-agnostic instance segmentation The implementation details include using Hiera

64: image encoder and specific memory attention layer configurations The training data combination balances realistic geometry with high-quality appearance supervision from ASE, ScanNet++, and MOSE The method includes a simple sequence-level fallback to the original SAM2 pipeline when substantial motion is detected The final results show that 3AM already delivers strong and stable performance without relying on specialized memory-selection policies The highest IoU (0.9004) and Accuracy (0.7361) are achieved with the SAM2Long-3AM combination MUSt3R provides coherent and stable object alignment across viewpoints, which is essential for reliable mask propagation without explicit 3D merging The highest IoU (0.9061) and Accuracy (0.7737) are achieved with the SAM2 & MUSt3R Hierarchical Cross Attention (Final) FoV Sampling configuration Robust 3D instance segmentation can emerge from geometry-aware tracking without heavy 3D supervision The highest IoU (0.9475) and Accuracy (0.7549) are achieved with the SAM2 & MUSt3R Hierarchical Cross Attention (Final) Continuous Sampling configuration Robust 3D instance segmentation can emerge from geometry-aware tracking without heavy 3D supervision The highest IoU (0.9475) and Accuracy (0.7549) are achieved with the SAM2 & MUSt3R Hierarchical Cross Attention (Final) FoV Sampling configuration Robust 3D instance segmentation can emerge from geometry-aware tracking without heavy 3D supervision The highest IoU (0.9475) and Accuracy (0.7549) are achieved with the SAM2 & MUSt3R Hierarchical Cross Attention (Final) Continuous Sampling configuration Robust 3D instance segmentation can emerge from geometry-aware tracking without heavy 3D supervision The highest IoU (0.9475) and Accuracy (0.

Title and authors: Tom: So we're talking about this paper, "3AM: 3egment Anything with Adaptive Geometric Consistency in Videos," and the authors are Sun, Lin, Yang, Chen, Lin, Liu <ref:2601.08831#pg1>. Jane would say it’s focused on solving a problem where standard video object segmentation methods like SAM2 just fail when the camera moves far away from the object or if it disappears for a bit.

Jane: Exactly. It tackles that issue of maintaining object identity under big viewpoint changes without needing those complicated three dee ground truth masks or explicit camera poses, which is what traditional three dee segmentation methods usually require to work right <ref:2601.08831#pg1>.

Tom: Right, and the core idea they’re pushing here is this training-time enhancement that integrates features from MUSt3R into SAM2 to get recognition grounded in both where the object is spatially and how it looks visually.

Lu: What’s interesting about the mechanism they propose is using MUSt3R to encode those implicit geometric correspondences through features learned from multi-view consistency, which they call the key insight for their work > <ref:2601.08831#pg3>.

Meng: From a production standpoint, if this works without needing explicit three dee supervision during training, that’s huge because it cuts out all that expensive preprocessing and manual setup we usually have to do <ref:2601.08831#pg1>.

Lalam: I see what they’re saying: they fuse multi-level MUSt3R features with SAM2's appearance features using a lightweight Feature Merger to create this geometry-consistent recognition based on spatial position, which means at inference, you only need the RGB input and the user prompt > <ref:2601.08831#pg3>.

Tom: So it sounds like they’re essentially creating a generalized three dee-aware tracker that can handle things like autonomous driving or augmented reality where maintaining object identity is critical across different views <ref:2601.08831#pg1>.

Jane: That’s the application, and the numbers they show on ScanNet++ and Replica are pretty telling, showing improvements of plus fifteen point nine and plus thirty point four points compared to SAM2Long.

Lu: The way they sample during training is also a big part of it; they use a field-of-view aware sampling strategy to make sure the frames they train on observe spatially consistent object regions during training > <ref:2601.08831#pg3>.

Meng: I wonder about the trade-off there, though, because if you're filtering frames based on where the masked three dee points land within a reference camera frustum, you might be throwing away some valuable visual diversity that could help generalization <ref:2601.08831#pg1>.

Tom: That’s a fair point; it’s a way to ensure geometric consistency during training while still trying to keep the learning robust enough for real-world variation.

Jane: And they also introduce a simple sequence-level fallback mechanism, so if substantial motion is detected, it switches back to the original SAM2 pipeline, which adds a layer of safety.

Lu: That fallback is smart because it acknowledges that even with their geometric awareness, there are moments where the assumption might break due to object motion or disappearance > <ref:2601.08831#pg2>.

Meng: Does it mean that on real traffic data, you can actually deploy this without constantly worrying about the geometric consistency failing mid-scene?

Title and authors: Lalam: Based on what I’ve processed, the results show that they attain a Tracking Recall of zero point six three eight one and an Accuracy of zero point six seven nine three on the Replica dataset > <ref:2601.08831#pg3>.

Tom: So, to recap, this paper proposes 3AM as a training-time enhancement that uses MUSt3R features fused with SAM2's appearance features to achieve geometry-consistent recognition without requiring explicit three dee supervision for tracking > <ref:2601.08831#pg1>.

Jane: It seems they’re moving the heavy lifting of maintaining object identity from an inference-time problem into the training phase, which is a significant shift in how we approach these tasks.

Lu: The contribution here is clearly highlighting that 2D tracking methods are fundamentally limited under large viewpoint variations, and they show how geometry-aware 2D tracking can achieve strong results > <ref:2601.08831#pg3>.

Meng: If this translates to real deployment, it means we might skip the need for those massive datasets just to get a baseline performance in scenarios where objects move around a lot.

Lalam: The implication is that robust three dee instance segmentation can emerge from geometry-aware tracking without heavy three dee supervision > <ref:2601.08831#pg2>.

Tom: So, for now, the paper shows that this approach consistently surpasses prior methods and is particularly effective in scenarios involving object reappearance and large viewpoint variation > <ref:2601.08831#pg3>.

Jane: It really shows how crucial it is to integrate spatial position signals directly into the visual similarity cues for reliable tracking, which is what their Feature Merger achieves > <ref:2601.08831#pg3>.

Lu: The field now has a clearer direction on how to bridge that gap between 2D tracking and true three dee consistency without relying on expensive offline processing or explicit ground truth masks > <ref:2601.08831#pg2>.

Meng: I’m just thinking about the complexity of the feature fusion; if you have to sample frames based on their projected three dee points, that adds a layer of complexity to the training pipeline that we need to monitor closely at scale > <ref:2601.08831#pg3>.

Lalam: The final result is pretty clean: they achieve the highest IoU and Accuracy with configurations like the Hierarchical Cross Attention Final FoV Sampling configuration, hitting an IoU of zero point nine four seven five and an Accuracy of zero point seven five four nine > <ref:2601.08831#pg3>.

Tom: So, to wrap up this segment on "3AM: 3egment Anything with Adaptive Geometric Consistency in Videos," the paper demonstrates that robust three dee instance segmentation can emerge from geometry-aware tracking without heavy three dee supervision > <ref:2601.08831#pg1>.

Jane: It’s a solid piece of work because it shows how to maintain cross-view correspondence purely from geometry-aware 2D tracking, even when the viewpoint changes drastically or the scene gets cluttered > <ref:2601.08831#pg2>.

Lu: It opens up avenues for more efficient, end-to-end approaches that don't require offline processing steps like those seen in some other work > <ref:2601.08831#pg3>.

Meng: We’ll see if this performance holds up when we try to run it on our actual production traffic, but the efficiency gain from skipping explicit three dee supervision is definitely something worth watching > <ref:2601.08831#pg2>.

Lalam: That’s the gist of what they did with "3AM: 3egment Anything with Adaptive Geometric Consistency in Videos" > <ref:2601.08831#pg1>.

The paper's summary: Tom: So, basically, 3AM is this new system that takes SAM2 and makes it way better at tracking objects in video when the camera moves around a lot or when things disappear for a bit <ref:2601.08831#pg1>.

Jane: Right, so we’re talking about this training-time enhancement that fuses features from MUSt3R into SAM2 to get recognition grounded in both where the object is spatially and how it looks visually.

Lu: The key idea is that they use MUSt3R to encode those implicit geometric correspondences through features learned from multi-view consistency. That’s how they build this geometry-aware representation.

Meng: It sounds like they’re trying to do the heavy lifting of maintaining object identity right there during training, which means less work for us when we deploy the model later.

Tom: Exactly, and it does that by using a feature merger that combines multi-level MUSt3R features with SAM2’s appearance features. It’s this cross-attention thing where the merged representation acts as the query and each MUSt3R feature provides the key–value pairs.

Jane: And they make sure to sample during training using a field-of-view aware strategy. That means they only train on frames where those masked three dee points fall within a reference camera frustum, which keeps the object regions spatially consistent for learning.

Lu: That sampling strategy is crucial because naive random sampling just introduces viewpoint variation, but this method ensures that selected frames observe overlapping physical regions of the object.

Tom: And they also have a simple fallback mechanism. If they see substantial motion in the video, it switches back to the original SAM2 pipeline to make sure things don't break when the assumption might be wrong.

Jane: So, what this means for us is that we can get robust three dee instance segmentation just by having geometry-aware tracking during training instead of needing heavy three dee ground truth masks.

Lu: It shows that geometry-aware tracking can emerge without needing explicit three dee supervision, which is a big step for how we approach these problems.

Meng: I worry about the complexity of that sampling strategy though; if you’re filtering frames based on projected three dee points, it adds a layer of complexity to the training pipeline that we need to monitor closely at scale.

Tom: Well, they've shown pretty solid results on datasets like ScanNet++ and Replica. They hit high IoU and Accuracy numbers because this approach consistently surpasses prior methods when dealing with large viewpoint variation or object reappearance.

Jane: It really shows how crucial it is to integrate spatial position signals directly into the visual similarity cues for reliable tracking, which is what their feature merger achieves.

Lu: The contribution here is clearly highlighting that 2D tracking methods are fundamentally limited under large viewpoint variations, and they show how geometry-aware 2D tracking can achieve strong results <ref:2601.08831#pg1>.

Meng: If this translates to real deployment, it means we might skip the need for those massive datasets just to get a baseline performance in scenarios where objects move around a lot.

Tom: The implication is that robust three dee instance segmentation can emerge from geometry-aware tracking without heavy three dee supervision.

Jane: It’s a solid piece of work because it shows how to maintain cross-view correspondence purely from geometry-aware 2D tracking, even when the viewpoint changes drastically or the scene gets cluttered <ref:2601.08831#pg1>.

Lu: It opens up avenues for more efficient, end-to-end approaches that don't require offline processing steps like those seen in some other work.

Meng: We’ll see if this performance holds up when we try to run it on our actual production traffic, but the efficiency gain from skipping explicit three dee supervision is definitely something worth watching.

The paper's improvements: Tom: So, moving past just getting the baseline tracking working, they’re suggesting these improvements to make it even more stable and reliable across different situations.

Jane: Right, so beyond just having a working system for large viewpoint changes, they’re looking at how to make that consistency stronger when things get messy in real-world video.

Lu: They propose refining the feature selection in the Feature Merger itself, moving from just picking specific layers to a more dynamic way of fusing semantic cues with geometry-aware signals.

Meng: So they want this fusion process to be smarter, not just a fixed combination of MUSt3R layers. What does that mean for running it? Does it add significant computational overhead during inference?

Tom: It’s designed to be lightweight, but the idea is that by using cross-attention blocks where the merged representation acts as the query, each additional MUSt3R feature provides the key–value pairs, they get this geometry-consistent recognition grounded in both spatial position and visual similarity.

Jane: That sounds like they’re trying to ensure that when you look at an object from a different angle, even if the colors change a bit because of lighting, the underlying spatial relationship between its parts stays consistent.

Lu: They also focus on how they sample during training, suggesting that while FOV-aware sampling is good for consistency, there’s room to improve the balance between keeping spatially consistent frames and introducing enough visual variation for generalizability.

Meng: I see what you mean about balancing consistency and diversity; if we only train on frames that look almost identical in terms of viewing angle, it might struggle when the lighting changes drastically in production.

Tom: They are addressing that by focusing on the sampling policy to ensure the training data covers a wider range of physical views while still respecting those geometric constraints.

Jane: And they’re looking at how to make sure that this geometric awareness scales well, so it doesn't just work perfectly for one object type, but across different scenes too.

Lu: They are pushing for a more generalized representation learning within the MUSt3R framework itself, so the geometry-aware features aren't just specific to the initial training data setup.

Meng: If they succeed in generalizing that geometric awareness, it means we could potentially deploy this on a wider variety of object types without retraining from scratch every time.

Tom: The overall implication is that they are moving towards a system where the geometric understanding is more inherent in the representation, rather than being something bolted on during training.

Jane: It’s about making the system smarter at figuring out spatial relationships on its own, which should make it much more robust when it encounters novel viewpoints or occlusions.

Lu: They are aiming for a system that doesn't just track pixels; they want to track the underlying three dee structure implicitly through these fused features.

Meng: So, less explicit supervision and more implicit structural understanding—that’s the goal I need to see in a production environment to feel comfortable deploying it.

Tom: It’s about achieving that stability without relying on those heavy three dee supervision steps we usually have to do upfront.

Conclusion: Tom: So, to wrap up, 3AM is this system that uses MUSt3R features fused with SAM2’s appearance features to get recognition grounded in both where the object is spatially and how it looks visually <ref:2601.08831#pg1>.

Jane: It’s basically a training-time enhancement that tackles the problem of maintaining object identity under large viewpoint changes without needing explicit three dee ground truth masks.

Lu: The core contribution is showing that robust three dee instance segmentation can emerge from geometry-aware tracking without heavy three dee supervision.

Meng: So, it means we don't need those massive 3d datasets just to get a reliable baseline when objects move around a lot <ref:2601.08831#pg1>.

Tom: It consistently surpasses prior methods on challenging datasets, particularly in scenarios involving object reappearance and large viewpoint variation.

Jane: It really shows how crucial it is to integrate spatial position signals directly into the visual similarity cues for reliable tracking.

Lu: They've shown that geometry-aware 2D tracking can achieve strong results even when the viewpoint changes drastically <ref:2601.08831#pg1>.

Meng: If this translates to real deployment, it means we can skip a lot of the expensive offline processing steps we usually have to do just to get a baseline performance in dynamic scenes.

Tom: The final results show that 3AM attains the highest Tracking Recall and Accuracy on datasets like Replica <ref:2601.08831#pg1>.

Jane: It’s a solid piece of work because it shows how to maintain cross-view correspondence purely from geometry-aware 2D tracking, even when the scene gets cluttered <ref:2601.08831#pg1>.

Lu: It opens up avenues for more efficient, end-to-end approaches that don't require those heavy offline processing steps.

Meng: We’ll see if this performance holds up when we try to run it on our actual production traffic, but the efficiency gain from skipping explicit three dee supervision is definitely something worth watching.

Lalam: I'm confident in my assessment; the results show that 3AM consistently surpasses prior methods and is particularly effective in scenarios involving object reappearance and large viewpoint variation <ref:2601.08831#pg1>.

Tom: So, for now, this paper shows that this approach consistently outperforms prior methods on challenging datasets like Replica and ScanNet++.

Jane: It really demonstrates how much geometry-aware tracking can help stabilize the system when things get unpredictable in a video.

Lu: The implication here is a clearer direction for how we build 2d models that are fundamentally aware of their spatial context <ref:2601.08831#pg1>.

Meng: It’s about making the system smarter at figuring out spatial relationships on its own, which should make it much more robust when it encounters novel viewpoints or occlusions.

Lalam: The paper proves that geometry-aware tracking can emerge without needing explicit three dee supervision.

Tom: And that’s how we wrap up this look at 3AM: 3egment Anything with Adaptive Geometric Consistency in Videos <ref:2601.08831#pg1>.

Jane: It’s a solid piece of work because it shows how to maintain cross-view correspondence purely from geometry-aware 2D tracking <ref:2601.08831#pg1>.

Lu: It opens up avenues for more efficient, end-to-end approaches that don't require those heavy offline processing steps.

Meng: We’ll see if this performance holds up when we try to run it on our actual production traffic, but the efficiency gain from skipping explicit three dee supervision is definitely something worth watching.

Lalam: The paper proves that geometry-aware tracking can emerge without needing explicit three dee supervision.