PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models

arXiv:2606.16569 · cs.CV, cs.RO · Submitted 2026-06-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’ve just covered the mechanics of how PROSE works and its results on the Aria benchmarks. Now let's talk about what exactly the paper is called and who came up with it, so we know what we're talking about.

Jane: Right, before we get into the nitty-gritty details of how it works together, we should just lay out the title and see if that name makes sense in this context.

Tom: Okay, so the full title is "PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models," and it’s written by Zhiang Chen, Nahyuk Lee, Boyang Sun, Taein Kwon, Marc Pollefeys, and Zuria Bauer.

Lu: I looked at their work and what really grabbed my attention is how they combine the VLM for both building the three dee scene graph and matching across the scans <ref:2606.16569#pg1>. That’s a big architectural move.

Meng: Combining those functions under one VLM is neat because it suggests a more integrated approach, but I wonder about the complexity of running such a large model in production.

Lalam: The paper essentially says that this single pretrained VLM can handle the entire pipeline, from listing objects to matching instances across two scans.

Jane: So if you’re listening just listening to this show, what does that mean for us? It means we’re talking about a system that tries to solve a complex spatial problem—aligning two different camera views of the same room—using just what’s already learned by large vision models.

Tom: Precisely. The implication is that you don't need massive amounts of custom data or hours of training to build this kind of scene understanding system; you just leverage the pre-trained knowledge in the VLM to get a functional registration pipeline.

Lu: They compare it against methods like SGAligner and SG-PGM, which are scene graph registration papers. The key difference they point out is that PROSE differs on every axis because it’s training-free >

Meng: So they’re saying the main advantage isn't just a better accuracy number, but the fact that you don't have to deal with the whole process of training a custom scene graph model yourself.

Lalam: I see how that fits with what we discussed earlier about their pipeline—it leverages existing models for geometry, segmentation, and language tasks in sequence.

Jane: So when we talk about the overall idea of PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models, it’s a way to achieve persistent spatial memory without needing any manual setup.

The paper's summary: Tom: We talked about the title and authors, and now we need to get into the actual summary of what PROSE is trying to accomplish in plain terms.

Jane: So, at its core, this paper is tackling the problem of aligning two RGB sequences taken at different times when one or both cameras are egocentric.

Lu: The summary says that aligning two captures of the same indoor scene under different times is a primitive for persistent autonomy—think a robot revisiting a room it mapped last month, or an AR headset re-anchoring content across sessions.

Meng: That’s the practical application right there: making robots and AR systems remember where they are in three dee space even when the camera moves around <ref:2606.16569#pg1>.

Lalam: The paper breaks down PROSE into steps: it first parses each scan into a per-scan three dee scene graph, then matches instances across those scans, and finally estimates the rigid transform T between them <ref:2606.16569#pg1>.

Tom: So, to put that in simple terms for someone listening just listening to the show: it’s a system that uses a VLM to understand what objects are in each video frame individually and then figures out how those objects correspond between two different videos taken at different times.

Jane: That's a good way to think about it. The key is that the AI is using its understanding of objects, not just raw pixels, to figure out the spatial relationship between the cameras.

Lu: They focus on generating an open-vocabulary scene graph alongside the transform that a simulated agent can use to plan paths over a registered map.

Meng: So it’s not just about getting a pose; it’s about giving you a structured representation of the room that lets you actually move in it.

Lalam: And they emphasize that this approach is robust because object-level semantic correspondence is far less sensitive to noise than dense geometric descriptor matching.

Tom: That’s an important distinction, and it means the system is more resilient when dealing with messy, real-world video data compared to methods that rely only on raw geometry.

Jane: So in short, PROSE is a training-free pipeline for cross-time egocentric scene registration that recovers both instance correspondences and a rigid transform while producing an open-vocabulary scene graph in the same pass.

Lu: The implication is that you can get this level of output without needing any depth sensor, training data, or annotated graph to start.

The paper's improvements: Tom: Now let’s talk about what they actually improved over the previous work we discussed. They found several key enhancements in PROSE.

Jane: So, what are these specific improvements? What makes this version better than the methods that came before?

Lu: The main improvement is that they address a few failure modes of naive prompting by targeting egocentric failures directly, like Landmark-Salience and Hallucination-Guard.

Meng: Targeting those specific issues shows they didn't just apply a generic prompt; they really thought about where the AI would fail in an egocentric setting.

Tom: And then they use SAM three to turn those object names into temporally consistent instance masks, which is a clever way to ensure the segmentation stays reliable over time <ref:2606.16569#pg1>.

Lalam: After that, they merge these masks by backprojecting every masked pixel into three dee and voxelizing the combined cloud, relabeling points to the locally majority ID, and merging near-duplicate instances using a geometric overlap test <ref:2606.16569#pg1>.

Jane: That whole process sounds like a lot of work to get those per-frame masks to actually combine into a single three dee scene representation <ref:2606.16569#pg1>. How much computation is involved there?

Tom: It’s heavy on computation, but it’s necessary because they are fusing the semantic information back into a dense three dee cloud for accurate matching <ref:2606.16569#pg1>.

Lu: The second big improvement is in cross-scan instance correspondence matching, where they use height-based binning to ensure queries are tractable and rules out implausible matches like a ceiling lamp against a floor rug.

Meng: Height binning sounds like it’s a very practical way to manage the computational load while still maintaining high accuracy by filtering out obvious geometric impossibilities.

Jane: So, that height-based binning is actually described as a key contributor to correspondence accuracy in this work. That makes sense because it tackles the problem of competing objects at different heights during prompting.

Tom: And after they have those correspondences, they use RANSAC on the pair’s point-level correspondences for outlier rejection before scoring each candidate transform Ti by its inlier ratio.

Lu: Scoring each candidate transform Ti by its inlier ratio means they select the one with the strongest geometric consensus, which is what drives their final rigid transform T.

Meng: It’s a solid way to filter out those noisy matches that might otherwise skew the alignment, relying on geometry rather than just semantic guesses.

Jane: So they are combining semantic filtering—the height binning and verification passes—with geometric scoring to get the final pose estimate.

Conclusion: Tom: Alright, we’ve gone through the whole paper, so let's bring it home with a summary of what this means for us moving forward.

Jane: So, to wrap up the discussion on "PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models," the main point is that this method provides a complete pipeline for aligning two egocentric RGB sequences.

Lu: It’s training-free, it recovers instance correspondences and estimates a rigid transform while producing an open-vocabulary scene graph in one go.

Meng: It gives us a structured representation that can be used immediately for downstream tasks like path planning because the resulting graph is dense and semantically rich.

Lalam: This makes it useful because we don't need to do any training or annotation work to get this representation ready for real-world robot deployment.

Tom: It really shows that you can leverage pretrained AI models to tackle complex spatial problems without needing specialized training for every single application.

Jane: The limitation they state is that VLM spatial reasoning can be sensitive to prompt phrasing and preprocessing, and it also degrades under rapid motion or sparse scenes.

Lu: They also note that since it’s a multi-stage pipeline, errors can compound across those components, which means the method isn't perfect on its own.

Tom: So while PROSE is a training-free solution that requires no depth sensor, no training data, and no annotated graph for this kind of cross-time registration, we still have to be mindful of the input quality.

Jane: That’s the final thought on this paper by Zhiang Chen et al. on PROSE: Training-Free Egocentric Scene Registration with Vision-Language Models.

Zhiang Chen, Nahyuk Lee, Boyang Sun, Taein Kwon, Marc Pollefeys, Zuria Bauer

ETH Zurich · VGG, University of Oxford

cs.CV, cs.RO

Submitted: 2026-06-15

Updated: 2026-10-13

Comments: Accepted to CoRL 2026

Project page: https://rckola.github.io/prose

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: PROSE is a training-free pipeline for cross-time egocentric scene registration that uses a single pretrained VLM both to build a per-scan 3D scene graph and to match instances across two

Key concepts

Scene Graph Parsing: This step converts a single RGB image sequence into a structured 3D graph. It identifies distinct objects (nodes) and their spatial relationships (edges) based on geometry and segmentation. This creates an object-level representation of the scene, which is then used to understand what objects are present in each scan.
Object Instance Correspondence: This is the core matching process where PROSE finds which object in the first scan corresponds to which object in the second scan. It uses a height-based binning strategy and VLM verification to ensure matches are reliable, even when objects overlap or move quickly. This allows for accurate cross-scan alignment.
Pose Hypothesis and Voting: Instead of fitting one transformation to all matches at once, PROSE generates separate rigid pose hypotheses for each matched object pair. It then scores these hypotheses based on how well they align the source point cloud with the reference cloud, selecting the best consensus transform for final registration.

Terminology

Point Clouds — 2× in this paper · explained in 2 episodes
Accuracy — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Digital Twin — 1× in this paper · explained in 5 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist: PROSE is a training-free pipeline for cross-time egocentric scene registration that uses a single pretrained VLM both to build a per-scan 3D scene graph and to match instances across two captures of the same scene.

How it works

PROSE addresses the challenge of aligning two egocentric RGB sequences taken at different times by leveraging a pretrained Vision-Language Model (VLM) as the central engine for both scene understanding and cross-scan matching. The pipeline first lifts each RGB sequence into an object-level 3D scene graph using off-the-shelf foundation models for geometry, segmentation, and language. This involves several steps:

Scene parsing operates on each RGB sequence and produces a scene graph G = (V, E) whose nodes V are 3D object instances and whose edges E connect instances with nearby centroids.
Object listing involves prompting a pretrained VLM to list the objects in each scene. This prompt is augmented with five design choices targeting egocentric failure modes, such as Landmark-Salience and Hallucination-Guard.
Instance segmentation uses SAM 3 to turn those names into temporally consistent instance masks.
Per-instance fusion merges these per-frame masks by backprojecting every masked pixel into 3D and voxelizing the combined cloud, relabeling points to the locally majority ID, and merging near-duplicate instances using a geometric overlap test.

Cross-scan instance correspondence

The core of PROSE involves matching object instances across the two scene graphs G ref and G src. Naive prompting fails because the query is intractably large and confusable objects at different heights compete, leading to hallucinations. To make matching tractable and reliable, PROSE leverages object heights as a prior by comparing only instances at similar heights through height-based binning into K=5 equal-count quantile bins with 20% overlap between adjacent bins. Within each bin, the VLM is queried separately, and candidates are filtered by a paired same/different verification pass. The final correspondence set C is formed by pairs that survive this rigorous verification process.

Pose hypothesis and voting

Once the instance correspondences C = (A i, A'i) are established between the two scene graphs G ref and G src, PROSE estimates the rigid transform T ∈ SE(3) aligning source to reference. Rather than fitting T to all correspondences jointly, it generates one transform hypothesis per matched pair and selects the one with the strongest geometric consensus. This is achieved by running RANSAC on the pair’s pointlevel correspondences for outlier rejection and scoring each candidate transform Ti by its inlier ratio: the fraction of the source point cloud that lands within a fixed distance of the reference cloud. The highest-scoring Ti becomes the final T.

Experimental Results and Evaluation

PROSE was evaluated on the egocentric Aria Digital Twin (ADT) and Aria Everyday Activities (AEA) benchmarks. On ADT, PROSE improves over both geometric and learned scene-graph baselines on registration recall, rotation, and translation error, both on ground-truth and RGB-reconstructed point clouds. The method outperforms baselines on registration recall by a wide margin while also leading on recall. For instance, PROSE attains the best accuracy across all three ARIA splits on ground-truth clouds. Furthermore, PROSE produces an open-vocabulary scene graph alongside the transform that a simulated agent uses to plan paths over a registered map.

Downstream Application

PROSE demonstrates its utility in downstream tasks like path planning by applying it to a small room with an iPad. The resulting scene graph provides both the densest spatial coverage and the richest semantic target set among compared methods. This rich representation enables a mobile robot to successfully navigate to target objects and places in real indoor scenes.

Conclusion

PROSE introduces a training-free pipeline for cross-time egocentric scene registration that jointly recovers cross-scan instance correspondences and a rigid transform while producing an open-vocabulary scene graph in the same pass. The method is robust to noise and partial coverage because object-level semantic correspondence is far less sensitive to the noise than dense geometric descriptor matching. PROSE stands as a training-free solution that requires no depth sensor, training, or annotated graph.

Limitations

Several limitations remain; VLM spatial reasoning is sensitive to prompt phrasing and preprocessing. The method relies on static landmarks and degrades under rapid motion or sparse scenes. As a multi-stage pipeline, errors also compound across components.

Acknowledgments

This work was supported under project ID a144 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.

References

[1] C. Liu, Z. Qiao, J. Shi, K. Wang, P. Liu, and S. Shen., Sg-reg: Generalizable and efficient scene graph registration., IEEE Transactions on Robotics, 41:3870–3889., 2025., doi:10.1109/TRO.

[2] H. Yang, J. Shi, and L. Carlone., Teaser: Fast and certifiable point cloud registration., IEEE Transactions on Robotics, 37(2):314–333., 2020.

[5] Z. Qin, H. Yu, C. Wang, Y. Guo, Y. Peng, and K. Xu., Geometric transformer for fast and robust point cloud registration., In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11143–11152, June 2022.

[6] M. Seo, H. Lim, K. Lee, L. Carlone, and J. Park., Buffer-x: Towards zero-shot point cloud registration in diverse scenes., 2503.07940 (arXiv), 2025., arXiv:2503.07940

[7] S. D. Sarkar, O. Miksik, M. Pollefeys, D. Barath, and I. Armeni., Sgaligner: 3d scene alignment with scene graphs., In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21927–21937, 2023.

[8] Y. Xie, A. Pagani, and D. Stricker., Sg-pgm: Partial graph matching network with semantic geometric fusion for 3d scene graph alignment and its downstream tasks., In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28401–28411, 2024.

[9] Z. Qi, Z. Zhang, Y. Fang, J. Wang, and H. Zhao., Gpt4scene: Understand 3d scenes from videos with vision-language models., arXiv preprint arXiv:2501.01428, 2025., arXiv:2501.01428

[10] G. Frisoni, L. Molfetta, M. Buzzoni, and G. Moro., Graph-of-mark: Promote spatial reasoning in multimodal language models with graph-based visual prompting., In AAAI-26, Sponsored by the Association for the Advancement of Artificial Intelligence.

Improvements for AI systems

Bold Header: Integrated Scene Graph Generation and Pose Estimation

The improved system can perform cross-scan instance correspondence and estimate a rigid transform by leveraging a single pretrained VLM to both list the objects in each sequence, lifting them into an object-level 3D scene graph and then prompting the same VLM to match instances across the two RGB sequences.

Bold Header: Robust Instance Matching via Height Binning

The system can significantly improve matching reliability by implementing a specific filtering step: height-based binning, which is described as a key contributor to correspondence accuracy, ensuring that queries are tractable and rules out implausible matches like a ceiling lamp against a floor rug.

Bold Header: Training-Free, Open-Vocabulary Scene Graph Transfer

The resulting scene graph can be directly utilized for downstream tasks because the method emits an open-vocabulary scene graph alongside the transform, which a simulated agent uses to plan paths over a registered map, and this graph's structure transfers directly to downstream tasks.