SceneVGGT: VGGT-based online 3D semantic SLAM for indoor scene understanding and navigation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "SceneVGGT: VGGT-based online 3D semantic SLAM for indoor scene understanding and navigation".
Dev: Here's the gist of the paper. SceneVGGT presents SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation Built on VGGT, our method scales to long video streams via a slidingwindow pipeline We align local submaps using camera-pose transformations, enabling memory- and speed-efficient mapping while preserving geometric consistency Semantics are lifted from 2D instance masks to 3D objects using the VGGT tracking head, maintaining temporally coherent identities for change detection As a proof of concept, object locations are projected onto an estimated floor plane for assistive navigation Core Framework Components The proposed SceneVGGT pipeline consists of three main modules designed for robust scene understanding and navigation (SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation).
Dev: Online 3D scene alignment for streams involves applying VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization (SceneVGGT pipeline consists of three main modules, where (1) we apply VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization).
Title and authors: Rosa: Moving on to the specific title and authors, SceneVGGT: VGGT-based online three dee semantic SLAM for indoor scene understanding and navigation, the authors are Gelencser-Horvath, Dinya, Boglarka, Halász, Muqsit, Karacs <ref:2602.15899#pg1,SceneVGGT: VGGT-based online 3D semantic SLAM for indoor scene understanding and>.
Dev: It’s a team that clearly has deep roots in vision and geometry; they’re combining the powerful VGGT architecture with established SLAM concepts for this specific application.
Taro: I was looking at their work on related papers like SpectraTac and RoboWits, and it seems like the focus here is really on making these complex visual models practical for real-time, long-term scene understanding rather than just short clips.
Rosa: Exactly; they’re tackling the idea of how to make three dee semantic mapping work continuously on video streams without the usual computational explosion <ref:2602.15899#pg1>.
Dev: The core implication here is that we can get dense, semantically labeled three dee maps from continuous video input, which is a huge step for autonomous navigation because it gives the robot a rich understanding of what’s around it, not just geometry <ref:2602.15899#pg1>.
Taro: So this means instead of just knowing where obstacles are geometrically, you also know what they are—like distinguishing a chair from a person—which changes how the agent plans its path.
Rosa: Right, and that semantic understanding is what fuels the navigation module they’ve built for assistive use, allowing things like finding a specific seat based on object class.
Dev: What I find compelling is their approach to scale; by grounding it with LiDAR depth data, they solve the problem of long-term drift that usually plagues systems relying only on camera data.
Taro: That addresses a major weakness in monocular SLAM, so if you’re driving or walking and need accurate positioning over a long period, this metric consistency is vital.
Rosa: So we've seen the title and authors, they are building a scalable system that links visual understanding to physical navigation using VGGT.
Dev: Next up, we look at the actual technical summary of what SceneVGGT actually does under the hood.
The paper's summary: Rosa: Okay, so what’s SceneVGGT doing in practice? Essentially, they propose a framework that combines SLAM with semantic mapping specifically designed for autonomous and assistive navigation.
Dev: It builds on VGGT to create this pipeline where they use a sliding window approach to handle long video streams efficiently and align local submaps using camera-pose transformations for speed and memory savings.
Taro: The summary emphasizes how they lift 2D semantics into three dee objects using the VGGT tracking head, which maintains temporally coherent identities for change detection across frames <ref:2602.15899#pg1,3D objects using the VGGT tracking head>.
Rosa: That means they are not just getting a static picture of the scene; they are tracking persistent objects over time by enforcing that same real-world object gets the same label throughout the whole map.
Dev: They do this by defining a tracklet as a set of instance masks that correspond to the same physical object across time, and they initialize these tracklets only on the first keyframe of each block.
Taro: And they handle scene changes by putting objects into states like recent, removed, or retained based on observation checks; if an object misses observations repeatedly, its confidence score drops to zero.
Rosa: This whole flow leads into the navigation proof-of-concept where they project object locations onto an estimated floor plane for assistive navigation.
Dev: That floor-plane projection is a dimensionality reduction technique that turns the problem into a 2D one, which makes efficient assistive or autonomous navigation much more feasible while using the robust SLAM back end <ref:2602.15899#pg1>.
Taro: The summary also mentions how they continuously update a point density map by dividing total projected points on each grid cell by the number of frames with any observation projected on that cell, making it less sensitive to object dwell time.
Rosa: So, in short, they are creating a system where visual data feeds into three dee semantic mapping, which is then compressed onto a floor plane for navigation <ref:2602.15899#pg1>.
Dev: It’s a comprehensive approach that addresses the scalability of deep learning models on video streams while ensuring the resulting map is both geometrically consistent and semantically meaningful.
The paper's improvements: Taro: Now, let’s talk about what they actually improved over previous methods. One major improvement they highlight is combining VGGT with LiDAR grounding to get that absolute scale and spatial accuracy.
Rosa: That’s the first big win for long-term mapping stability; it directly mitigates the long-term drift that typically affects monocular systems by using sensor depth data to establish metric units as required for navigation and egolocation.
Dev: From a control standpoint, they also improved their tracking reliability by introducing a comparison between newly initialized tracklets and inactive ones using Chamfer distance on sparse point clouds to avoid penalizing partial observations across block boundaries.
Taro: That’s important because when the world misbehaves—say, an object gets momentarily hidden—this method helps them re-identify objects more robustly than just relying on a simple frame-to-frame mask transfer.
Rosa: They also made change detection more resilient by using that object state machine with recent, removed, and retained states; this lets them capture scene updates accurately when an object genuinely disappears or reappears.
Dev: And for performance under memory constraints, they implemented a sliding window pipeline where submaps are aligned using camera-pose transformations which keeps the GPU memory usage under seventeen gigabytes regardless of the input sequence length.
Taro: So, they’ve made it fast enough to be used in real-time for assistive navigation while keeping the system robust against various forms of world misbehavior.
Rosa: Overall, SceneVGGT achieves competitive point-cloud performance on the ScanNet++ benchmark while keeping that memory footprint manageable.
Conclusion: Dev: So to wrap up, SceneVGGT is a VGGT-based online three dee semantic SLAM framework that scales to long indoor video streams via a sliding window design and camera-pose–based submap alignment <ref:2602.15899#pg1,VGGT-based online 3D semantic SLAM>.
Rosa: By integrating LiDAR-derived scale at each step, they effectively mitigate the long-term drift that typically affects monocular-based systems while preserving metric consistency required for navigation.
Taro: The 2D-to-three dee semantic lifting with VGGT tracking enforces temporally coherent instance identities and enables change-aware map updates through persistent object memory <ref:2602.15899#pg1>.
Dev: And finally, floor-plane projection provides a compact top-down representation that supports downstream assistive navigation under realistic constraints, demonstrating the practical utility of the approach under realistic compute and memory constraints.
Rosa: It’s a powerful system because it connects high-level semantic understanding directly to low-level spatial tasks like navigation in a way that is both efficient and robust.
Dev: Yeah, it shows how combining transformer models with traditional SLAM techniques can create something that handles the challenges of long video streams while still being memory-efficient.
Taro: I think the focus on resilience against world misbehavior through state management is what makes this framework really useful for real-world deployment where you can’t always guarantee perfect input.
Rosa: And that's where it shines; it moves beyond just generating a map to creating a usable environment representation for tasks like assistive navigation.
Peter Catholic University, Budapest, Hungary · Eotvös Lor and University, Budapest, Hungary
cs.RO, eess.IV
Submitted: 2026-02-12
Updated: 2026-02-19
DOI: 10.1109/ICIP61757.2026.11630194
Code: https://github.com/ultralytics/ultralytics
Project page: https://hbvc-ai.github.io/SceneVGGT
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 74/100
The gist: SceneVGGT presents SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation Built on VGGT…
Key concepts
- Sliding-Window Pipeline
- The system divides continuous video streams into small, overlapping blocks. It processes these blocks sequentially, estimating the transformation between them to build a map incrementally without needing to process the entire long stream at once. This approach manages high computational and memory costs effectively.
- VGGT Tracking Head
- This component takes 2D instance masks from frames and uses contextual cues to propagate them across time. It is responsible for maintaining persistent object identities, ensuring that the same real-world object gets the same label throughout the entire mapping process, even when objects are temporarily occluded.
- Floor-Plane Projection
- To simplify navigation, 3D observations are projected onto a 2D floor plane estimated from the reconstructed map. This reduces navigation complexity to a 2D problem. A point density map is maintained on this projection, which helps in selecting goals and identifying safe traversal areas.
Terminology
Summary
SceneVGGT presents SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation Built on VGGT, our method scales to long video streams via a slidingwindow pipeline We align local submaps using camera-pose transformations, enabling memory- and speed-efficient mapping while preserving geometric consistency Semantics are lifted from 2D instance masks to 3D objects using the VGGT tracking head, maintaining temporally coherent identities for change detection As a proof of concept, object locations are projected onto an estimated floor plane for assistive navigation
Core Framework Components
The proposed SceneVGGT pipeline consists of three main modules designed for robust scene understanding and navigation (SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation).
-
Online 3D scene alignment for streams involves applying VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization (SceneVGGT pipeline consists of three main modules, where (1) we apply VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization). p. 1
Where the paper says it
EmbodiedSAM [14] leverages SAM for online 3D instance segmentation, highlighting the promise of foundation models for embodied perception, but its 2D-to-3D lifting remains sensitive to mask layering/temporal consistency. METHODS The proposed SceneVGGT pipeline consists of three main modules, where (1) we apply VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization.
-
Lifting 2D semantics with change awareness integrates semantics into this mapping process by enforcing persistent object identities, ensuring that the same real-world object is assigned a consistent label throughout the entire map (we integrate semantics into this mapping process by enforcing persistent object identities, ensuring that the same real-world object is assigned a consistent label throughout the entire map). p. 1
Where the paper says it
METHODS The proposed SceneVGGT pipeline consists of three main modules, where (1) we apply VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization. We (2) integrate semantics into this mapping process by enforcing persistent object identities, ensuring that the same real-world object is assigned a consistent label throughout the entire map.
-
Navigation demonstrates a proof-of-concept navigation module that exploits a floor-plane projection for dimensionality reduction, enabling efficient assistive or autonomous navigation while leveraging the robustness of the underlying SLAM back-end (we demonstrate a proof-of-concept navigation module that exploits a floor-plane projection for dimensionality reduction, enabling efficient assistive or autonomous navigation while leveraging the robustness of the underlying SLAM back-end). p. 1
Where the paper says it
Additionally, if an object is moved or appears, the 3D map is accordingly updated, allowing for a temporally consistent mapping. Finally, we (3) demonstrate a proof-of-concept navigation module that exploits a floor-plane projection for dimensionality reduction, enabling efficient assistive or autonomous navigation while leveraging the robustness of the underlying SLAM back-end.
Stream Processing and Alignment
To handle continuous streams efficiently, SceneVGGT utilizes a sliding-window approach to mitigate high computational and memory costs associated with naive frame processing (we adopt a sliding-window approach. The stream is partitioned into contiguous, disjoint blocks of size n). p. 1 , t − 1—
we adopt a sliding-window approach. The stream is partitioned into contiguous, disjoint blocks of size n.Where the paper says it
The stream is partitioned into contiguous, disjoint blocks of size n.
Interblock pose transforms are estimated using the k overlapping anchor frames to compose the current block’s relative transforms and align them with the previously accumulated trajectory (Interblock pose transforms are estimated using the k overlapping anchor frames, enabling us to compose the current block’s relative transforms and align them with the previously accumulated trajectory).
The system grounds VGGT-predicted depth estimations with sensor depth data to obtain absolute scale and spatial accuracy, operating in metric units for navigation (We ground the VGGT-predicted depth estimations with the depth data from the sensor to obtain absolute scale and spatial accuracy, and to enable the system to operate in metric units as required for navigation and egolocation). p. 1 This ensures that consecutive submaps
remain aligned to the initial global anchor. We ground the
VGGT-predicted depth estimations with the depth data from
the sensor to obtain absolute scale and spatial accuracy, and
to enable the system to operate in metric units as required
for navigation and egolocation.Where the paper says it
A combined validity mask is constructed for both depth maps, excluding pixels based on low confidence or unreliable operating ranges (we construct a combined validity mask for both depth maps that excludes (i) pixels whose VGGT-predicted depth confidence is below a threshold (we used c = 1. p. 1 To improve robustness, we
construct a combined validity mask for both depth maps that
excludes (i) pixels whose VGGT-predicted depth confidence
is below a threshold (we used c = 1. 1) or falls in the lowest
10% of all values and (ii) pixels lying outside the sensor’s
reliable operating range. To improve robustness, we
construct a combined validity mask for both depth maps that
excludes (i) pixels whose VGGT-predicted depth confidence
is below a threshold (we used c = 1. 1) or falls in the lowest
10% of all values and (ii) pixels lying outside the sensor’s
reliable operating range.Where the paper says it
Where the paper says it
Semantic Tracking and Consistency
Semantics are represented as 3D point clouds where points are labeled with instance types and IDs, maintaining consistent labels throughout the sequence to define global objects over time (We propose a method to represent real-world objects as 3D point clouds where points are labeled with instance types and IDs, maintaining these labels consistently throughout the sequence to define global objects over time).
A tracklet is defined as a set of instance masks that correspond to the same physical object across time.
The tracking head propagates instance masks from frame to frame using contextual cues, initializing a tracklet only on the first keyframe of each block (We propagate instance masks from frame to frame using the VGGT tracking head, leveraging its contextual cues even in challenging scenarios with numerous overlapping and occluding objects).
New tracklets are initialized only on the first keyframe of each block, and once a mask is assigned a tracklet ID, all pixels in that mask inherit that ID (New tracklets are initialized only on the first keyframe of each block. p. 2 New tracklets
are initialized only on the first keyframe of each block. Once
a mask is assigned a tracklet ID, all pixels in the mask inherit
that ID. New tracklets
are initialized only on the first keyframe of each block. Once
a mask is assigned a tracklet ID, all pixels in the mask inherit
that ID.Where the paper says it
Where the paper says it
To manage scene changes, objects are categorized into three states: RECENT, REMOVED, or RETAINED based on observation and occlusion checks (An object can be in one of three states: (1) RECENT, (2) REMOVED, or (3) RETAINED. p. 2 An object can be in one of three states:
(1) RECENT, (2) REMOVED, or (3) RETAINED. Objects observed in the current block are in RECENT state with a confidence score of c = 1. An object can be in one of three states:
(1) RECENT, (2) REMOVED, or (3) RETAINED. Objects observed in the current block are in RECENT state with a confidence score of c = 1.Where the paper says it
Where the paper says it
Navigation and Floor Plane Projection
The navigation module reduces the problem to 2D by projecting observations onto a floor plane estimated from the reconstructed 3D point cloud using RANSAC (We derive a top-down 2D map by projecting observations onto the floor, reducing navigation to a 2D problem. The floor plane is estimated from the reconstructed 3D point cloud using a RANSAC-based method). p. 2 We derive a top-down 2D map by projecting observations onto the floor, reducing navigation to a 2D problem. The floor plane is estimated from the reconstructed 3D point
cloud using a RANSAC-based method [15].Where the paper says it
If the reference plane changes, all accumulated points are reprojected onto the new plane.
A point density map is continuously updated by dividing total projected points on each grid cell by frames with any observation projected on that cell, yielding a mean map less sensitive to object dwell time (We continuously update a point density map by dividing the total number of points projected on each grid cell by the number of frames with any observation projected on the cell, yielding a mean map that is less sensitive to object dwell time).
The proof-of-concept navigation module selects goals based on semantic information (e. p. 3 The
map is refreshed in each block to update the changes of object states, ensuring that it remains consistent and reliable for
navigation. We propose a proof-of-concept navigation module that selects the goal based on semantic information (e. We propose a proof-of-concept navigation module that selects the goal based on semantic information (e. , finding a
seat) by filtering the scene with a semantic mask for the target
object class, discarding small connected components to suppress spurious detections, and then choosing the remaining
object whose centroid is closest to the user’s current position. We propose a proof-of-concept navigation module that selects the goal based on semantic information (e. , finding a
seat) by filtering the scene with a semantic mask for the target
object class, discarding small connected components to suppress spurious detections, and then choosing the remaining
object whose centroid is closest to the user’s current position.Where the paper says it
Where the paper says it
Where the paper says it
To ensure safe traversal, occupied cells are enlarged by morphological dilation to create a spatially extended representation of obstacles, marking unknown floor areas as non-traversable (To account for the agent’s non-point-like size, we enlarge occupied cells by morphological dilation to create a spatially extended representation of the obstacles on the map, yielding a conservative free-space map for safe traversal. p. 3 To account for the agent’s non-point-like size, we enlarge occupied cells by morphological dilation to create a spatially
extended representation of the obstacles on the map, yielding a conservative free-space map for safe traversal. Some
floor cells may remain unobserved at a given step, since LiDAR observations are incomplete and the map is updated
locally in a sliding-window, block-wise manner. Some
floor cells may remain unobserved at a given step, since LiDAR observations are incomplete and the map is updated
locally in a sliding-window, block-wise manner. We mark
unknown area on the floor as non-traversable in the obstacle
map, while handling it separately from detected obstacles by
excluding morphological inflation.Where the paper says it
Where the paper says it
Performance and Evaluation
SceneVGGT achieves competitive performance, maintaining GPU memory usage under 17 GB regardless of input sequence length (The pipeline’s GPU memory usage remains under 17 GB, irrespectively of the length of the input sequence and achieves competitive point-cloud performance on the ScanNet++ benchmark).
Overall, SceneVGGT ensures robust semantic identification and is fast enough to support interactive assistive navigation with audio feedback.
For semantic evaluation on ScanNet++, ID consistency is measured by the dominance ratio of the most frequent predicted ID across frames (We measure ID consistency by the dominance ratio: for each ground-truth ID, we take the most frequent non-zero predicted ID across frames as dominant and report its share over all frames of that ground-truth track (consistency without counting objects appearing mid-block)).
The complete pipeline, including instance segmentation, labeling, tracking, and navigation modules adds an additional 2 GB of VRAM but maintains a peak memory usage of 17 GB (The complete SceneVGGT pipeline, including instance segmentation, labeling, tracking, and navigation modules add an additional 2 GB of VRAM, bringing the total peak memory usage to 17 GB). p. 3 89%
TSR, respectively. The complete SceneVGGT pipeline however, including instance segmentation, labeling, tracking, and
navigation modules add an additional 2 GB of VRAM, bringing the total peak memory usage to 17 GB, independently of
the sequence length.Where the paper says it
Runtime for interactive assistive navigation is approximately 3. p. 0 The pipeline’s GPU memory usage remains under
17 GB, irrespectively of the length of the input sequence
and achieves competitive point-cloud performance on the
ScanNet++ benchmark. Overall, SceneVGGT ensures robust semantic identification and is fast enough to support
interactive assistive navigation with audio feedback. Nevertheless, this setup is sufficient to demonstrate the capabilities of the proposed approach. Runtime is not benchmarked on a standardized public
protocol, instead, we provide an indicative measurement on
a representative in-the-wild sequence of 1 500 frames with a
stride 2. Runtime is not benchmarked on a standardized public
protocol, instead, we provide an indicative measurement on
a representative in-the-wild sequence of 1 500 frames with a
stride 2. The alignment stage takes 126. The alignment stage takes 126. 92 fps for the
750 frames), adding semantics (+31. 92 fps for the
750 frames), adding semantics (+31. 35 s) and instance tracking with object and change management (+50. 35 s) and instance tracking with object and change management (+50. 79s) yields
210. 79s) yields
210. 14s total (3. 14s total (3. 57 fps), which is sufficient for interactive
assistive navigation and leaves headroom for future audio
feedback.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations
The current pipeline does not explicitly model continuous object motion or fully dynamic scenes, and future plans include extending change detection to capture updates in unsegmented regions (While our change-aware object memory supports appearance/disappearance and relocation updates, the current pipeline does not explicitly model continuous object motion or fully dynamic scenes. Moreover, we plan to extend change detection beyond recognized instances to also capture updates in unsegmented regions). p. 4 Limitations and future plans
While our change-aware object memory supports appearance/disappearance and relocation updates, the current pipeline
does not explicitly model continuous object motion or fully
dynamic scenes. Moreover, we plan to extend change detection beyond recognized instances to also capture updates in
unsegmented regions.Where the paper says it
Conclusion
SceneVGGT is a VGGT-based online 3D semantic SLAM framework that scales to long indoor video streams via a sliding-window design and camera-pose–based submap alignment (We present SceneVGGT, a VGGT-based online 3D semantic SLAM framework that scales to long indoor video streams via a sliding-window design and camera-pose–based submap alignment). By integrating LiDAR-derived scale at each step, we mitigate long-term drift while preserving metric consistency required for navigation (By integrating LiDAR-derived scale at each step, we effectively mitigate the long-term drift that typically affects monocular-based systems, while preserving metric consistency required for navigation). p. 4 CONCLUSION
We present SceneVGGT, a VGGT-based online 3D semantic
SLAM framework that scales to long indoor video streams
via a sliding-window design and camera-pose–based submap
alignment. By integrating LiDAR-derived scale at each step,
we effectively mitigate the long-term drift that typically affects monocular-based systems, while preserving metric consistency required for navigation. By integrating LiDAR-derived scale at each step,
we effectively mitigate the long-term drift that typically affects monocular-based systems, while preserving metric consistency required for navigation. The proposed 2D-to-3D semantic lifting with VGGT tracking enforces temporally coherent instance identities and enables change-aware map updates through a persistent object memory. The proposed 2D-to-3D semantic lifting with VGGT tracking enforces temporally coherent instance identities and enables change-aware map updates through a persistent object memory. Finally, floor-plane
projection provides a compact top-down representation that
supports downstream assistive navigation, demonstrating the
practical utility of the approach under realistic compute and
memory constraints.Where the paper says it
Where the paper says it
Where the paper says it
References
[1] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny, “VGGT: Visual geometry grounded transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025
[3] SceneVGGT pipeline consists of three main modules, where (1) we apply VGGT with LIDAR grounding to build an incremental 3D map using a sliding-window alignment strategy and pose-graph optimization
[4] The pipeline’s GPU memory usage remains under 17 GB, irrespectively of the length of the input sequence and achieves competitive point-cloud performance on the ScanNet++ benchmark
[5] We present SceneVGGT, a VGGT-based online 3D semantic SLAM framework that scales to long indoor video streams via a sliding-window design and camera-pose–based submap alignment
[8] Yuzheng Liu, Siyan Dong, Shuzhe Wang, Yingda Yin, Yanchao Yang, Qingnan Fan, and Baoquan Chen, “SLAM3R: Real-time dense scene reconstruction from monocular RGB videos,” arXiv preprint arXiv:2412.09401
[10] Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu, “Streaming 4D visual geometry transformer,” arXiv preprint arXiv:2507.11539
[12] Shuai Yuan, Yantai Yang, Xiaotian Yang, Xupeng Zhang, Zhonghao Zhao, Lingming Zhang, and Zhipeng Zhang, “InfiniteVGGT: Visual geometry grounded transformer for endless streams,” arXiv preprint arXiv:2601.02281
[13] Hao Li, Zhengyu Zou, Fangfu Liu, Xuanyang Zhang, Fangzhou Hong, Yukang Cao, Yushi Lan, Manyuan Zhang, Gang Yu, Dingwen Zhang, and Ziwei Liu, “IGGT: Instance-grounded geometry transformer for semantic 3D reconstruction,” arXiv preprint arXiv:2510.22706
[19] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai, “Scannet++: A high-fidelity dataset of 3D indoor scenes,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023
[20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick, “Microsoft COCO: Common objects in context,” in Computer Vision – ECCV 2014
[21] Chien-Yao Wang and Hong-Yuan Mark Liao, “YOLOv9: Learning what you want to learn using programmable gradient information,” arXiV, 2024
[3] You Shen, Zhipeng Zhang, Yansong Qu, and Liujuan Cao, “FastVGGT: Training-free acceleration of visual geometry transformer,” 2025
[6] Dominic Maggio, Hyungtae Lim, and Luca Carlone, “VGGT-SLAM: Dense RGB SLAM optimized on the SL(4) manifold,” arXiv preprint arXiv:2505.12549
[7] Dominic Maggio and Luca Carlone, “VGGT-SLAM 2.0: Real time dense feed-forward scene reconstruction,” 2026
[11] Anonymous, “IncVGGT: Incremental VGGT for memory-bounded long-range 3D reconstruction,” in ICLR 2026 Conference Submission, 2026
[4] Lijie Wang, Lianjie Guo, Ziyi Xu, Qianhao Wang, Fei Gao, and Xieyuanli Chen, “LiDAR-VGGT: Cross-modal coarse-to-fine fusion for globally consistent and metric-scale dense mapping,” arXiv preprint arXiv:2511.01186
[9] Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, and Vincent Leroy, “MUSt3R: Multi-view network for stereo 3D reconstruction,” arXiv preprint arXiv:2503.01661
[15] Martin A. Fischler and Robert C. p. 5 [15] Martin A. Fischler and Robert C. Fischler and Robert C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,”
Communications of the ACM, vol.Where the paper says it
Where the paper says it
[22] Glenn Jocher, Jing Qiu, and Ayush Chaurasia, “Ultralytics YOLO,” https://github.com/ultralytics/ultralytics, Jan. p. 5 [22] Glenn Jocher, Jing Qiu, and Ayush Chaurasia,
“Ultralytics YOLO,” https://github. com/
ultralytics/ultralytics, Jan.Where the paper says it
[18] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon, “Scene coordinate regression forests for camera relocalization in RGB-D images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2013
[2602.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve long-term mapping stability by combining VGGT with LiDAR grounding to obtain absolute scale
to enable the system to operate in metric units as required for navigation and egolocation.
This mitigateslong-term drift that typically affects monocular-based systems
by using sensor depth data for scale alignment. -
Enhance semantic tracking reliability through persistent object identities and temporal consistency, ensuring that
the same real-world object is assigned a consistent label throughout the entire map.
This prevents identity switching across frame boundaries duringchange detection.
-
Enable efficient scene understanding during navigation by utilizing a floor-plane projection to reduce navigation complexity, as described by projecting observations onto the floor and updating a point density map. This allows for
downstream assistive navigation modules
to operate effectively in 2D space. -
Increase resilience of object tracking across temporal gaps by comparing newly initialized tracklets against inactive ones using Chamfer distance on sparse point clouds, avoiding
penalizing partial observations.
This ensures robust re-identification across block boundaries. -
Improve real-time performance under memory constraints by using a sliding window pipeline that partitions the stream into blocks and aligning submaps using
camera-pose–based submap alignment.
This makes the system suitable for processing long video streams while keeping GPU memory usageunder 17 GB, irrespectively of the length of the input sequence.
-
Enhance change detection capabilities by maintaining an object state machine (RECENT, REMOVED, RETAINED) based on confidence scores and visibility checks. This allows the system to capture scene updates by marking objects as
REMOVED
whenrepeated misses drive c to zero,
consistent with a true scene change.
Abstract
We present SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation. Built on VGGT, our method scales to long video streams via a sliding-window pipeline. We align local submaps using camera-pose transformations, enabling memory- and speed-efficient mapping while preserving geometric consistency. Semantics are lifted from 2D instance masks to 3D objects using the VGGT tracking head, maintaining temporally coherent identities for change detection. As a proof of concept, object locations are projected onto an estimated floor plane for assistive navigation. The pipeline's GPU memory usage remains under 17 GB, irrespectively of the length of the input sequence and achieves competitive point-cloud performance on the ScanNet++ benchmark. Overall, SceneVGGT ensures robust semantic identification and is fast enough to support interactive assistive navigation with audio feedback.
Sources
- SwiftVGGT: A Scalable Visual Geometry Grounded Transformer for Large-Scale Scenes
- LiDAR-VGGT: Cross-Modal Coarse-to-Fine Fusion for Globally Consistent and Metric-Scale Dense Mapping
- VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences
- VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold
- SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos
- MUSt3R: Multi-view Network for Stereo 3D Reconstruction
- Streaming 4D Visual Geometry Transformer
- InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams
- IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstruction
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time
- Frontier Based Exploration for Autonomous Robot
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation