VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction
cs.CV
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 21 pages, including references and appendices
Code: https://github.com/kosakayamahoo-design/VASC
License: http://creativecommons.org/licenses/by/4.0/
The gist: Feed-forward 3D vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass.
Terminology
Abstract
Feed-forward 3D vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass. However, their quadratic global attention makes long image sequences expensive, while existing sparse methods may favor highly attended yet value-redundant regions. To address these limitations, we introduce VASC, a training-free sparse attention method combining value-aware block selection and execution-aware cross-layer memory. Our value-aware block selection integrates pooled query--key relevance with neighboring value contrast, reducing redundancy while preserving query-relevant and distinctive content. Cross-layer memory tracks unserved demand across layers and updates this state according to actual execution, enabling previously underserved blocks to compete under a fixed computation budget. Experiments on 7Scenes and NeuralRGB-D with VGGT and π cubed demonstrate improved pose estimation and reconstruction quality compared with FasterVGGT, together with up to 2.29 times faster inference than dense VGGT. Code is available at https://github.com/kosakayamahoo-design/VASC.
Sources
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- Longformer: The Long-Document Transformer
- Token Merging: Your ViT But Faster
- GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction
- Generating Long Sequences with Sparse Transformers
- Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations
- TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
- FastVGGT: Training-Free Acceleration of Visual Geometry Transformer
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models