TAP-Out: Tracking Any Point in 360 via Outpainting
cs.CV
Submitted: 2026-10-30
Updated: 2026-11-12
Comments: Accepted at BMVC 2026. 22 pages, including supplementary material. Project page: https://finlay-hudson.github.io/tapout/ Code: https://github.com/finlay-hudson/TAP-Out
License: http://creativecommons.org/licenses/by/4.0/
The gist: Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences.
Terminology
Abstract
Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences. We present TAP-Out, an offline method that makes this out-of-view (OOV) reasoning explicit by leveraging generative video outpainting as a world-model prior. Given a narrow-FoV input video, TAP-Out outpaints a complete 360° equirectangular video, locates target objects within the panoramic scene via video object segmentation, and synthesises object-centred perspective crops on which standard in-frame trackers operate. This decoupled design composes off-the-shelf foundation models without requiring task-specific architectural modifications. On the TAPVid360 benchmark, TAP-Out achieves state-of-the-art performance, improving OOV tracking accuracy by approximately 3x over the best prior method. We demonstrate its flexibility on out-of-frame object segmentation and detection and we provide detailed diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality, characterising when and why the approach succeeds or fails. Code is publicly available at https://github.com/finlay-hudson/TAP-Out
Sources
- VaViM and VaVAM: Autonomous Driving through Video Generative Modeling
- Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning
- Track Anything Behind Everything: Zero-Shot Amodal Video Object Segmentation
- CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos
- Beyond the Frame: Generating 360 Panoramic Videos from Perspective Videos
- Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations
- SAM 2: Segment Anything in Images and Videos
- SceneTracker: Long-term Scene Flow Estimation Network
- Video models are zero-shot learners and reasoners
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry
- TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs