ReLift: Dense 3D Tracking as Lifted Trajectory Repair
cs.CV
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/Lijiaxin0111/Open-d4rt
Project page: https://st4rtrack.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion.
Terminology
Abstract
Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.
Sources
- Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos
- Any4D: Unified Feed-Forward Metric 4D Reconstruction
- CoWTracker: Tracking by Warping instead of Correlation
- MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds
- Depth Anything 3: Recovering the Visual Space from Any Views
- Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
- 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking
- DELTAv2: Accelerating Dense 3D Tracking
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- V-DPM: 4D Video Reconstruction with Dynamic Point Maps
- Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
- DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video
- VGGT-
- Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video
- Efficiently Reconstructing Dynamic Scenes One D4RT at a Time
- MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs