TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects

summary

Video file (mp4)

The gist

The gist: TrackDeform3D presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras by identifying 3D keypoints and robustly tracking

In short

TrackDeform3D is an autonomous system using RGB-D cameras and robotic arms to collect 3D keypoint data from deformable objects like wires or fabrics. It solves a complex optimization problem at every frame to ensure keypoints stay in the same relative positions (trajectory consistency) and are correctly placed across different collection sessions (initialization consistency), producing high-quality, large-scale datasets for robotics research.

Key concepts

Anchor Points
These are specific 3D locations on the object that are constrained to remain fixed during the tracking optimization process. They act as stable reference points, ensuring that keypoints maintain consistent spatial relationships across different video frames and even different data collection sessions, which stabilizes the entire tracking solution.
Edge Length Constraints
This constraint forces neighboring keypoints to maintain a uniform distance between them throughout the movement. By enforcing this rule during the tracking calculation, the system prevents noisy fluctuations and sudden jumps in keypoint positions, resulting in smooth and temporally consistent trajectories suitable for physics simulations.
Point Cloud Differencing
This is an unsupervised method used to automatically separate the deformable object from its background in a point cloud. By comparing consecutive depth frames, the system filters out static background points and clusters the remaining points into a segmented object cloud, allowing autonomous detection even when objects are partially obscured.
Topology Identification
This step involves analyzing the local geometry of the segmented point cloud to determine which points belong to keypoints versus structural elements. This helps in identifying meaningful locations for tracking and ensures that the extracted data accurately reflects the underlying structure of the deformable object.

Terminology used across episodes

This episode discusses

The paper

TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects · Read on arXiv

Yeheng Zong, Yizhou Chen, Alexander Bowler, Chia-Tung Yang, Ram Vasudevan

University of Michigan

Structured 3D representations such as keypoints and meshes offer compact, expressive descriptions of deformable objects, jointly capturing geometric and topological information useful for downstream tasks such as dynamics modeling and motion planning. However, robustly extracting such representations remains challenging, as current perception methods struggle to handle complex deformations. Moreover, large-scale 3D data collection remains a bottleneck: existing approaches either require prohibitive data collection efforts, such as labor-intensive annotation or expensive motion capture setups, or rely on simplifying assumptions that break down in unstructured environments. As a result, large-scale 3D datasets and benchmarks for deformable objects remain scarce. To address these challenges, this paper presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras. The proposed method identifies 3D keypoints and robustly tracks their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data. TrackDeform3D is evaluated against several state-of-the-art tracking methods across diverse object categories and demonstrates consistent improvements in both geometric and tracking accuracy. Using this framework, this paper presents a high-quality, large-scale dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data. Project page: https://roahmlab.github.io/trackDeform3D-core-tracking/

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects".

Jane: Here's the gist of the paper. The gist: TrackDeform3D presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras by identifying 3D keypoints and robustly tracking their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data Deformable objects such as wire harnesses or fabrics are ubiquitous in homes and factories, yet remain challenging for robots to manipulate.

Tom: Because these objects possess infinitely many degrees of freedom, directly modeling their full state is intractable.

Jane: Researchers therefore often approximate their state with structured 3D representations such as keypoints or meshes, which provide a compact yet informative description of their geometry and topology.

Tom: Large-scale, high-quality keypoint and mesh trajectory datasets of deformable objects could significantly advance robotic manipulation.

Title and authors: Tom: So we're looking at a paper called TrackDeformthree dee: Markerless and Autonomous three dee Keypoint Tracking and Dataset Collection for Deformable Objects <ref:2603.17068#pg1,TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable>. I mean, "markerless" is the key here, because usually tracking these things requires you to stick physical markers on them, which is labor-intensive and messy.

Jane: Right, so if you don't need those physical markers to track a wire harness or a piece of fabric, that's a big deal because it opens up data collection ways that are way less painful for researchers and companies.

Lu: What makes this paper interesting is that they aren't just showing how to track one object; they are proposing an entire framework for autonomous collection using only standard RGB-D cameras.

Meng: Autonomous collection sounds good on paper, but what does "autonomous" actually mean in practice? Does it mean it works perfectly every time without human intervention during setup?

Tom: Well, the core idea is that the system can identify three dee keypoints and robustly track them using just RGB-D cameras, which solves a huge bottleneck we've seen <ref:2603.17068#pg1>.

Jane: And what they claim to solve is getting high-quality data for things like wire harnesses or fabrics where you can't just model the whole thing because it has infinite degrees of freedom.

Lu: They are proposing a system that identifies these keypoints and then enforces geometric consistency, which means if you collect data over different sessions, the keypoints stay in roughly the same place.

Meng: Consistency across sessions is hard to get without some kind of fixed reference point or something, right?

Tom: Exactly. They use anchor points for that purpose, which they say constrains the solution space and stabilizes where those keypoints are supposed to be >

The paper's summary: Jane: So, what is the actual process described in TrackDeformthree dee? It sounds like it’s a multi-step pipeline for taking video and turning it into these consistent three dee point clouds <ref:2603.17068#pg1>.

Tom: Basically, they start with an input of an RGB-D video sequence from time zero to T. The first big step is lifting that depth video to get a point cloud, and then they do some segmentation to figure out which parts are the deformable object itself >

Lu: They use point cloud differencing for this segmentation, where they filter out static background points before applying something called DBSCAN to cluster the remaining points into a segmented point cloud >

Jane: Then comes the initialization phase, where they classify the object type based on its local geometry and then determine anchor points and warm-start positions >

Tom: The real heavy lifting is in the problem formulation at each time step t, where they minimize an energy function to encourage uniform spacing among neighboring keypoints until it reaches an equilibrium >

Meng: Uniform spacing sounds like a lot of math to solve in real time. How are they managing that optimization loop without it taking forever?

Jane: They solve this constrained optimization problem by using an anchor set A to stabilize the solution space and ensure the keypoint configuration is stable across different collection sessions >

Tom: And for tracking, they re-detect those anchor points every frame, and then they solve that same optimization recursively, warm-started from the previous time step's solution >

Lu: Plus, they apply a temporal moving average filter to suppress high-frequency jitter in the resulting trajectories to make them smooth and temporally consistent >

The paper's improvements: Jane: So what are the actual improvements they point out over existing methods? It sounds like they’re focusing on initialization and tracking consistency.

Tom: They claim their two contributions are twofold, first an affordable tracking system that enforces geometric consistency for robust initialization and tracking across multiple collection sessions >

Lu: And the second is this large-scale dataset of 1D and 2D deformable objects, comprising six objects and totaling one hundred ten minutes of trajectory data for training and benchmarking deformable object models > <ref:2603.17068#pg2,objects and totaling 110 minutes of trajectory data for training and benchmarking>

Meng: A dataset of that size is huge. But what about the quality? Are these trajectories as clean as you'd expect from an autonomous system?

Jane: The paper shows they achieved lower Absolute edge length error, reported as RMSE in millimeters, and Chamfer distance compared to baselines like CDCPD2 and SpatialTrackerV2 >

Tom: And they also did ablation studies showing that both the anchor points and the edge length constraints are necessary during tracking; removing either one causes keypoints to drift or lose consistent spacing >

Lu: That’s important because it confirms that those specific components aren't just tacked on; they are essential for keeping the data geometrically accurate >

Conclusion: Jane: So to wrap up, what does TrackDeformthree dee actually mean for the field of modeling deformable objects? It seems like a lot of consistency achieved through careful optimization.

Tom: It means we can finally have a large-scale resource, which is six objects and one hundred ten minutes of data, annotated with keypoint trajectories and topology information > <ref:2603.17068#pg2>

Lu: This dataset addresses the scale and diversity gaps left by prior work, especially by including branched deformable objects which are crucial for things like wire harness modeling >

Meng: For deployment at scale, I’m concerned about the reliance on those anchor points being reliably detected every single frame. If those fail, the entire tracking system collapses >

Jane: That's a fair concern. But overall, TrackDeformthree dee successfully enforces anchor, edge length, and point cloud projection constraints to produce consistent and geometrically accurate keypoint trajectories across diverse object categories like DLOs, BDLOs, and cloth >

Tom: The work confirms the contribution of each optimization component through those ablation studies validating the necessity of anchor and edge length constraints for accurate and stable tracking >

Lu: It’s a solid pipeline that moves the goalpost on how we build these representations for complex geometries >

Meng: It's a big dataset, which is what matters most right now if we want to train models that actually work in the real world >

More episodes

← Home