Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images".

Jane: Here's the gist of the paper. This work proposes a novel method for markerless 3D tissue tracking by leveraging two CoTracker models, one for temporal tracking and one for stereo matching, to estimate 3D motion from stereo endoscopic images.

Tom: Proposed Method 1.

Jane: Stereo matching identifies corresponding pixels between two views of the same sceneBy combining two CoTracker3 models, one for temporal tracking and one for stereo correarXiv:2508.07851v1

cs.CV: 11 Aug 2025>.

Paper summary: Tom: So we're looking at this paper now called "Tracking Any Point Methods for Markerless three dee Tissue Tracking in Endoscopic Stereo Images" <ref:2508.07851#pg1,Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo>. Basically, they’re tackling the problem that surgery is getting more minimally invasive, but that makes tracking moving tissue really hard.

Jane: Right, so the main idea here is to get accurate tissue tracking without needing any markers on it. They propose a new way using something called 2D Tracking Any Point or TAP networks to estimate three-dimensional motion from stereo endoscopic images <ref:2508.07851#pg1,motion from stereo endoscopic images>.

Lu: What they claim is that this method combines two different CoTracker models; one handles the temporal tracking over time, and the other does the stereo matching to find where things are in the other view.

Meng: That sounds like a lot of components, but what's actually going on mechanically? How do those two models work together to give you that three dee motion estimate <ref:2508.07851#pg1>?

Tom: Well, they use an online CoTracker model for tracking points along the time dimension in one view, and then they stack those frames as stereo pairs. Then a second offline CoTracker model handles the stereo correspondence between those views to find matching points.

Jane: So once they have those spatial correspondences between the left and right camera views, what do they use that information to figure out?

Lu: From those spatial correspondences, they estimate the depth, which is how far away something is along that Z-axis, and then they figure out the lateral position on X and Y axes.

Meng: That sounds like a standard reconstruction process. What about errors? If you're estimating depth from just two views without markers, isn't that where things usually fall apart?

Tom: They use a small ensemble of points to mitigate those depth estimation errors, instead of relying on just one point. And they derive the single position estimate along all axes by computing the median over all those points.

Jane: That sounds like a way to be more robust to noise in that estimation process. And what kind of setup did they use for testing this method?

Tom: They used a clinical laparoscopic setup, which involves a da Vinci laparoscope with stereo cameras running at one thousand eighty by one thousand nine hundred twenty pixels at thirty frames per second.

Lu: They simulated tissue motion using a robotic arm, an ABB IRB-one hundred twenty moving the target along predefined trajectories while logging the ground truth positions.

Paper summary: Meng: I'm wondering about the data they used for these experiments. Did they test this on actual surgical footage, or was it mostly synthetic?

Tom: They conducted experiments on both a synthetic three dee-printed phantom and a chicken tissue phantom, which is pretty common for testing things before moving to real surgery <ref:2508.07851#pg1,a synthetic 3D-printed phantom and a chicken tissue phantom>.

Jane: And what were the results like when they tested it on the chicken tissue phantom specifically? That's where things are often most challenging.

Lu: Tracking on the chicken tissue phantom yielded more reliable results, with Euclidean distance errors as low as one point one millimeters at a velocity of ten millimeters per second <ref:2508.07851#pg1,Tracking on the chicken tissue phantom yielded more reliable results, with Euclidean>.

Meng: A millimeter is small, but for surgery, how does that translate to actual safety? What's the caveat there?

Tom: They also found that tracking on the chicken breast phantom proved more reliable overall, with notably fewer outliers in their measurements.

Jane: So while they got those good numbers at a slow speed, what happened when they pushed the speed up? How does performance change at higher velocities?

Lu: The mean Euclidean distance error for full three dee movements ranges from about one point one millimeters for ten millimeters per second to around three point five millimeters at a velocity of eighty millimeters per second.

Meng: So the error jumps pretty significantly when you double the speed, which is what we expect, but that increase to over three millimeters at eighty miles an hour might be a concern for high-speed movements.

Tom: They also found that for the three dee-printed phantom, depth errors start around one millimeter at low velocities but increase steadily, surpassing two millimeters for speeds above fifty millimeters per second <ref:2508.07851#pg3,for the 3D-printed phantom, depth errors start around 1>.

Jane: That means the synthetic phantom shows a clear speed limit before accuracy drops off significantly. What about the real tissue?

Lu: In contrast, the chicken tissue phantom exhibits slightly higher errors at lower speeds but maintains depth accuracy below two millimeters even at eighty millimeters per second.

Meng: So it seems the biological tissue is surprisingly resilient compared to the plastic phantom when you push it hard. That's an interesting point for deployment.

Tom: Their discussion suggests that the chicken tissue exhibits more distinct features, which makes it easier for CoTracker to lock onto consistent visual landmarks compared to a feature-poor three dee print surface <ref:2508.07851#pg1>.

Jane: So, what does this mean for the practical application of this paper? What's the big picture here for surgeons who don't read papers?

Paper summary: Lu: The paper highlights that TAP-based models have potential for accurate, markerless three dee tracking in challenging surgical scenarios where dynamic tissue motion is a major problem <ref:2508.07851#pg1>.

Meng: From a production standpoint, what are the main hurdles you see right now when trying to move this from an experimental setup to something surgeons can actually trust?

Tom: One limitation they mention is that the current tracking errors are still relatively high for clinical application, so it’s not quite ready for immediate surgery.

Jane: And what about the technical limitations of the proposed method itself? What did they admit wasn't working perfectly yet?

Lu: They pointed out a few things, like selecting more informative or discriminative points could improve performance further, but arbitrarily increasing point density doesn't actually yield better results and can even introduce noise or redundancy.

Meng: I hate hearing that arbitrarily increasing point density doesn't help; that just adds computational overhead without payoff. What about the architecture itself?

Tom: They also noted a limitation that TAP methods like CoTracker aren't inherently designed for stereo matching, which might lead to suboptimal results in some cases.

Jane: And because they use two CoTracker instances, each image from one camera stream is processed twice, which increases the computational load on the system.

Lu: Future work should explore using a shared encoder to reduce that redundancy and improve efficiency when running the model at scale.

Meng: That's what I care about—the runtime cost. If we can cut down that processing load with a shared encoder, this becomes something we could actually deploy in a real-time surgical assistant.

Tom: So, to wrap up this paper on "Tracking Any Point Methods for Markerless three dee Tissue Tracking in Endoscopic Stereo Images", it shows a solid framework by combining temporal and stereo tracking models <ref:2508.07851#pg1,Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo>.

Jane: It really is showing how you can use two successive 2D tracking algorithms to achieve a single method for three dee tracking without any external markers <ref:2508.07851#pg1>.

Lu: The key contribution is leveraging the CoTracker3 online model for temporal tracking and the offline version for stereo correspondence to estimate depth and position.

Meng: For a surgeon, this means potentially less setup time because they don't have to apply physical markers to the tissue first.

Tom: But we still have those residual errors at eighty millimeters per second, so it’s not perfect yet.

Jane: It shows the direction for research towards using these 2D tracking approaches for more complex medical imaging problems where things move a lot <ref:2508.07851#pg1>.

Conclusion: Tom: So we’ve been looking at how they used two different CoTracker models to estimate threeD motion from those stereo endoscopic images, and now we’re getting to the conclusion of this paper, "Tracking Any Point Methods for Markerless three dee Tissue Tracking in Endoscopic Stereo Images".

Jane: Right. The main point here is that they put together a method that lets you track tissue in three dimensions without needing any physical markers on it, using just the images from two cameras.

Lu: They achieved this by combining an online CoTracker model for tracking points over time with an offline CoTracker model for stereo matching to find those spatial correspondences.

Meng: So, what does that actually mean for a surgeon who doesn't want to pre-mark everything?

Tom: It means you can get some real-time three dee guidance on soft tissue movement during surgery without having to manually place any markers on the patient first.

Jane: That’s the big idea. The paper shows that this combination of temporal and stereo tracking works pretty robustly across different speeds, even when they tested it on chicken tissue.

Lu: They found that tracking on the chicken breast phantom was more reliable than their three dee printed surface because the biological tissue had better visual landmarks for those models to lock onto.

Tom: That’s interesting because it points to how texture and features matter a lot when you’re training these kinds of AI models.

Jane: Exactly. And even though they found errors up to about three point five millimeters at high speeds, the speed where that drops off is a key number we need to watch for.

Lu: The authors are being pretty honest about the limits, saying that selecting better points could improve things more, but just adding more points doesn't actually help and can even make it noisy.

Meng: So, what’s the practical reality here? Is this ready for a real operating room setup right now?

Tom: No. They admitted that the current tracking errors are still too high for actual clinical use today, so it’s definitely more of a research tool than something you can just plug in tomorrow.

Jane: It gives us a clear roadmap then: we need to focus on optimizing those point selections and finding ways to make the system lighter computationally before we see this in surgery.