Geometry-Aware Camera Localization for Bronchoscopy

The paper, authored by Lumin Chen, Qingyao Tian, Jinpeng Li, Haoyu Jiang, Huai Liao, Xinyan Huang, Hongbin Liu, and Dong Yi, and published in the Proceedings of the 34th ACM International Conference on Multimedia (MM ’26), addresses the problem of camera localization in bronchoscopy. As the abstract states: Camera localization in bronchoscopy remains a challenging problem due to stringent accuracy requirements, real-time constraints, and limited training data. The authors highlight that "Compared to natural scenes, the confined anatomical structures demand millimeter-level precision, while intraoperative guidance necessitates low-latency inference. However, existing methods often fail to effectively exploit preoperative geometric priors, limiting their robustness and accuracy."

The paper identifies the unique challenges of bronchoscopic localization: "achieving reliable localization in bronchoscopy is inherently challenging due to the complex and deformable anatomical structures, the prevalence of repetitive and low-texture surfaces, and the extremely limited field of view. These factors make it difficult to establish stable visual correspondences, while the small spatial scale of the airway further imposes stringent requirements on localization accuracy and real-time performance."

The authors argue that general camera localization methods are ill-suited for this domain: "Most approaches rely on large-scale annotated datasets and visual features learned from natural images, which suffer from severe domain gaps when applied to medical imagery. In addition, techniques designed for outdoor or indoor environments often assume rich textures, wide fields of view, and static scenes, conditions that do not hold in endoscopy."

To address these limitations, the authors propose a unified geometry-aware bronchoscope localization framework (GABL) that effectively fuses preoperative structural priors with paired intraoperative video to estimate 6-DoF camera poses. The core idea is to systematically inject geometric priors at three complementary scales: structure, motion, and appearance. The framework consists of three main components: (1) anchor-based coarse-to-fine localization using a graph-guided scheme, (2) a Transformer-based tracking model for temporal dynamics, and (3) a novel RGB-depth matching objective for appearance-geometry consistency.

The method constructs geometric priors offline from preoperative CT images. "We use the centerlines extracted from airway models in the dataset as the skeleton representation of the bronchi. Due to the narrow tubular structure of the airways, the centerline closely approximates the bronchoscope trajectory. The authors uniformly sample a set of anchor points along the skeleton and construct an anchor graph based on the topological structure of the bronchial tree, ensuring coverage of key anatomical regions such as the carina and multi-level bifurcations."

For each anchor point, camera poses are generated to simulate bronchoscope navigation. "We fix the camera center t at each anchor point and define the target point as its adjacent node in the tree structure. Under this look-at constraint, the viewing direction is uniquely determined, which constrains two degrees of freedom of the rotation R. As a result, the rotation space is reduced from three degrees of freedom to one, with the remaining degree corresponding to roll. Depth maps are then rendered by projecting the 3D geometry under the given camera pose using a rasterization pipeline with z-buffering. The authors emphasize an advantage over learned depth estimators: Compared to Endo-FASt3r and DARES, which rely on deep learning models for depth estimation, our approach eliminates the domain gap introduced by learned predictors while maintaining pixel-wise geometric consistency with the camera pose."

The framework learns a unified embedding space using modality-specific encoders. Camera and anchor poses are encoded via a multi-layer perceptron (MLP) with three linear layers, while depth maps are processed by a ResNetV2-18 backbone. RGB frames are encoded using a ResNetV2-50 backbone. The airway anchor tree is represented as an undirected graph G = (V, E) where nodes correspond to anchor points and edges encode their topological connectivity. Each node is initialized by combining anchor pose and depth embeddings, then processed through a three-layer graph convolutional network (GCN) to propagate information over the graph, with node updates following the formula: z comb(l+1) i = Σ j∈N(i) (1/√(d i d j)) W(l) z comb(l) j. This enables each anchor to aggregate information beyond its local neighborhood, capturing the hierarchical structure of the airway.

For temporal dynamics, the authors employ a causal Transformer (pruned from Qwen3 backbone) consisting of 81.82M parameters, trained from scratch. A key design element is stochastic pose dropout: a binary mask m ∈ 0,1 n×s is sampled from a Bernoulli distribution with p = 0.25 retention probability. The authors explain: This stochastic pose dropout prevents the model from overfitting to precise pose cues and encourages robust temporal modeling based primarily on RGB observations, effectively regularizing motion dynamics learning.

Anchor Localization: The model uses a two-stage strategy. In coarse localization, we compute the similarity logits between the frame embedding and all anchor embeddings via inner product, formulated as a classification problem with cross-entropy loss. The predicted anchor is the one with the maximum logit. In fine localization, the model predicts a refined pose by the Pose Regressor using an uncertainty-weighted MSE loss following Kendall et al.: L fine = (1/2N) Σ [T̂ i − T i2 · exp(−log σ2 fine,i) + log σ2 fine,i]. The authors note several advantages: the coarse stage transforms the continuous pose estimation problem into a discrete anchor classification task, which significantly reduces the search space and stabilizes training, while the fine stage refines the coarse estimate to achieve precise pose prediction. They further emphasize that the integration of geometric priors at multiple stages improves both translational and rotational accuracy, while effectively mitigating error accumulation over long sequences.

Temporal Motion Tracking: Given the high frame rate of bronchoscopic video, the motion between adjacent frames is typically small. The Pose Tracker predicts a relative pose offset ΔT i, composed with the previous pose: T̂ track i = T i−1 ⊕ ΔT i, supervised by an uncertainty-weighted MSE: L track = (1/2N) Σ [T̂ track i − T i2 · exp(−log σ2 track,i) + log σ2 track,i].

Appearance-Geometry Matching: This module aligns RGB and depth features, introducing intraoperative geometric supervision into the localization framework. With a 50% probability, the corresponding ground-truth depth map is selected as a positive sample; otherwise, a random frame is selected as a negative candidate. Instead of hard labels, we define soft labels based on pose similarity: y match i,j = exp(−T i − T j22), which assigns higher scores to geometrically close frames. The objective uses binary cross-entropy: L match = −Σ [y match log σ(s match) + (1 − y match) log(1 − σ(s match))].

Inference Strategy: During inference, the model takes only RGB frames as input. The detector provides a geometry-aware pose estimate but may exhibit frame-to-frame jitter because of sparse anchors, whereas the tracker produces smoother, temporally consistent estimates but may accumulate errors over time. The method selects between the two predictions based on their discrepancy: "If the discrepancy exceeds an adaptive threshold, the tracked pose is used; otherwise, the detected pose is selected and the threshold is reset. The threshold increases linearly during consecutive tracking steps, encouraging a timely return to detection and preventing long-term drift."

The method is evaluated on the BREATH benchmark dataset, which contains 66 bronchoscopy procedures, comprising 148,926 frames with 6-DoF pose annotations, along with airway meshes reconstructed from preoperative CT scans. The dataset is split at the case level (56 training, 10 testing) to avoid data leakage. Evaluation uses Absolute Trajectory Error (ATE) decomposed into translation (mm) and rotation (degrees), along with SR-5 and SR-10 success rates (percentages of frames with translational errors below 5 mm and 10 mm, respectively).

Implementation details: We apply farthest point sampling to select 512 anchor points along the airway skeleton, and construct a directed graph based on the topology of the bronchial tree. Depth maps are rendered with PyTorch3D. Training uses two NVIDIA A100 GPUs with per-GPU batch size of 6, video clips of 64 frames, images resized to 256×256, AdamW optimizer with learning rate 1×10−3, all losses weighted 1:1:1:1, trained for 100 epochs. Data augmentation includes random skipping of 1 to 5 frames and a 50% probability of temporal reversal.

Main results: GABL achieves an ATEtrans of 7.01 mm and an ATErot of 29.56°, improving over the second-best method by 8.37% and 31.76%, respectively. Its SO(3) axis-wise rotation errors are 13.36°, 14.48°, and 15.39°. GABL achieves SR-5 and SR-10 scores of 61.04% and 83.66%.

Comparing with prior methods:

• 3DGS-based methods: MonoGS (28.80 mm, 105.17°), EndoGSLAM (26.16 mm, 73.59°). These tend to suffer from error accumulation due to the absence of explicit geometric constraints.

• Depth-based methods: Endo-FASt3r (22.65 mm, 79.15°), VNB (43.52 mm, 109.22°), DARES (44.58 mm, 109.46°). These provide limited structural guidance, which results in suboptimal performance in complex bronchial environments.

• Landmark-based methods: PANSv2 (10.17 mm, 44.49°), BREATH-VL (7.65 mm, 43.32°). These rely on sparse or localized cues, which restrict their effectiveness in regions far from detected landmarks.

In contrast, GABL "leverages dense geometric priors by uniformly sampling anchor points along the entire airway skeleton. This design provides global structural coverage, continuous geometric supervision along the trajectory, and robustness in regions lacking distinctive landmarks."

Inference speed: On an NVIDIA RTX 4090 GPU, our model achieves 33.6 FPS, significantly outperforming PANSv2 and BREATH-VL, which run at 8.5 FPS and 5.6 FPS, respectively, while EndoGSLAM achieves only 2.7 FPS. The method thus achieves a 4× to 12× speedup over prior approaches, while maintaining superior localization accuracy, satisfying the real-time requirements of clinical endoscopy.

Ablation studies:

The authors validate four key components. Removing the GCN (replacing with a linear layer and removing the anchor tree input) increases ATEtrans to 10.67 mm, showing that the tree-structured airway representation provides meaningful global geometric priors. Removing the Pose Regressor increases ATErot dramatically to 107.54°, indicating that coarse localization alone provides limited orientation information, and fine-grained geometric refinement is necessary for accurate pose estimation. Removing the Pose Tracker increases ATEtrans by 9.47 mm to 16.48 mm, showing that temporal geometric constraints are essential to prevent error accumulation along the trajectory. Removing the Matcher degrades performance to 7.63 mm and 30.86°.

For data augmentation, the ablations show reductions in ATEtrans of 0.70 to 0.96 mm: "Random clip skipping exposes the model to temporally discontinuous inputs, improving robustness to irregular motion. Temporal reversal simulates backward navigation of the bronchoscope, enabling the model to better handle bidirectional motion patterns."

For pose dropout rate in the causal Transformer, a rate of 0.75 achieves the best performance (7.01 mm, 29.56°). "Lower dropout rates result in a performance drop of 0.58 to 1.03 mm in ATEtrans, suggesting insufficient regularization. When the dropout rate is set to 1 (i.e., pose information is completely discarded), the model's performance declines, suggesting that combining pose information with RGB data is more effective for motion supervision."

The paper concludes: "We present GABL, a geometry-aware localization framework for bronchoscopy that systematically integrates geometric priors at three complementary scales: structure, motion, and appearance. By combining anchor-based coarse-to-fine localization, temporal motion modeling, and cross-modal appearance-geometry supervision, our method unifies preoperative and intraoperative information within a single framework. Extensive experiments on the BREATH dataset demonstrate that GABL achieves a mean ATEtrans of 7.01 mm and a mean ATErot of 29.56°, with a success rate (SR-5) of 61.04%, outperforming existing methods while maintaining real-time inference performance. These results highlight the effectiveness of multi-scale geometric constraints for robust camera localization in challenging endoscopic environments and suggest promising potential for broader applications in medical and robotic navigation."

The work was supported by the InnoHK initiative of the Government of the Hong Kong Special Administrative Region.

Improvements to AI systems:

1. Inject multi-scale geometric priors into visual localization.

• Replace purely learned visual features with precomputed 3D structure (e.g., CT-derived airway centerlines) by sampling anchor points along the anatomical skeleton and building a topological graph.

• Encode anchor poses and rendered depth maps into a unified embedding space, then propagate information over the graph with a GCN to capture global bronchial-tree context.

• This eliminates the need for large RGB-only training sets and reduces domain shift from natural-image pretraining.

2. Split pose estimation into coarse anchor classification + fine pose regression.

• Formulate coarse localization as a discrete classification over anchor nodes to reduce search space and stabilize training.

• Add a Pose Regressor that refines the coarse anchor pose with uncertainty-weighted MSE loss, recovering fine-grained translation and full rotation (including roll).

• Use this two-stage design to suppress jitter and improve orientation accuracy—especially important in low-texture, repetitive airway surfaces.

3. Learn temporal motion dynamics with a causal Transformer and stochastic pose dropout.

• Process RGB video clips with a temporal model that predicts relative pose offsets between adjacent frames, composed with the previous pose to produce smooth tracking.

• Apply Bernoulli-mask pose dropout during training so the model relies mainly on RGB observations, preventing overfitting to exact pose cues and regularizing motion dynamics.

• Use a hybrid inference strategy: if the tracked pose diverges from the detected pose beyond an adaptive threshold, fall back to the detected pose and reset the threshold; otherwise, trust the smoother tracking output. This avoids long-term drift while retaining per-frame accuracy.

4. Add cross-modal appearance-geometry matching supervision.

• Align RGB frame embeddings with rendered depth (geometry) embeddings using a binary cross-entropy objective with soft labels based on pose similarity.

• Alternate positive/negative depth samples during training so the representation learns to associate visual appearance with geometric configuration, improving robustness in ambiguous views.

• This enforces intraoperative geometric consistency without requiring predicted depth, eliminating depth-estimation domain gaps.

5. Leverage topology-aware anchor sampling and graph propagation.

• Use farthest point sampling over the airway skeleton to get uniformly distributed anchors, and construct a directed graph reflecting bronchial-tree connectivity.

• Apply symmetric normalization in GCN updates so each anchor aggregates context from beyond its local neighborhood, covering carina and multi-level bifurcations.

• Provide dense geometric priors along the entire trajectory rather than sparse landmarks, enabling robust localization even far from distinctive anatomical features.

What the improved AI system can do:

• Real-time 6-DoF camera localization in bronchoscopy at 33.6 FPS, achieving 7.01 mm mean translation error and 29.56° rotation error, with 61.04% success rate under 5 mm—outperforming prior 3DGS-based, depth-based, and landmark-based methods by 8%–32% while running 4–12× faster.

• Navigate endoscopes autonomously through complex, low-texture, narrow airways by combining coarse anatomical context from a pre-operative CT graph with smooth temporal tracking from live video.

• Maintain accurate pose even with noisy, incomplete, or repetitive visual input (e.g., near bifurcations or featureless regions) by exploiting geometric priors and cross-modal appearance-geometry matching.

• Recover from tracking drift: the adaptive detector-tracker selection resets the system to reliable anchor-based poses when temporal error accumulates, enabling long-duration procedures without global optimization.

• Transfer to other tubular or confined environments (e.g., colonoscopy, ureteroscopy, catheter navigation, endovascular robotics) where millimeter-level accuracy, low latency, and sparse texture are critical, and where CT/MR-derived centerlines are available.

• Train efficiently with limited annotated video by using rendered geometry from preoperative scans as self-supervision, reducing dependence on large labeled datasets.

Camera localization in bronchoscopy remains a challenging problem due to stringent accuracy requirements, real-time constraints, and limited training data. Compared to natural scenes, the confined anatomical structures demand millimeter-level precision, while intraoperative guidance necessitates low-latency inference. However, existing methods often fail to effectively exploit preoperative geometric priors, limiting their robustness and accuracy. To address these limitations, we propose a unified geometry-aware bronchoscope localization framework (GABL) that effectively fuses preoperative structural priors with paired intraoperative video to estimate 6-DoF camera poses. Specifically, to address visual ambiguity in complex airways, we propose a graph-guided coarse-to-fine localization scheme that effectively leverages structural priors for precise pose estimation. Furthermore, to mitigate pose jitter and bridge the visual-structural gap, we integrate a Transformer-based tracking model with a novel RGB-depth matching objective, jointly enforcing spatio-temporal and geometric consistency. Extensive experiments demonstrate that our method yields remarkable reductions of 8.37% and 31.76% in translation and rotation errors over the prior state-of-the-art, alongside 4 times inference speedup (33.6 FPS) for robust real-time bronchoscope localization. Project website: https://paulili08.github.io/GABL/.