TALON: Token-Aligned Lightweight Adapters for 6-DoF Spacecraft Pose Estimation

summary

The gist

Monocular 6-DoF spacecraft pose estimation methods predominantly process individual frames, discarding the temporal information present in an image sequence acquired during spacecraft manoeuvres…

In short

TALON introduces spatiotemporal 3D adapters injected before self-attention layers of a frozen Vision Transformer to estimate 6-DoF spacecraft pose from monocular video. It uses a patch-token alignment loss, conditioned on keypoint structure, to geometrically ground the adapted features. This method significantly reduces pose error by aligning learned features with the object's actual 3D keypoint layout, achieving strong performance with minimal parameter overhead.

Key concepts

Spatiotemporal 3D Adapters
These are lightweight modules injected into the last Transformer blocks of a frozen vision model. They are designed to process tokens that carry both spatial information (where they are in the image) and temporal information (how they change across frames). This allows the frozen attention mechanism to reason over context that is enriched by sequential data, improving temporal coherence.
Patch-Token Alignment Loss
This loss function guides the adapted tokens to encode specific geometric keypoint structures. It works by creating a Gaussian geometric prior around visible keypoints and minimizing the KL divergence between this prior and the token distribution's cosine similarity to a prototype derived from local tokens. This forces the features to spatially represent where keypoints are located.
Prototype-Conditioned KL-Divergence Objective
This specific loss objective is used within the alignment loss to enforce geometric grounding. It compares the learned token representation against a prototype derived from local tokens, using a Kullback-Leibler (KL) divergence measure. This technique ensures that each keypoint induces a spatially precise activation in the token field, anchoring predictions to 3D structure.
Frozen DINOv3 Backbone
The model uses a pre-trained vision transformer backbone, specifically DINOv3, which is kept frozen during training. Instead of retraining the entire massive network, TALON introduces only small adapters on top of this foundation. This leverages the powerful general feature extraction capabilities already learned by DINOv3 while efficiently adding task-specific knowledge.

Terminology used across episodes

This episode discusses

The paper

TALON: Token-Aligned Lightweight Adapters for 6-DoF Spacecraft Pose Estimation · Read on arXiv

Abid Ali Arunkumar, Rathinam Djamila, Aouada

Monocular 6-DoF spacecraft pose estimation methods predominantly process individual frames, discarding the temporal information present in an image sequence acquired during spacecraft manoeuvres. Few temporal approaches require full backbone fine-tuning or auxiliary optical flow networks, risking catastrophic forgetting or increasing computational cost, respectively. We propose TALON (Token-Aligned Lightweight adapters for Orbital Navigation): spatiotemporal 3D adapters injected before the self-attention layers of a frozen ViT vision transformer, combined with a patch-token alignment loss that geometrically grounds the adapted features to keypoint structure through a prototype-conditioned KL-divergence objective. Pre-attention placement allows the frozen attention to reason over temporally enriched tokens, achieving stronger performance with a single adapter per block than post-attention alternatives. The alignment loss shapes the intermediate representations so that each keypoint induces a spatially precise activation in the token field, while the framework adds less than 5% parameters to the frozen backbone. On SPADES dataset, TALON reduces the pose error by 50% over the prior state-of-the-art, and on SwissCube dataset it surpasses the prior best by 21.8% in ADD-0.1d accuracy. Zero-shot cross-domain evaluation from sim-to-real on SPARK real data reduces pose error by 4.7x, and ablations characterise the role of adapter depth across in-domain and cross-domain settings.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "TALON: Token-Aligned Lightweight Adapters for 6-DoF Spacecraft Pose Estimation".

Jane: Here's the gist of the paper. Monocular 6-DoF spacecraft pose estimation methods predominantly process individual frames, discarding the temporal information present in an image sequence acquired during spacecraft manoeuvres Estimating the six degree-of-freedom (6D) pose of a non-cooperative spacecraft from monocular imagery is a core capability for autonomous rendezvous, proximity operations targeting on-orbit servicing, and active debris removal missions The framework proposes TALON (Token-Aligned Lightweight adapters for Orbital Navigation): spatiotemporal 3D adapters in injected before the self-attention layers of a frozen ViT vision transformer, combined with a patchtoken alignment loss that geometrically grounds the adapted features to keypoint structure through a prototype-conditioned KL-divergence objective Core Problem Addressed The central problem addressed in this work is bridging the gap where foundation model patch tokens are trained to be general-purpose descriptors and they do not encode the specific geometric keypoint structure of a target object This is compounded by two challenges in the spacecraft domain: most methods process each frame independently, discarding sequential structure, and annotated spacecraft data remains scarce, making parameter-efficient adaptation essential Spatiotemporal Adaptation Mechanism The model adapts a frozen DINOv3 vision transformer through two co-trained components: spatiotemporal 3D adapters and a patch-token alignment loss The spatiotemporal 3D adapters are injected before multi-head self-attention in the last L transformer blocks, allowing the frozen attention to reason over temporally enriched tokens This design allows for a single adapter per block, adding less than 5% parameters to the frozen backbone Geometric Grounding via Alignment Loss The patch-token alignment loss supervises the adapted tokens to encode keypoint geometry during training This loss constructs a Gaussian geometric prior around each visible keypoint, builds a prototype from local tokens, and minimizes the KL divergence between the prior and the cosine-similarity distribution of tokens to that prototype The combined token loss is Ltok = Lalign + wdiv Ldiv, which shapes the adapted patch tokens into a spatially precise representation of keypoint geometry Performance and Generalization Results Experiments on SPADES, SPARK, and SwissCube confirm the effectiveness of this design On SPADES, TALON reduces the pose error by 50% over the prior state-of-the-art On SwissCube, TALON attains 78.8% ADD-0.1d accuracy, surpassing the prior best by 21.8 percentage points Zero-shot transfer to the SPARK real subset reduces the pose error by 4.7× relative to the prior state-of-the-art The alignment loss contributes most at fine spatial scales and on challenging sequences, acting as a geometric regulariser that anchors predictions to the 3D structure of the target rather than to domain-specific appearance Ablation Insights Ablations characterize the role of adapter depth across in-domain and cross-domain settings In-domain performance scales monotonically with adapter depth (L=2 → 4 → 8), while the alignment loss preserves transferability across depths The small absolute differences between configurations suggest that adapter depth acts primarily as an in-domain accuracy knob in this framework Additionally, the pre-attention injection placement achieves the lowest pose score, with both rotation and translation errors below those of the post-MHSA configuration The alignment loss roughly halves the pose score (0.1896 → 0.0777), with the rotation error falling more sharply than the translation error The approach keeps a frozen DINOv3 vision transformer as the backbone and introduces two complementary mechanisms: spatiotemporal 3D adapters injected before self-attention for temporal reasoning, and a KL-divergence patch-token alignment loss that spatially grounds the adapted features to keypoint structure The framework successfully optimizes temporal coherence and geometric precision together within a single module Several directions remain open, including extending the approach to jointly estimate detection and pose or handling unknown geometries through category-level priors The relationship between adapter depth and cross-domain generalisation also warrants further study Finally, integrating the token alignment objective with other dense prediction tasks, such as segmentation or depth estimation, could yield a more complete geometric understanding of the scene beyond keypoint localisation Page 1 --- TALON: TOKEN-ALIGNED LIGHTWEIGHT ADAPTERS FOR 6-DOF SPACECRAFT POSE ESTIMATION Abid Ali Arunkumar Rathinam Djamila Aouada Pruning redundant reasoning tokens Estimating the six degree-of-freedom (6D) pose of a non-cooperative spacecraft from monocular imagery is a core capability for autonomous rendezvous, proximity operations targeting on-orbit servicing, and active debris removal missions How the training signal is built The total loss combines four terms, each targeting a distinct level of the prediction hierarchy: Ltotal = Lkp + λhm Lhm + λtok Ltok + λsmooth Lsmooth The loss weights are λhm=1.0, λsmooth=0.3, and λtok=0.3 with gradual reduction to 0.1 (applied after warmup) Where the gains come from The largest gain appears at medium range, where the spacecraft is sufficiently resolved to expose its geometric structure, but small enough that fine keypoint features are difficult to recover from appearance cues alone The alignment loss has the most leverage when the per-token receptive field covers a small spatial extent of the object, producing a sharper heatmap peak than a feature that merely encodes generic appearance Ablation Insights The pre-attention injection achieves the lowest pose score, with both rotation and translation errors below those of the post-MHSA configuration Enabling the loss roughly halves the pose score (0.1896 → 0.0777), with the rotation error falling more sharply than the translation error The frozen DINOv3 baseline quantifies the complementary contribution, with a pose score more than four times that of the full model The framework assumes a known 3D keypoint model and a bounding-box crop, both of which may be unavailable in unconstrained operational settings Extending the approach to jointly estimate detection and pose, or to handle unknown geometries through category-level priors, would broaden its applicability The relationship between adapter depth and cross-domain generalisation also warrants further study Finally, integrating the token alignment objective with other dense prediction tasks, such as segmentation or depth estimation, could yield a more complete geometric understanding of the scene beyond keypoint localisation Page 4 --- TALON Pruning redundant reasoning tokens The backbone is a frozen DINOv3

37: .

Tom: To introduce temporal reasoning without altering pre-trained weights, we inject lightweight adapters into the last L blocks, operating exclusively on patch tokens, as illustrated in Fig.

Paper summary: Tom: So, to recap, this paper introduces TALON—Token-Aligned Lightweight Adapters for 6DoF Spacecraft Pose Estimation <ref:2605.31217#pg1,TALON—Token-Aligned Lightweight Adapters for>. The central thesis is addressing the fact that most monocular methods discard temporal information from image sequences.

Jane: They claim that by injecting spatiotemporal three dee adapters before self-attention and using a specific patch-token alignment loss, you can achieve better pose estimation while keeping the model lightweight, adding less than five percent parameters to a frozen vision transformer <ref:2605.31217#pg1>.

Lu: The core mechanism involves two parts: the temporal adaptation via those adapters to let attention reason over temporally enriched tokens, and the geometric grounding through that KL divergence objective.

Meng: So it’s not just one trick; it’s a combination of adapting for time and forcing geometric structure onto the tokens simultaneously? That sounds complex to implement reliably.

Tom: It is complex, but they claim this pre-attention placement allows the frozen attention to reason over temporally enriched tokens, which is stronger than post-attention alternatives.

Jane: And on top of that, they manage to shape the intermediate representations so each keypoint induces a spatially precise activation in the token field. That’s how it grounds those features geometrically.

Lalam: The main contribution they highlight are the token alignment loss and how it builds a Gaussian spatial prior around each visible keypoint, pooling nearby tokens to create a prototype.

Tom: So, this isn't just about adding temporal adapters; it’s about training the temporal adaptation and geometric grounding together through that combined token loss.

Jane: That synergy is what they argue makes it work better than doing those two things separately. It’s about learning the temporal context *and* the geometry at the same time, which is what they call their main contribution.

Conclusion: Tom: So we're wrapping up TALON, this paper by Abid Ali Arunkumar Rathinam Djamila Aouada. The title itself is quite descriptive of what it’s doing: Token-Aligned Lightweight Adapters for 6DoF Spacecraft Pose Estimation <ref:2605.31217#pg1,Token-Aligned Lightweight Adapters for>.

Jane: In simple terms, the paper proposes a method that takes a large vision transformer backbone and adds very small, specialized layers to handle both time and geometry simultaneously.

Lu: It’s about efficiently injecting those modules before the self-attention layers to let the frozen attention reason over tokens that already carry temporal context.

Meng: So for anyone listening, this means we can get better pose estimates from single images by using this specific training signal, without needing to re-train the entire massive backbone.

Tom: That's right. It’s about leveraging pre-trained weights while adding a small amount of specialized knowledge to solve the 6DoF pose problem efficiently <ref:2605.31217#pg1>.

Jane: And it opens up possibilities for autonomous rendezvous and proximity operations by giving us that geometric precision in unconstrained environments.

More episodes

← Home