Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning

The gist The proposed TopologyAware Multimodal Representation Alignment (ToMA) framework uses persistent homology to identify topologically salient edges and aligns them across modalities through available cross-modal correspondences, yielding stable gains in zero-shot classification and image-text retrieval

How it works

The ToMA framework integrates two stages of SemiCLIP [5] and aligns image and text representation spaces through topologically salient edges rather than through persistence diagram (PD)-level matching. First, under the 1-skeleton filtration by Eq. (1), the edge set is decomposed into two topologically distinct types: H0-death edges and H1-birth edges. The death simplices of H0 form the minimum spanning tree (MST), which captures connectivity structure, while the birth simplices of H1 are the non-MST edges that close cycles, capturing higher-order relational structure.

Second, ToMA aligns these edges across image and text using cross-modal correspondences. For an H0-death edge, the directional consistency is defined as T 0-death µ(Mi → Mj) = D x(µd)0 − x(µd)1, πi→j (x(µd)0) − πi→j (x(µd)1), where ⟨u, v⟩ denotes the cosine similarity between the two edge-direction vectors. Similarly, for an H1-birth edge, the directional consistency is defined as T 1-birth µ(Mi → Mj) = D x(µb)0 − x(µb)1, πi→j (x(µb)0) − πi→j (x(µb)1), where ⟨u, v⟩ denotes the cosine similarity between the two edge-direction vectors. The bidirectional ToMA loss is defined as L∗ToMA(Mi, Mj) = 1 − (T∗µ(Mi → Mj) + T∗µ(Mj → Mi)) for ∗ ∈ 0-death or 1-birth.

Key Components and Contributions

The framework leverages both H0-death edges and lightweight H1-birth edges, allowing it to capture both connectivity and cycle structure without constructing 2-simplices. The death simplices of H0 recover the minimum spanning tree (MST) of the embeddings, inducing a unique path between any pair of points. By emphasizing these topologically significant edges and aligning their directions rather than their Euclidean distances, ToMA preserves meaningful structural relations while remaining well matched to CLIP, whose geometry is governed by cosine similarity.

Experimental Results and Analysis

Experiments on remote sensing and fashion datasets show that ToMA generally improves over the baseline. The gains are most pronounced on remote sensing, where ToMA strengthens zero-shot classification under both in-distribution and distribution-shift settings while also improving imagetext retrieval across a wide range of settings. Additional analysis shows that ToMA is more stable than alternative topology-based objectives and that lightweight H1-birth edges provide useful higher-order structural signals. Specifically, the H1-only variant yields the strongest zero-shot result, reaching 86.3 at c = 0.5, suggesting that even the lightweight 1-dimensional signal carries informative structure beyond the connectivity information captured by H0. Furthermore, ToMA preserves a larger connected backbone that is consistently organized across image and text, including the chain 15 → 16 → 13 → 4 → 9 → 23 → 12 → 2 → 24.

Ablation and Robustness

Ablation studies confirm that ToMA is reasonably robust to the choice of the loss coefficient c, especially in the range c ∈ (0.1, 0.5, 1). The results indicate that both H0- and H1-based signals provide useful structural supervision. However, excessively large topology weights can be harmful, as performance tends to drop for retrieval across several variants when c = 2. The absolute value in Eq. (4) is a design choice that allows ToMA to focus on structural consistency of corresponding topological relations while remaining invariant to parallel and anti-parallel realizations of the same edge-induced relational subspace.

Conclusion and Broader Impact

ToMA provides a structural signal that yields stable gains in zero-shot classification and image-text retrieval, suggesting that topology-aware structural alignment is a promising direction for improving multimodal adaptation in specialized domains. The method's effectiveness does not rely on the default ViT-B/16 configuration, as it transfers across both convolutional and transformer-based encoders. The work suggests that topology-aware alignment is a promising direction for improving multimodal adaptation in specialized domains.

Limitations

Our evaluation is limited to specialized-domain semi-supervised vision-language adaptation settings, specifically remote sensing and fashion datasets. The study is restricted to cross-modal semi-supervised alignment. Furthermore, the method relies on a lightweight topological construction defined on the 1-skeleton, which may miss richer higher-order topological structure in the representation space.

Broader Impact

ToMA may help improve domain-specific retrieval and classification when labeled image-text pairs are scarce, which may support applications in domain-specific retrieval and classification. At the same time, stronger representation alignment may also be used in sensitive settings, especially in remote sensing. The method may inherit biases or limitations present in the underlying pretrained model and benchmark datasets, requiring careful evaluation before deployment in high-stakes settings.

Computational Efficiency

The proposed topology-aware alignment is computationally lightweight in practice, introducing almost no additional computational overhead over SemiCLIP. The total training time increases only marginally, from 799.5s to 802.0s under controlled settings.

Final Summary

ToMA is a topology-aware multimodal representation alignment framework for semisupervised vision-language learning that aligns persistent homology-selected edge directions across modalities rather than matching persistence diagrams directly. By leveraging H0-death and lightweight H1-birth edges under a 1-skeleton filtration, ToMA provides a structural signal that yields stable gains in zero-shot classification and image-text retrieval. The results suggest that topology-aware structural alignment is a promising direction for improving multimodal adaptation in specialized domains. It is important to note that the absolute value in Eq. (4) is a design choice that allows ToMA to focus on structural consistency of corresponding topological relations while remaining invariant to parallel and anti-parallel realizations of the same edge-induced relational subspace. The results suggest that H1 contributes most effectively when used together with H0, providing complementary higher-order structural signals that improve cross-modal alignment. It is important to note that the absolute value in Eq. (4) is a design choice that allows ToMA to focus on structural consistency of corresponding topological relations while remaining invariant to parallel and anti-parallel realizations of the same edge-induced relational subspace. The results suggest that H1 contributes most effectively when used together with H0, providing complementary higher-order structural signals that improve cross-modal alignment. The results suggest that the absolute value in Eq. (4) is a design choice that allows ToMA to focus on structural consistency of corresponding topological relations while remaining invariant to parallel and anti-parallel realizations of the same edge-induced relational subspace.

1. Bold header: Topology-aware representation alignment for semi-supervised vision-language learning. This system can improve generalization to specialized domains by aligning the global structural organization of image and text representation spaces rather than just pairwise embeddings, as stated in the abstract and introduction.

2. Bold header: Leveraging H1-birth edges for higher-order structural signals. The improved system can capture higher-order relations by utilizing lightweight H1-birth edges, which are described as capturing cycle structure without constructing 2-simplices.

3. Bold header: Domain-specific performance enhancement via ToMA and ToMA-domain variants. By applying the framework separately to each domain, the system can achieve superior results on specific datasets, such as achieving the best results on UCM-CLS and AID in remote sensing settings.

4. Bold header: Robustness across model architectures using ViT-B/16 as a backbone. The improved system demonstrates that the effectiveness of ToMA does not rely on the default ViT-B/16 configuration, showing that the alignment transfers across both convolutional and transformer-based encoders.

5. Bold header: Invariance to edge orientation via absolute value in loss function. By using the absolute value in Equation (4), the system ensures it aligns the underlying one-dimensional relational subspace induced by a topologically selected edge, rather than enforcing a fixed global orientation across modalities.