Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging".
Jane: Here's the gist of the paper. Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology We propose leveraging this crossinstance topological consistency as a supervisory signal The Core Proposal The key idea in this paper is to ensure that if two patches belong to the same region in scan A, their corresponding features from scan B should also co-cluster as the same region in B’s representation space This form of supervision goes beyond traditional instance-level learning: instead of focusing solely on the appearance of individual structures, the model learns the relative positioning of structures within their broader anatomical context To achieve this, we use a dual alignment approach: intra-instance and inter-instance consistency How the Training Signal is Built The overall training objective can be formulated as Ltotal = Luni + Lintra + Linter where Lintra and Linter will be introduced in the following part of this section The intra-instance loss Lintra is described in Section 4.1, while the inter-instance loss Linter is outlined in Section 4.2 Intra-instance Constraint (Lintra) involves a cross-modal triplet objective that preserves local neighborhood topology within the same individual across modalities Specifically, for a given p-th patch feature in Z h i, we construct triplet sets in modality j for contrastive loss as follows The final intra-instance loss is Lintra = 1/M(M − 1) Sum i,j∈M Lintra i→j intra Inter-instance Constraint (Linter) uses unsupervised Mutual Nearest Neighbor (MNN) search to establish pseudo-correspondences, ensuring partial neighborhood alignment across instances The inter-instance loss Linter is defined as Linter = 1/MH(M−1)(H-1) Sum i,t∈M Sum h,g∈H Linter i→t,h→g inter Where the Gains Come From The results show that our method effectively clusters feature tokens by anatomical identity across different individuals and modalities, which aligns well with our intended goal In contrast, baseline self-supervised methods tend to cluster features by individual identity rather than anatomical correspondence, indicating that they capture instance-specific patterns rather than generalizable anatomical structures The t-SNE visualization of token-level representations across patients shows TACO achieves clear and well-organized clusters across patients, demonstrating superior cross-instance learning Robustness and Efficiency Our method achieves state-of-the-art (SOTA) performance and exhibits significantly better robustness in missing-modality scenarios, where some imaging sequences are unavailable at test time Table 7 shows that the Hard-neg–Pos gap is 0.0441, which illustrates the strong separation between positive and negative samples Furthermore, TACO demonstrates superior performance in average over all settings, with 81.09% (TC), 66.27% (ET), and 90.09% (WT) The computational complexity comparison shows that TACO, which only has a small decoder head and does not include teacher-student structures, demonstrates the best performance in terms of parameters and TFLOPS This work introduces a novel approach to learning topological consistency in brain structures across instances, ex8 tending beyond traditional instance-level self-supervision through the proposed IM-NRC principle This could potentially reduce reliance on expensive expert annotations, improve data efficiency, and enable more robust downstream performance in clinically relevant tasks such as segmentation, detection, and disease classification The final work is 2605.14654.
Tom: Page 1 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology We propose leveraging this crossinstance topological consistency as a supervisory signal Page 2 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Prior: instance-level different instances different modalities (a) Comparison with prior methods (b) Topological consistency in brain anatomy Page 3 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Subsequent topology-aware methods primarily target tubular structures (Wen et al., 2025; Elsayed et al., 2025) However, very few works extend topological consistency to the unsupervised/self-supervised setting, despite the fact that human anatomy exhibits an invariant topological structure across patients Page 3 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Assumption 3.1 (IM-NRC).
Jane: Given patch token z h i,k ∈ Z h i, if z h i,k is located in decision region Rα i,h =
z h i σi,h(z h i) = α: , then z g j,k = C h→g i→j (z h i,k) should be in region
C h→g i→j (z h i) z h i ∈ Rα i,h: ⊆ Rα j,g =
z g j σj,g(z g j) = α: , where σi,h and σj,g are classifiers or discriminators Page 4 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging The reconstruction objective is Luni = xˆ h i m − x h i m2 / 2 Page 4 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Figure 2.
Tom: Pipeline of the proposed method Page 5 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Triintra i→j, a, h:= n(a',p',n') Page 6 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Lintra = 1/M(M − 1) Sum i,j∈M Lintra i→j intra Page 7 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Linter = 1/MH(M−1)(H-1) Sum i,t∈M Sum h,g∈H Linter i→t,h→g inter Page 8 --- Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging Figure 5.
Paper summary: Tom: So we're looking at this paper, "Beyond Instance-Level Self-Supervision in three dee Multi-Modal Medical Imaging <ref:2605.14654#pg1,Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging>." Basically, the authors are saying that current self-supervised methods treat every patient as totally separate, and they don't use the real key feature of medical scans: that structures stay in consistent places across different people.
Jane: That makes sense for a start. So the core idea here is to use this cross-instance topological consistency as a supervisory signal instead of just looking at how one individual's scan looks on its own.
Lu: Exactly. The main claim is that if two patches belong to the same region in one scan, their features from another scan should also group together as the same region in the second representation space. That moves beyond just learning individual structures and focuses on where things are relative to each other anatomically across patients and modalities.
Meng: So, what's the mechanism they use to enforce this spatial relationship? Because if it's just a general consistency check, it might not be strong enough for something like brain anatomy.
Tom: They use a dual alignment approach. First, they have an intra-instance constraint which preserves local neighborhood topology within the same person across different modalities. Then they have an inter-instance constraint to align things across different patients and modalities.
Jane: That sounds like two layers of supervision to make sure things are consistent both inside one scan and between scans. So, how do they build that intra-instance signal?
Lu: The intra-instance loss involves a cross-modal triplet objective that explicitly preserves local neighborhood topology for a given patch token in one modality against features from another modality. They define this as Lintra using the summation over all modalities and instances, which is defined as one over the size of the set times the difference between that size and one <ref:2605.14654#pg1>.
Meng: That sounds computationally intensive. How do we know if this actually helps them learn anatomy instead of just memorizing pixel colors?
Tom: The results show they effectively cluster feature tokens by anatomical identity across different individuals and modalities, which is what they aimed for. They found that baseline methods tend to cluster features by individual identity, but this method clusters them by actual anatomical correspondence.
Jane: That's a big difference there. So when you look at the t-SNE visualization of token-level representations across patients, what does it show?
Lu: The t-SNE visualization shows that TACO achieves clear and well organized clusters across patients, which demonstrates superior cross-instance learning compared to other methods.
Meng: So if we're talking about practical deployment, how robust is this thing when we don't have all the data we need for every modality? Does it break easily?
Tom: It shows better robustness in missing-modality scenarios. Table seven shows the hard negative to positive gap is zero point zero four four one, which means there's a strong separation between what should be matched and what shouldn't be matched. Plus, they show high performance across different settings: eighty-one point zero nine percent for TC, sixty-six point two seven percent for ET, and ninety point zero nine percent for WT.
Jane: That means it holds up well even when some imaging sequences are unavailable at test time, which is really important for real-world clinical use where data is often incomplete or mixed. What about the computational cost?
Tom: They also compared the complexity and TACO ends up with a very small decoder head and no teacher student structures, which means it shows the best performance in terms of parameters and flops.
Lu: So they achieved this by focusing on topological consistency rather than just trying to match every pixel directly. It’s a different kind of supervision altogether, moving from appearance matching to structural context matching.
Meng: If we think about what this means for a production system, it suggests that instead of training models on one patient and hoping they transfer well, we can train them to understand the underlying spatial structure of the anatomy itself.
Jane: It could potentially reduce the need for expensive expert annotations because it’s learning inherent anatomical rules rather than just memorizing specific examples. That's a big efficiency gain for downstream tasks like segmentation or disease classification.
Tom: Right, so this paper is proposing a way to inject anatomical knowledge directly into the self-supervised pre-training process through these intra and inter instance constraints. It's really about teaching the AI where structures belong relative to each other, not just what they look like in isolation.
Lu: The IM-NRC principle they introduce is essentially formalizing how we expect features from different images to relate based on their anatomical location, which is a way of imposing structure onto the learning process.
Meng: If this holds up when we deploy it, it means we could get much better performance in those tricky missing-modality situations where traditional methods usually fall apart.
Jane: So, to wrap up this paper "Beyond Instance-Level Self-Supervision in three dee Multi-Modal Medical Imaging," the authors introduce a method that treats cross-instance topological consistency as a supervisory signal for self-supervised pre-training in medical imaging <ref:2605.14654#pg1,Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging>.
Tom: And what it means is moving away from just instance-level learning to learning the relative positioning of structures within their broader anatomical context across different scans. It seems to offer strong performance and better robustness in real clinical settings.
Conclusion: Jane: So we’ve been looking at how this new self-supervised method works, and now we're getting to where they land with their final thoughts on "Beyond Instance-Level Self-Supervision in three dee Multi-Modal Medical Imaging."
Tom: Yeah, the title is pretty clear. They’re moving past just treating each scan like a totally separate thing.
Lu: Exactly. The authors are proposing this idea that you can use the consistent spatial relationships between structures across different patient scans and different imaging types as a guide to learning.
Jane: So what does that actually mean in simple terms for someone who isn't deep in the math?
Tom: It means instead of just training a model to recognize one specific brain shape, you’re teaching the AI where things *should* be located anatomically, regardless of whether it's patient one or patient ten.
Lu: Right. They’re using this idea—which they call the IM-NRC principle—to ensure that if a patch is in one spot in one scan, its feature representation will be in the same relative spot in another scan too.
Jane: That sounds like it could really help with making medical AI more reliable when doctors are dealing with mixed data or incomplete scans.
Tom: It’s about making the learning process robust to missing information because it learns structure first, not just memorized patterns.
Lu: And the results they showed, especially how well it clusters features across different patients in that t-SNE visualization, support that idea of capturing this general anatomical context.
Jane: It seems like they’ve managed to build a system that learns the underlying rules of brain organization instead of just learning specific examples.
Tom: Which is exactly what you want for anything from disease classification to segmentation where consistency across patients matters more than one perfect image.
Lu: This work suggests we can reduce the reliance on massive amounts of manually annotated data because it’s finding this structure through self-supervision alone.
Jane: So, if this approach works well in practice, what does that unlock for the next generation of medical imaging tools?
Fudan University of China
cs.CV
Submitted: 2026-05-14
Updated: 2026-11-11
Comments: ICML2026
Code: https://github.com/Ashespt/TACO
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance…
Key concepts
- Instance-level Learning
- Traditional self-supervised methods focus on learning representations for each patient or scan individually, treating them as separate entities. This approach often fails to capture the consistent spatial arrangement of structures within the human body, such as how the thalamus relates to the basal ganglia across different people.
- Intra-instance Constraint (Lintra)
- This loss enforces local neighborhood topology within a single individual across different modalities. It uses a cross-modal triplet objective to ensure that features from different scans of the same person are positioned correctly relative to each other in the feature space, preserving local anatomical relationships.
- Inter-instance Constraint (Linter)
- This loss establishes pseudo-correspondences between different individuals using unsupervised Mutual Nearest Neighbor search. It forces features from the same anatomical region in one patient to align with corresponding features in another patient's representation space, enforcing cross-patient topological consistency.
Terminology
Summary
Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology We propose leveraging this crossinstance topological consistency as a supervisory signal
The Core Proposal
The key idea in this paper is to ensure that if two patches belong to the same region in scan A, their corresponding features from scan B should also co-cluster as the same region in B’s representation space This form of supervision goes beyond traditional instance-level learning: instead of focusing solely on the appearance of individual structures, the model learns the relative positioning of structures within their broader anatomical context To achieve this, we use a dual alignment approach: intra-instance and inter-instance consistency
How the Training Signal is Built
The overall training objective can be formulated as Ltotal = Luni + Lintra + Linter where Lintra and Linter will be introduced in the following part of this section The intra-instance loss Lintra is described in Section 4.1, while the inter-instance loss Linter is outlined in Section 4. p. 2 Specifically, the intra-instance loss
Lintra is described in Section 4. 1, while the inter-instance
loss Linter is outlined in Section 4.Where the paper says it
** Intra-instance Constraint (Lintra) involves a cross-modal triplet objective that preserves local neighborhood topology within the same individual across modalities Specifically, for a given p-th patch feature in Z h i, we construct triplet sets in modality j for contrastive loss as follows The final intra-instance loss is Lintra = 1/M(M − 1) Sum i,j∈M Lintra i→j intra **
** Inter-instance Constraint (Linter) uses unsupervised Mutual Nearest Neighbor (MNN) search to establish pseudo-correspondences, ensuring partial neighborhood alignment across instances The inter-instance loss Linter is defined as Linter = 1/MH(M−1)(H-1) Sum i,t∈M Sum h,g∈H Linter i→t,h→g inter **
Where the Gains Come From
The results show that our method effectively clusters feature tokens by anatomical identity across different individuals and modalities, which aligns well with our intended goal In contrast, baseline self-supervised methods tend to cluster features by individual identity rather than anatomical correspondence, indicating that they capture instance-specific patterns rather than generalizable anatomical structures The t-SNE visualization of token-level representations across patients shows TACO achieves clear and well-organized clusters across patients, demonstrating superior cross-instance learning
Robustness and Efficiency
Our method achieves state-of-the-art (SOTA) performance and exhibits significantly better robustness in missing-modality scenarios, where some imaging sequences are unavailable at test time Table 7 shows that the Hard-neg–Pos gap is 0.0441, which illustrates the strong separation between positive and negative samples Furthermore, TACO demonstrates superior performance in average over all settings, with 81. p. 3 Hence, for a given p-th patch feature in Z
h
i
, we can construct
triplet sets in modality j for contrastive loss as follows
Triintra
i→j, a, h :=n
(a
′
, p′
, n′
)
a
′ = C
h
i→j
(a),
(a
′
, n′
) ∈ P′ω
i→j, h(a) × N ′ω
i→j, h(a)
o
,
(5)
where N
′ω
i→j,h(a) ⊂ N ′
i→j,h(a) is a randomly selected subset with ω samples (paired with top-ω positives). In the
triplet set, (a
′
, p′
) is the positive pair and (a
′
, n′
) is the negative pair.Where the paper says it
Conclusion
This work introduces a novel approach to learning topological consistency in brain structures across instances, ex8 tending beyond traditional instance-level self-supervision through the proposed IM-NRC principle This could potentially reduce reliance on expensive expert annotations, improve data efficiency, and enable more robust downstream performance in clinically relevant tasks such as segmentation, detection, and disease classification The final work is 2605.14654.
--- Page 1 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology We propose leveraging this crossinstance topological consistency as a supervisory signal
--- Page 2 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Prior: instance-level different instances different modalities (a) Comparison with prior methods (b) Topological consistency in brain anatomy
--- Page 3 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Subsequent topology-aware methods primarily target tubular structures (Wen et al., 2025; Elsayed et al. p. 2 Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Subsequent topology-aware methods primarily target tubular structures (Wen et al. , 2025; Elsayed et al.Where the paper says it
--- Page 3 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Assumption 3.1 (IM-NRC). p. 2 This consistency motivates us to learn representations that are invariant with
respect to modality and instance, while being governed by
spatial relationships. To this end, we propose the Instanceagnostic Cross-Modal Neighborhood Ranking Consistency
(IM-NRC) principle for medical multi-modal alignment.Where the paper says it
--- Page 4 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
The reconstruction objective is Luni = xˆ h i m − x h i m2 / 2
--- Page 4 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Figure 2. Pipeline of the proposed method
--- Page 5 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Triintra i→j, a, h:= n(a',p',n')
--- Page 6 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Lintra = 1/M(M − 1) Sum i,j∈M Lintra i→j intra
--- Page 7 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Linter = 1/MH(M−1)(H-1) Sum i,t∈M Sum h,g∈H Linter i→t,h→g inter
--- Page 8 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Figure 5. The t-SNE visualization of token-level representations across patients
--- Page 9 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Features from a randomly initialized model collapse into a single amorphous cloud, indicating no meaningful structure
--- Page 10 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
BrainMVP forms a few large clusters dominated by a single color, i.e., features are separated mainly by patient identity, suggesting patient-specific memorization that can hurt crosspatient transfer
--- Page 11 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
TACO yields a compact distribution, repeatedly occurring clusters where colors are well interleaved. This pattern indicates modality/instance-invariant alignment and neighborhood consistency across instances, supporting the observed gains in both segmentation and classification
--- Page 12 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
In contrast, TACO produces compact, wellseparated micro-clusters with interleaved colors and mostly intra-cluster edges, demonstrating modality-invariant alignment and stable neighborhood consistency, which supports its robustness in downstream tasks and missing-modality settings
--- Page 13 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
The results show that our method achieves the best average on every dataset and the best overall average across all datasets The impact of δ. Fig. 4 (c) shows the impact of the margin δ in Lintra and Linter
--- Page 13 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Table 5 Ablation results of Lintra and Linter
--- Page 13 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
The impact of δ. Fig. 4 (c) shows the impact of the margin δ in Lintra and Linter
--- Page 13 ---
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
The impact of δ. Fig.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Better cross-instance topological consistency for medical imaging pre-training: The system can learn
crossindividual, multi-modal representations that capture relations across individuals,
which is superior to treating each individual as an isolated instance. -
Development of a dual alignment approach: The model can enforce intra-instance alignment using a
cross-modal triplet objective
and inter-instance alignment usingunsupervised Mutual Nearest Neighbor (MNN) search to establish pseudo-correspondences.
-
Robustness to missing modalities: The system exhibits
significantly better robustness when modalities are missing at test time,
enabling reliable performance even when some imaging sequences are unavailable. -
Population-level representation learning: The framework learns
modality-invariant, populationlevel representations for medical data that preserve the intrinsic local topology of each individual while ensuring cross-modal topological consistency.
-
Enhanced generalization across instances and modalities: The method is designed to
enhance generalization ability across instances/people and modalities in medical self-supervised learning, which is crucial for foundation modeling in medical imaging.
Abstract
Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction. They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology. We propose leveraging this cross-instance topological consistency as a supervisory signal. The challenge arises from the inherent variability in medical imaging, which can differ significantly across instances and modalities. To tackle this, we focus on two alignment regimes. (i) Intra-instance: with pixel-level correspondences available, a cross-modal triplet objective explicitly preserves local neighborhood topology. (ii) Inter-instance: without such supervision, we derive pseudo-correspondences to control partial neighborhood alignment and prevent topology collapse across modalities. We validate our approach across 7 downstream multi-modal tasks, achieving average improvements of 1.1% and 5.94% in segmentation and classification tasks, respectively, and demonstrating significantly better robustness when modalities are missing at test time.
Sources
- The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
- Improved Baselines with Momentum Contrastive Learning
- The ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge 2023: Intracranial Meningioma
- Decoupled Weight Decay Regularization
- Robust Perception through Equivariance
- Structure-aware Semantic Discrepancy and Consistency for 3D Medical Image Self-supervised Learning
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs