Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

arXiv:2609.04726 · cs.CV · Submitted 2026-09-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Bridging Modalities and Tasks".

Jane: Here's the gist of the paper. Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities; however, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images <ref:2609.04726#pg2> The proposed framework achieves competitive S2O translation quality and semantic segmentation performance <ref:2609.04726#pg9>.

Tom: Pruning redundant reasoning tokens The paper introduces a unified collaborative dual-task learning framework termed BMT that jointly optimizes SAR-to-optical image translation and semantic segmentation through a shared hierarchical Vision Transformer <ref:2609.04726#pg2> This framework integrates several key components to address the limitations of existing methods <ref:2609.04726#pg2>.

Jane: How the training signal is built The core of the framework is a shared hierarchical Vision Transformer encoder designed as a collaborative multi-stage architecture <ref:2609.04726#pg2> This encoder utilizes a LocalViTBlock module that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism <ref:2609.04726#pg2> The ControlNet-style conditional injection mechanism encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolutions <ref:2609.04726#pg2> Furthermore, a bounded Kendall’s tau uncertainty weighting scheme is adopted to prevent either task from dominating the shared representation <ref:2609.04726#pg2>.

Tom: Where the gains come from The framework achieves its benefits through several specific design elements <ref:2609.04726#pg2> 1) A LocalViTBlock module that combines global self-attention and spatial depthwise convolution via a learnable gating mechanism to capture both long-range dependencies and fine-grained local features <ref:2609.04726#pg2> 2) A ControlNet-style multi-scale conditional injection mechanism that leverages the multi-level Daubechies-2 discrete wavelet transform (DWT) to decompose SAR images and generate multi-scale, multi-frequency texture features <ref:2609.04726#pg2> 3) An enhanced output module that calibrates channel-level color statistics through feature fusion, complemented by anti-aliasing convolutions to extract color statistics from global, regional, and local scales <ref:2609.04726#pg2> Pruning redundant reasoning tokens The objective function is structured to balance the two tasks effectively <ref:2609.04726#pg2> The total generator loss is formulated as a weighted sum over K subtasks using the Bounded Kendall Weight to prevent any single task from dominating the shared representations <ref:2609.04726#pg2> Auxiliary losses, including multi-scale VGG-16 perceptual loss, gradient loss, and color histogram loss, further enhance perceptual quality and color fidelity <ref:2609.04726#pg2>.

Paper summary: Tom: So, wrapping up the discussion on "Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation", this paper is about unifying two complex machine learning problems into one structure.

Jane: They take the challenge of making SAR images interpretable by linking that visual translation task with the need for accurate semantic understanding, which they tackle by sharing a hierarchical Vision Transformer encoder.

Lu: The core contribution is the BMT framework itself, integrating specific components like the LocalViTBlock and a multi-scale conditional injection mechanism to handle both modalities and tasks together efficiently.

Meng: From an engineering standpoint, it’s interesting how they managed to design that shared encoder without letting one task completely hijack the learning process.

Lalam: The paper verifies that this unified approach works by showing SAR-to-optical conversion can happen while simultaneously incorporating structure information useful for segmentation tasks.

Jane: Essentially, the authors are showing that you don't have to choose between making a good visual image and getting accurate labels; they can do both at the same time.

Tom: The implication is that for future research, we should look at how to fine-tune these coordination mechanisms even further to see if we can squeeze out more accuracy in both areas.

Conclusion: Tom: So, we're wrapping up this look at "Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation." Basically, this paper is about taking two separate big problems—turning radar images into clear photos and labeling those photos with what's in them—and solving them using one single shared AI structure.

Jane: That makes sense. So, the core idea is that they don't need two separate models fighting each other; they build one unified system where the conversion process actually helps the labeling process too.

Lu: Exactly. They introduce this hierarchical Vision Transformer encoder that’s designed to handle both tasks at once, using specific modules like a LocalViTBlock and a multi-scale conditional injection mechanism to keep things balanced.

Meng: From what I see on paper, they're using this shared structure because it’s supposed to be more efficient than running two completely different models sequentially. But how much training time does that actually save in practice?

Lalam: Based on the results, they achieved competitive quality for both image translation and segmentation at the same time, meaning you get a better visual output and better labels without needing two separate datasets or training loops. I'm pretty confident in that coordination.

Tom: The numbers are what matter here. They showed significant improvements in metrics like SSIM—which measures how visually similar the images look—and FID, which measures the quality of the generated images, on their test data set. Those gains suggest this integrated approach is genuinely effective compared to methods that only focus on one task at a time.

Jane: It sounds like for someone who just wants a high-quality result quickly, this unified method offers a real boost in both the visual fidelity and the accuracy of those labels they generate.

Lu: The ablation studies confirmed that the LocalViT block was key to maintaining stability across both tasks, which tells us that capturing those fine details locally is crucial when you’re trying to manage global information.

Meng: I'm still thinking about deployment cost. If we use this structure in production, does the complexity of that multi-scale conditional injection mechanism add enough overhead to make it practical for real-time applications?

Lalam: The authors acknowledge a trade-off between segmentation precision and generation quality—strengthening one part hurts the other—but they found the combination actually gives the best balance overall. It’s not perfect, but it's a solid coordinated system.

Tom: So, while it doesn't solve everything perfectly yet, this framework demonstrates that you can effectively use SAR data to reconstruct optical scenes while simultaneously gathering information that helps downstream tasks like segmentation.

Jane: That coordination aspect is huge. It moves the goal from just "can we translate?" to "how do we translate in a way that makes us better at understanding the scene?"

Lu: And they've set up clear directions for future work, focusing on fine-grained coordination mechanisms to push performance even further.

Northwestern Polytechnical University

cs.CV

Submitted: 2026-09-04

Updated: 2026-11-11

Code: https://github.com/Lewisyuaner/BMT-S2O-main

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities; however, compared with optical images…

Key concepts

BMT Framework
A unified collaborative dual-task learning framework that jointly optimizes SAR-to-optical image translation and semantic segmentation. It uses a shared hierarchical Vision Transformer encoder to handle both tasks simultaneously, preventing one from dominating the other during training.
LocalViTBlock Module
A core component within the encoder that combines global self-attention with spatial depthwise convolution. This module is designed to capture both long-range dependencies across the image and fine-grained local features simultaneously, improving feature extraction quality.
ControlNet-style Conditional Injection
A mechanism used to inject control signals (SAR wavelet features and segmentation labels) into the Vision Transformer layers. This multi-scale injection uses a pyramid structure to provide rich, multi-frequency texture information at various levels of the network for better task coordination.

Terminology

Summary

Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities; however, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images <ref:2609. p. 0

Where the paper says it

1 Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images.

04726#pg2> The proposed framework achieves competitive S2O translation quality and semantic segmentation performance <ref:2609. p. 0
Where the paper says it

We evaluate the framework under both paired and unpaired translation settings, on the public WHU-OPT-SAR paired dataset and a self-constructed unpaired ship dataset built from HRSID and DIOR, respectively. The experimental results show that the proposed method achieves competitive S2O translation quality and semantic segmentation performance.

04726#pg9>.

Pruning redundant reasoning tokens

The paper introduces a unified collaborative dual-task learning framework termed BMT that jointly optimizes SAR-to-optical image translation and semantic segmentation through a shared hierarchical Vision Transformer <ref:2609.04726#pg2> This framework integrates several key components to address the limitations of existing methods <ref:2609. p. 0

Where the paper says it

However, three key gaps remain: (1) Existing S2O methods lack semantic awareness; while the generated images are visually plausible, they often overlook whether the generated images are useful for downstream high-level visual tasks. This shortcoming is particularly pronounced when the S2O transformation needs to serve semantic-oriented applications; (2) Current segmentation models trained solely on synthetic aperture radar (SAR) data struggle to address issues such as speckle noise and limited labeled data; (3) No existing framework can simultaneously optimize both tasks through a shared architectural backbone featuring multiscale cross-modal conditional injection and dynamic task balancing.

04726#pg2>.

How the training signal is built

The core of the framework is a shared hierarchical Vision Transformer encoder designed as a collaborative multi-stage architecture <ref:2609.04726#pg2> This encoder utilizes a LocalViTBlock module that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism <ref:2609. p. 0

Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

04726#pg2> The ControlNet-style conditional injection mechanism encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolutions <ref:2609. p. 0
Where the paper says it

We propose a unified collaborative dual-task learning framework, termed BMT (Bridging Modalities and Tasks), that jointly optimizes S2O image translation and semantic segmentation through a shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale refinement processing, color correction and anti-aliasing, which calibrates channel-level color statistics through feature fusion; (3) a ControlNet-style conditional injection mechanism that encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolution; (4) a bounded Kendall uncertainty weighting scheme that prevents either task from dominating the shared representation.

04726#pg2> Furthermore, a bounded Kendall’s tau uncertainty weighting scheme is adopted to prevent either task from dominating the shared representation <ref:2609. p. 1
Where the paper says it

This module calibrates channel-level color statistics through feature fusion, complemented by anti-aliasing convolutions, to extract channel-level color statistics from global, regional, and local scales. Fourth, inspired by training strategies for multi-task learning, we adopt a bounded Kendall’s tau uncertainty weighting scheme to prevent any single task from dominating the shared representations.

04726#pg2>.

Where the gains come from

The framework achieves its benefits through several specific design elements <ref:2609.04726#pg2>

  1. A LocalViTBlock module that combines global self-attention and spatial depthwise convolution via a learnable gating mechanism to capture both long-range dependencies and fine-grained local features <ref:2609.04726#pg2>

  2. A ControlNet-style multi-scale conditional injection mechanism that leverages the multi-level Daubechies-2 discrete wavelet transform (DWT) to decompose SAR images and generate multi-scale, multi-frequency texture features <ref:2609.04726#pg2>

  3. An enhanced output module that calibrates channel-level color statistics through feature fusion, complemented by anti-aliasing convolutions to extract color statistics from global, regional, and local scales <ref:2609.04726#pg2>

The objective function is structured to balance the two tasks effectively <ref:2609.04726#pg2> The total generator loss is formulated as a weighted sum over K subtasks using the Bounded Kendall Weight to prevent any single task from dominating the shared representations <ref:2609. p. 5

Where the paper says it

The generator loss is a weighted sum over K subtasks. To prevent one task’s weight from collapsing near zero, we introduce the Bounded Kendall Weight: wk = clampexp(−sk), wmin, wmax (17) where sk is the learnable log-variance and [wmin, wmax] = [0.

04726#pg2> Auxiliary losses, including multi-scale VGG-16 perceptual loss, gradient loss, and color histogram loss, further enhance perceptual quality and color fidelity <ref:2609. p. 4
Where the paper says it

A multi-scale VGG-16 perceptual loss promotes photorealism: Lperc(G) = X ℓ∈L wℓ · ϕℓ(G(x)) − ϕℓ(y) 1 (11) where ϕℓ denotes VGG-16 features after ReLU at layer ℓ, and ℓ = {ℓ1, ℓ2, ℓ3, ℓ4} with corresponding weights w = {1. The gradient loss directly penalizes blurring by computing the L1 distance between spatial gradients of the generated and target images: Lgrad = ∥∇xG(x) − ∇xy∥1 + ∥∇yG(x) − ∇yy∥1 (12) The color histogram loss uses kernel-density estimation with Gaussian RBF kernels to build soft histograms in a LAB-like color-opponent space, then computes L1 distance between the histograms of generated and target images: Lhist = 1 3 X 3 c=1 ∥Histc(G(x)) − Histc(y)∥1 (13) This loss directly penalizes color-distribution drift, such as background color bleeding into ship targets.

04726#pg2>.

Experimental results on the WHU-OPT-SAR dataset show that BMT outperforms all compared generative models, achieving significant improvements particularly in SSIM (45.54) and FID (156. p. 6

Where the paper says it

Quantitative experiments on the WHU-OPT-SAR dataset show that BMT outperforms all compared generative models, achieving significant improvements particularly in SSIM (45. 54) and FID (156.

31) <ref:2609. p. 0
Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

04726#pg2> Furthermore, the collaborative segmentation metric (BMT∗) improves the mIoU and mPA metrics by more than 10% compared to baseline methods, proving its ability to coordinate cross-task consistency <ref:2609. p. 7
Where the paper says it

By sharing the coding layer, the S2O conversion and downstream tasks are parallelized, thus forcing collaboration and alignment between conversion and segmentation. It can be seen that compared with the segmentation results of the baseline method, the original SAR and optical images, the collaborative segmentation metric (BMT*) of the proposed framework improves the mIoU and mPA metrics by more than 10%, which proves its ability to coordinate cross-task consistency, that is, the ability to generate optical images conforming to the real distribution while maintaining the semantic integrity of the original SAR data.

04726#pg2>. The performance on the HRSID-DIOR dataset also demonstrates that BMT maintains a level comparable to mainstream baseline methods under unpaired tasks <ref:2609. p. 7
Where the paper says it

809, second only to UNSB. It can be seen that under the unpaired unsupervised task, BMT can still maintain a level comparable to mainstream baseline methods, because the fluctuation of FID/KID metrics of around 10% in unpaired tasks does not mean a significant deviation in generated textures, but more indicates the diversity of styles.

04726#pg2>

Ablation studies confirm that the LocalViT block is the most stable source of gain in the overall framework, increasing SSIM and mIoU/mPA on both datasets <ref:2609.04726#pg2> The trade-off between segmentation accuracy and generation quality is evident, as strengthening segmentation consistency alone sacrifices generation quality while introducing color loss, but the complete loss combination achieves the best trade-off <ref:2609. p. 9

Where the paper says it

of tasks and its potential in cross-modal tasks. Ablation experiments show that the contributions of each module and loss term are not linearly superimposed, but there is an obvious trade-off between segmentation accuracy and the quality of the generated distribution.

04726#pg2>.

The paper concludes that BMT verifies the positive cross-task complementarity by showing that conversion can effectively reconstruct optical images while integrating information beneficial to downstream tasks <ref:2609.04726#pg2> Fine-grained coordination mechanisms will be explored in the future to further improve performance <ref:2609. p. 0

Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

04726#pg2>. The references section provides foundational work on conditional GANs, diffusion models, and various segmentation architectures <ref:2609. p. 0
Where the paper says it

On the one hand, significant progress has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6], generating increasingly realistic optical images. On the other hand, segmentation models based on convolutional neural networks (CNNs), Transformers, and hybrid architectures [7–10] have achieved high accuracy in remote sensing benchmarks.

04726#pg2>.

REFERENCES

[1] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. p. 0

Where the paper says it

Despite these complementary characteristics, the conversion from synthetic aperture radar (SAR) to optical images (S2O) and SAR semantic segmentation have consistently been studied as separate tasks. On the one hand, significant progress has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6], generating increasingly realistic optical images.

arXiv preprint arXiv:1411. p. 0
Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

1784, 2014 <ref:2609. p. 0
Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

04726#pg7>

[3] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. p. 0

Where the paper says it

Despite these complementary characteristics, the conversion from synthetic aperture radar (SAR) to optical images (S2O) and SAR semantic segmentation have consistently been studied as separate tasks. On the one hand, significant progress has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6], generating increasingly realistic optical images.

In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242–2251, 2017

[4] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. p. 1

Where the paper says it

Several studies [20–22] focus on core challenges in S2O translation, while others [23] explore the impact on downstream classification. Diffusion models [4, 5] have emerged as a powerful alternative, offering greater generative diversity through iterative denoising.

Advances in neural information processing systems, 33:6840–6851, 2020

[7] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers <ref:2609. p. 0

Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

04726#pg2>

[11] Hang Liu, Jiarui Lin, Cang Gu, Yujie Zhang, and Huihui Li. Hvtc-gan: A high-level vision task cooperative gan for sar-to-optical translation via semantic segmentation <ref:2609. p. 0

Where the paper says it

We propose a unified collaborative dual-task learning framework, termed BMT (Bridging Modalities and Tasks), that jointly optimizes S2O image translation and semantic segmentation through a shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale refinement processing, color correction and anti-aliasing, which calibrates channel-level color statistics through feature fusion; (3) a ControlNet-style conditional injection mechanism that encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolution; (4) a bounded Kendall uncertainty weighting scheme that prevents either task from dominating the shared representation.

04726#pg2>

[12] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. p. 2

Where the paper says it

As shown in Figure 1, given an image x ∈ R 3×H×W , it is first projected into a sequence of non-overlapping patches through a convolutional layer. Between different stages, the Merging layer performs spatial downsampling, which mimics the inductive bias of convolutional neural networks and provides a computationally affordable way to process highresolution features in the early stages.

In Proc. MICCAI, pages 234–241, 2015

[13] H. Zhao, J. Shi, X. Qi, X. Wang, and J. p. 0

Where the paper says it

1 Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images.

Jia. Pyramid scene parsing network

[14] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y Wei, Z Zhang, S Lin, and B Guo. p. 0

Where the paper says it

1 Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images.

Swin Transformer: Hierarchical vision transformer using shifted windows

[15] L. Huang, B Jiang, S Lv, Y Liu, and Y Fu. p. 0

Where the paper says it

1 Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images.

Deeplearning-based semantic segmentation of remote sensing images: A survey

[18] Nina Merkle, Stefan Auer, Rupert Muller, and Peter ¨ Reinartz. Exploring the potential of conditional adversarial networks for optical and SAR image matching

[28] Roberto Cipolla, Yarin Gal, and Alex Kendall. Multitask learning using uncertainty to weigh losses for scene geometry and semantics

[35] Huihui Li, Cang Gu, Dongqing Wu, Gong Cheng, Lei Guo, and Hang Liu. Multiscale generative adversarial network based on wavelet feature learning for sar-tooptical image translation

[16] X. Yang, Z. Wang, J Zhao, et al. p. 0

Where the paper says it

1 Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images.

Fg-gan: A finegrained generative adversarial network for unsupervised sar-to-optical image translation

[25] Xinyu Bai, Xinyang Pu, and Feng Xu. Conditional diffusion for sar to optical image translation. p. 0

Where the paper says it

We propose a unified collaborative dual-task learning framework, termed BMT (Bridging Modalities and Tasks), that jointly optimizes S2O image translation and semantic segmentation through a shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale refinement processing, color correction and anti-aliasing, which calibrates channel-level color statistics through feature fusion; (3) a ControlNet-style conditional injection mechanism that encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolution; (4) a bounded Kendall uncertainty weighting scheme that prevents either task from dominating the shared representation.

IEEE Geoscience and Remote Sensing Letters, 21:1–5, 2024

[30] Shunjun Wei, Xiangfeng Zeng, Qizhe Qu, Mou Wang, Hao Su, and Jun Shi. Hrsid: A high-resolution sar images dataset for ship detection and instance segmentation

[41] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation

[42] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross ´ Girshick. Mask r-cnn

[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. p. 3

Where the paper says it

These descriptors capture color distribution information ranging from overall hue to local texture variations. After being fused through a two-layer MLP, they generate a compact color condition vector: vcolor = MLP[µ1, σ1, µ4, σ4, µ16, σ16] (2) where µs and σs are the mean and standard deviation at scale s.

Advances in neural information processing systems, 30, 2017

[46] Mikołaj Binkowski, Danica J Sutherland, Michael Arbel, ´ and Arthur Gretton. Demystifying mmd gans. p. 0

Where the paper says it

Despite these complementary characteristics, the conversion from synthetic aperture radar (SAR) to optical images (S2O) and SAR semantic segmentation have consistently been studied as separate tasks. On the one hand, significant progress has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6], generating increasingly realistic optical images.

arXiv preprint arXiv:1801. p. 0
Where the paper says it

The framework includes the following key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which combines the spatial depthwise convolution branch with global self-attention, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks. Second, we arXiv:2609.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Unified Collaborative Learning Framework BMT: The system can jointly optimize SAR-to-optical image translation and semantic segmentation through a shared hierarchical Vision Transformer encoder, allowing for a collaborative training of S2O and semantic segmentation tasks under various scenarios and conditions.

  2. LocalViTBlock Module: This module fuses global self-attention with spatial depthwise convolution via a learnable gating mechanism, enabling the encoder to simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks.

  3. ControlNet Style Conditional Injection: The system can provide fine-grained spatial guidance by encoding SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injecting them at each encoder layer through zero initialized convolutions, ensuring that the conditional path is in a no-operation state at the initial stage of training.

  4. Enhanced Output Module: This module calibrates channel level color statistics through feature fusion using multi scale refinement processing, color correction, and anti aliasing convolutions to extract statistical descriptors like vcolor from global, regional, and local scales.

  5. Bounded Kendall Uncertainty Weighting Scheme: This scheme prevents either task from dominating the shared representation by using a learnable log variance to modulate task weights between wmin and wmax, ensuring neither task starves the shared encoder while allowing adaptive modulation of relative task importance throughout training.

  6. Improved S2O Translation Quality: On datasets like WHU-OPT-SAR, BMT achieves competitive translation quality by utilizing auxiliary losses like Lgrad (gradient loss) and Lhist (color histogram loss), which directly penalizes color-distribution drift, such as background color bleeding into ship targets.

  7. Enhanced Semantic Segmentation Accuracy: The framework improves segmentation metrics by applying a cycle segmentation loss and auxiliary losses, resulting in BMT achieving the highest semantic segmentation metrics on the source SAR dataset, with mIoU reaching 88.51% and mPA reaching 93.07%.

Abstract

Synthetic Aperture Radar (SAR) remains available through cloud and darkness, yet speckle and non-intuitive scattering make it hard to inspect and interpret. SAR-to-optical (S2O) translation renders radar observations as optical-like images for visual access, while semantic segmentation on SAR remains necessary since land-cover decisions cannot rely on synthesised pixels. We propose BMT, a hierarchical Vision Transformer coupling both tasks on a shared encoder via multi-scale one-hot conditioning, a gated LocalViTBlock, a colour-calibrating generation head, ControlNet-style zero-initialized injection, and uncertainty-based task weighting. The joint framework's value is not label-free training: it uses paired optical images and the same pixel annotations as a dedicated SAR segmentor, supervising the SAR decoder and teacher-forcing the generator with ground-truth one-hot maps. At test time, the model takes only SAR and returns an optical-like readout and a radar-decided land-cover map, with the translation objective supplying dense appearance supervision that speckle lacks. On WHU-OPT-SAR, the predicted-condition model achieves the best SSIM and PSNR among compared translators; its land-cover map surpasses the same backbone trained for labelling alone and is within about one point of DeepLabV3+.

Sources

Related papers