Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Bridging Modalities and Tasks".
Jane: Here's the gist of the paper. Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities; however, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images <ref:2609.04726#pg2> The proposed framework achieves competitive S2O translation quality and semantic segmentation performance <ref:2609.04726#pg9>.
Tom: Pruning redundant reasoning tokens The paper introduces a unified collaborative dual-task learning framework termed BMT that jointly optimizes SAR-to-optical image translation and semantic segmentation through a shared hierarchical Vision Transformer <ref:2609.04726#pg2> This framework integrates several key components to address the limitations of existing methods <ref:2609.04726#pg2>.
Jane: How the training signal is built The core of the framework is a shared hierarchical Vision Transformer encoder designed as a collaborative multi-stage architecture <ref:2609.04726#pg2> This encoder utilizes a LocalViTBlock module that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism <ref:2609.04726#pg2> The ControlNet-style conditional injection mechanism encodes SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injects them at each encoder layer through zero-initialized convolutions <ref:2609.04726#pg2> Furthermore, a bounded Kendall’s tau uncertainty weighting scheme is adopted to prevent either task from dominating the shared representation <ref:2609.04726#pg2>.
Tom: Where the gains come from The framework achieves its benefits through several specific design elements <ref:2609.04726#pg2> 1) A LocalViTBlock module that combines global self-attention and spatial depthwise convolution via a learnable gating mechanism to capture both long-range dependencies and fine-grained local features <ref:2609.04726#pg2> 2) A ControlNet-style multi-scale conditional injection mechanism that leverages the multi-level Daubechies-2 discrete wavelet transform (DWT) to decompose SAR images and generate multi-scale, multi-frequency texture features <ref:2609.04726#pg2> 3) An enhanced output module that calibrates channel-level color statistics through feature fusion, complemented by anti-aliasing convolutions to extract color statistics from global, regional, and local scales <ref:2609.04726#pg2> Pruning redundant reasoning tokens The objective function is structured to balance the two tasks effectively <ref:2609.04726#pg2> The total generator loss is formulated as a weighted sum over K subtasks using the Bounded Kendall Weight to prevent any single task from dominating the shared representations <ref:2609.04726#pg2> Auxiliary losses, including multi-scale VGG-16 perceptual loss, gradient loss, and color histogram loss, further enhance perceptual quality and color fidelity <ref:2609.04726#pg2>.
Paper summary: Tom: So, wrapping up the discussion on "Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation", this paper is about unifying two complex machine learning problems into one structure.
Jane: They take the challenge of making SAR images interpretable by linking that visual translation task with the need for accurate semantic understanding, which they tackle by sharing a hierarchical Vision Transformer encoder.
Lu: The core contribution is the BMT framework itself, integrating specific components like the LocalViTBlock and a multi-scale conditional injection mechanism to handle both modalities and tasks together efficiently.
Meng: From an engineering standpoint, it’s interesting how they managed to design that shared encoder without letting one task completely hijack the learning process.
Lalam: The paper verifies that this unified approach works by showing SAR-to-optical conversion can happen while simultaneously incorporating structure information useful for segmentation tasks.
Jane: Essentially, the authors are showing that you don't have to choose between making a good visual image and getting accurate labels; they can do both at the same time.
Tom: The implication is that for future research, we should look at how to fine-tune these coordination mechanisms even further to see if we can squeeze out more accuracy in both areas.
Conclusion: Tom: So, we're wrapping up this look at "Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation." Basically, this paper is about taking two separate big problems—turning radar images into clear photos and labeling those photos with what's in them—and solving them using one single shared AI structure.
Jane: That makes sense. So, the core idea is that they don't need two separate models fighting each other; they build one unified system where the conversion process actually helps the labeling process too.
Lu: Exactly. They introduce this hierarchical Vision Transformer encoder that’s designed to handle both tasks at once, using specific modules like a LocalViTBlock and a multi-scale conditional injection mechanism to keep things balanced.
Meng: From what I see on paper, they're using this shared structure because it’s supposed to be more efficient than running two completely different models sequentially. But how much training time does that actually save in practice?
Lalam: Based on the results, they achieved competitive quality for both image translation and segmentation at the same time, meaning you get a better visual output and better labels without needing two separate datasets or training loops. I'm pretty confident in that coordination.
Tom: The numbers are what matter here. They showed significant improvements in metrics like SSIM—which measures how visually similar the images look—and FID, which measures the quality of the generated images, on their test data set. Those gains suggest this integrated approach is genuinely effective compared to methods that only focus on one task at a time.
Jane: It sounds like for someone who just wants a high-quality result quickly, this unified method offers a real boost in both the visual fidelity and the accuracy of those labels they generate.
Lu: The ablation studies confirmed that the LocalViT block was key to maintaining stability across both tasks, which tells us that capturing those fine details locally is crucial when you’re trying to manage global information.
Meng: I'm still thinking about deployment cost. If we use this structure in production, does the complexity of that multi-scale conditional injection mechanism add enough overhead to make it practical for real-time applications?
Lalam: The authors acknowledge a trade-off between segmentation precision and generation quality—strengthening one part hurts the other—but they found the combination actually gives the best balance overall. It’s not perfect, but it's a solid coordinated system.
Tom: So, while it doesn't solve everything perfectly yet, this framework demonstrates that you can effectively use SAR data to reconstruct optical scenes while simultaneously gathering information that helps downstream tasks like segmentation.
Jane: That coordination aspect is huge. It moves the goal from just "can we translate?" to "how do we translate in a way that makes us better at understanding the scene?"
Lu: And they've set up clear directions for future work, focusing on fine-grained coordination mechanisms to push performance even further.
Northwestern Polytechnical University
cs.CV
Submitted: 2026-09-04
Updated: 2026-11-11
Code: https://github.com/Lewisyuaner/BMT-S2O-main
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities; however, compared with optical images…
Key concepts
- BMT Framework
- A unified collaborative dual-task learning framework that jointly optimizes SAR-to-optical image translation and semantic segmentation. It uses a shared hierarchical Vision Transformer encoder to handle both tasks simultaneously, preventing one from dominating the other during training.
- LocalViTBlock Module
- A core component within the encoder that combines global self-attention with spatial depthwise convolution. This module is designed to capture both long-range dependencies across the image and fine-grained local features simultaneously, improving feature extraction quality.
- ControlNet-style Conditional Injection
- A mechanism used to inject control signals (SAR wavelet features and segmentation labels) into the Vision Transformer layers. This multi-scale injection uses a pyramid structure to provide rich, multi-frequency texture information at various levels of the network for better task coordination.
Terminology
Summary
Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities; however, compared with optical images, their speckle noise and non-intuitive scattering mechanism limit the interpretability of the images <ref:2609. p. 0 1
Bridging Modalities and Tasks: A Unified Hierarchical ViT for
SAR-to-Optical Translation and Semantic Segmentation
Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li
Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive
scattering mechanism limit the interpretability of the images. We evaluate the framework under
both paired and unpaired translation settings, on the public
WHU-OPT-SAR paired dataset and a self-constructed unpaired
ship dataset built from HRSID and DIOR, respectively. The
experimental results show that the proposed method achieves
competitive S2O translation quality and semantic segmentation
performance.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper introduces a unified collaborative dual-task learning framework termed BMT that jointly optimizes SAR-to-optical image translation and semantic segmentation through a shared hierarchical Vision Transformer <ref:2609.04726#pg2> This framework integrates several key components to address the limitations of existing methods <ref:2609. p. 0 However, three key gaps remain: (1) Existing S2O methods
lack semantic awareness; while the generated images are
visually plausible, they often overlook whether the generated
images are useful for downstream high-level visual tasks. This shortcoming is particularly pronounced when the S2O
transformation needs to serve semantic-oriented applications;
(2) Current segmentation models trained solely on synthetic
aperture radar (SAR) data struggle to address issues such
as speckle noise and limited labeled data; (3) No existing
framework can simultaneously optimize both tasks through a
shared architectural backbone featuring multiscale cross-modal
conditional injection and dynamic task balancing.Where the paper says it
How the training signal is built
The core of the framework is a shared hierarchical Vision Transformer encoder designed as a collaborative multi-stage architecture <ref:2609.04726#pg2> This encoder utilizes a LocalViTBlock module that fuses global self-attention with spatial depthwise convolution through a learnable gating mechanism <ref:2609. p. 0 The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609. We propose
a unified collaborative dual-task learning framework, termed
BMT (Bridging Modalities and Tasks), that jointly optimizes
S2O image translation and semantic segmentation through a
shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with
spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale
refinement processing, color correction and anti-aliasing, which
calibrates channel-level color statistics through feature fusion; (3)
a ControlNet-style conditional injection mechanism that encodes
SAR wavelet features and segmentation labels into a multi-scale
feature pyramid and injects them at each encoder layer through
zero-initialized convolution; (4) a bounded Kendall uncertainty
weighting scheme that prevents either task from dominating
the shared representation. This module calibrates
channel-level color statistics through feature fusion, complemented by anti-aliasing convolutions, to extract channel-level
color statistics from global, regional, and local scales. Fourth,
inspired by training strategies for multi-task learning, we adopt
a bounded Kendall’s tau uncertainty weighting scheme to
prevent any single task from dominating the shared representations.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The framework achieves its benefits through several specific design elements <ref:2609.04726#pg2>
-
A LocalViTBlock module that combines global self-attention and spatial depthwise convolution via a learnable gating mechanism to capture both long-range dependencies and fine-grained local features <ref:2609.04726#pg2>
-
A ControlNet-style multi-scale conditional injection mechanism that leverages the multi-level Daubechies-2 discrete wavelet transform (DWT) to decompose SAR images and generate multi-scale, multi-frequency texture features <ref:2609.04726#pg2>
-
An enhanced output module that calibrates channel-level color statistics through feature fusion, complemented by anti-aliasing convolutions to extract color statistics from global, regional, and local scales <ref:2609.04726#pg2>
The objective function is structured to balance the two tasks effectively <ref:2609.04726#pg2> The total generator loss is formulated as a weighted sum over K subtasks using the Bounded Kendall Weight to prevent any single task from dominating the shared representations <ref:2609. p. 5 The generator loss is a weighted sum over K
subtasks. To prevent one task’s weight from collapsing near
zero, we introduce the Bounded Kendall Weight:
wk = clampexp(−sk), wmin, wmax
(17)
where sk is the learnable log-variance and [wmin, wmax] =
[0. A multi-scale VGG-16
perceptual loss promotes photorealism:
Lperc(G) = X
ℓ∈L
wℓ ·
ϕℓ(G(x)) − ϕℓ(y)
1
(11)
where ϕℓ denotes VGG-16 features after ReLU at layer ℓ,
and ℓ = {ℓ1, ℓ2, ℓ3, ℓ4} with corresponding weights w =
{1. The gradient loss directly penalizes blurring by computing
the L1 distance between spatial gradients of the generated and
target images:
Lgrad = ∥∇xG(x) − ∇xy∥1 + ∥∇yG(x) − ∇yy∥1 (12)
The color histogram loss uses kernel-density estimation with
Gaussian RBF kernels to build soft histograms in a LAB-like
color-opponent space, then computes L1 distance between the
histograms of generated and target images:
Lhist =
1
3
X
3
c=1
∥Histc(G(x)) − Histc(y)∥1 (13)
This loss directly penalizes color-distribution drift, such as
background color bleeding into ship targets.Where the paper says it
Where the paper says it
Experimental results on the WHU-OPT-SAR dataset show that BMT outperforms all compared generative models, achieving significant improvements particularly in SSIM (45.54) and FID (156. p. 6 Quantitative experiments on the WHU-OPT-SAR dataset
show that BMT outperforms all compared generative models, achieving significant improvements particularly in SSIM
(45. 54) and FID (156. The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609. By sharing the coding layer,
the S2O conversion and downstream tasks are parallelized,
thus forcing collaboration and alignment between conversion
and segmentation. It can be seen that compared with the
segmentation results of the baseline method, the original SAR
and optical images, the collaborative segmentation metric
(BMT*) of the proposed framework improves the mIoU and
mPA metrics by more than 10%, which proves its ability
to coordinate cross-task consistency, that is, the ability to
generate optical images conforming to the real distribution
while maintaining the semantic integrity of the original SAR
data. 809, second only to UNSB. It can be seen
that under the unpaired unsupervised task, BMT can still
maintain a level comparable to mainstream baseline methods,
because the fluctuation of FID/KID metrics of around 10%
in unpaired tasks does not mean a significant deviation in
generated textures, but more indicates the diversity of styles.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ablation studies confirm that the LocalViT block is the most stable source of gain in the overall framework, increasing SSIM and mIoU/mPA on both datasets <ref:2609.04726#pg2> The trade-off between segmentation accuracy and generation quality is evident, as strengthening segmentation consistency alone sacrifices generation quality while introducing color loss, but the complete loss combination achieves the best trade-off <ref:2609. p. 9 of tasks and its potential in cross-modal tasks. Ablation experiments show that the contributions of each
module and loss term are not linearly superimposed, but there
is an obvious trade-off between segmentation accuracy and the
quality of the generated distribution.Where the paper says it
The paper concludes that BMT verifies the positive cross-task complementarity by showing that conversion can effectively reconstruct optical images while integrating information beneficial to downstream tasks <ref:2609.04726#pg2> Fine-grained coordination mechanisms will be explored in the future to further improve performance <ref:2609. p. 0 The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609. On the one hand, significant progress
has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6],
generating increasingly realistic optical images. On the other
hand, segmentation models based on convolutional neural networks (CNNs), Transformers, and hybrid architectures [7–10]
have achieved high accuracy in remote sensing benchmarks.Where the paper says it
Where the paper says it
REFERENCES
[1] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. p. 0 Despite these complementary characteristics, the conversion
from synthetic aperture radar (SAR) to optical images (S2O)
and SAR semantic segmentation have consistently been studied as separate tasks. On the one hand, significant progress
has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6],
generating increasingly realistic optical images. The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609. The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609.Where the paper says it
Where the paper says it
Where the paper says it
[3] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. p. 0 Despite these complementary characteristics, the conversion
from synthetic aperture radar (SAR) to optical images (S2O)
and SAR semantic segmentation have consistently been studied as separate tasks. On the one hand, significant progress
has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6],
generating increasingly realistic optical images.Where the paper says it
[4] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. p. 1 Several studies [20–22] focus on core challenges in S2O translation, while others [23] explore the impact on downstream
classification. Diffusion models [4, 5] have emerged as a powerful alternative, offering greater generative diversity through iterative
denoising.Where the paper says it
[7] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers <ref:2609. p. 0 The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609.Where the paper says it
[11] Hang Liu, Jiarui Lin, Cang Gu, Yujie Zhang, and Huihui Li. Hvtc-gan: A high-level vision task cooperative gan for sar-to-optical translation via semantic segmentation <ref:2609. p. 0 We propose
a unified collaborative dual-task learning framework, termed
BMT (Bridging Modalities and Tasks), that jointly optimizes
S2O image translation and semantic segmentation through a
shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with
spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale
refinement processing, color correction and anti-aliasing, which
calibrates channel-level color statistics through feature fusion; (3)
a ControlNet-style conditional injection mechanism that encodes
SAR wavelet features and segmentation labels into a multi-scale
feature pyramid and injects them at each encoder layer through
zero-initialized convolution; (4) a bounded Kendall uncertainty
weighting scheme that prevents either task from dominating
the shared representation.Where the paper says it
[12] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. p. 2 As shown in Figure 1, given an image x ∈ R
3×H×W , it
is first projected into a sequence of non-overlapping patches
through a convolutional layer. Between different stages, the
Merging layer performs spatial downsampling, which mimics the inductive bias of convolutional neural networks and
provides a computationally affordable way to process highresolution features in the early stages.Where the paper says it
[13] H. Zhao, J. Shi, X. Qi, X. Wang, and J. p. 0 1
Bridging Modalities and Tasks: A Unified Hierarchical ViT for
SAR-to-Optical Translation and Semantic Segmentation
Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li
Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive
scattering mechanism limit the interpretability of the images.Where the paper says it
[14] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y Wei, Z Zhang, S Lin, and B Guo. p. 0 1
Bridging Modalities and Tasks: A Unified Hierarchical ViT for
SAR-to-Optical Translation and Semantic Segmentation
Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li
Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive
scattering mechanism limit the interpretability of the images.Where the paper says it
[15] L. Huang, B Jiang, S Lv, Y Liu, and Y Fu. p. 0 1
Bridging Modalities and Tasks: A Unified Hierarchical ViT for
SAR-to-Optical Translation and Semantic Segmentation
Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li
Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive
scattering mechanism limit the interpretability of the images.Where the paper says it
[18] Nina Merkle, Stefan Auer, Rupert Muller, and Peter ¨ Reinartz. Exploring the potential of conditional adversarial networks for optical and SAR image matching
[28] Roberto Cipolla, Yarin Gal, and Alex Kendall. Multitask learning using uncertainty to weigh losses for scene geometry and semantics
[35] Huihui Li, Cang Gu, Dongqing Wu, Gong Cheng, Lei Guo, and Hang Liu. Multiscale generative adversarial network based on wavelet feature learning for sar-tooptical image translation
[16] X. Yang, Z. Wang, J Zhao, et al. p. 0 1
Bridging Modalities and Tasks: A Unified Hierarchical ViT for
SAR-to-Optical Translation and Semantic Segmentation
Siyuan Liu, Xuze Zhang, Yongshun Wang, Licong Pan, Hang Liu, and Huihui Li
Abstract—Synthetic Aperture Radar (SAR) images have allweather, day-and-night observation capabilities. However, compared with optical images, their speckle noise and non-intuitive
scattering mechanism limit the interpretability of the images.Where the paper says it
[25] Xinyu Bai, Xinyang Pu, and Feng Xu. Conditional diffusion for sar to optical image translation. p. 0 We propose
a unified collaborative dual-task learning framework, termed
BMT (Bridging Modalities and Tasks), that jointly optimizes
S2O image translation and semantic segmentation through a
shared hierarchical Vision Transformer. The framework integrates: (1) a LocalViTBlock that fuses global self-attention with
spatial depthwise convolution through a learnable gating mechanism; (2) an enhanced output module combining multi-scale
refinement processing, color correction and anti-aliasing, which
calibrates channel-level color statistics through feature fusion; (3)
a ControlNet-style conditional injection mechanism that encodes
SAR wavelet features and segmentation labels into a multi-scale
feature pyramid and injects them at each encoder layer through
zero-initialized convolution; (4) a bounded Kendall uncertainty
weighting scheme that prevents either task from dominating
the shared representation.Where the paper says it
[30] Shunjun Wei, Xiangfeng Zeng, Qizhe Qu, Mou Wang, Hao Su, and Jun Shi. Hrsid: A high-resolution sar images dataset for ship detection and instance segmentation
[41] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation
[42] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross ´ Girshick. Mask r-cnn
[45] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. p. 3 These descriptors capture
color distribution information ranging from overall hue to local
texture variations. After being fused through a two-layer MLP,
they generate a compact color condition vector:
vcolor = MLP[µ1, σ1, µ4, σ4, µ16, σ16]
(2)
where µs and σs are the mean and standard deviation at
scale s.Where the paper says it
[46] Mikołaj Binkowski, Danica J Sutherland, Michael Arbel, ´ and Arthur Gretton. Demystifying mmd gans. p. 0 Despite these complementary characteristics, the conversion
from synthetic aperture radar (SAR) to optical images (S2O)
and SAR semantic segmentation have consistently been studied as separate tasks. On the one hand, significant progress
has been made in S2O conversion using generative adversarial networks (GANs) [1–3] and diffusion models [4–6],
generating increasingly realistic optical images. The framework includes the following
key design elements: First, we have designed the LocalViTBlock module with a learnable gating mechanism, which
combines the spatial depthwise convolution branch with global
self-attention, enabling the encoder to simultaneously capture
long-range dependencies and fine-grained local features that
are critical to both of the aforementioned tasks. Second, we
arXiv:2609.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified Collaborative Learning Framework BMT: The system can jointly optimize SAR-to-optical image translation and semantic segmentation through a shared hierarchical Vision Transformer encoder, allowing for a
collaborative training of S2O and semantic segmentation tasks under various scenarios and conditions.
-
LocalViTBlock Module: This module fuses global self-attention with spatial depthwise convolution via a learnable gating mechanism, enabling the encoder to
simultaneously capture long-range dependencies and fine-grained local features that are critical to both of the aforementioned tasks.
-
ControlNet Style Conditional Injection: The system can provide fine-grained spatial guidance by encoding SAR wavelet features and segmentation labels into a multi-scale feature pyramid and injecting them at each encoder layer through zero initialized convolutions,
ensuring that the conditional path is in a no-operation state at the initial stage of training.
-
Enhanced Output Module: This module calibrates channel level color statistics through feature fusion using multi scale refinement processing, color correction, and anti aliasing convolutions to extract statistical descriptors like vcolor from global, regional, and local scales.
-
Bounded Kendall Uncertainty Weighting Scheme: This scheme prevents either task from dominating the shared representation by using a learnable log variance to modulate task weights between wmin and wmax, ensuring
neither task starves the shared encoder while allowing adaptive modulation of relative task importance throughout training.
-
Improved S2O Translation Quality: On datasets like WHU-OPT-SAR, BMT achieves competitive translation quality by utilizing auxiliary losses like Lgrad (gradient loss) and Lhist (color histogram loss), which
directly penalizes color-distribution drift, such as background color bleeding into ship targets.
-
Enhanced Semantic Segmentation Accuracy: The framework improves segmentation metrics by applying a cycle segmentation loss and auxiliary losses, resulting in BMT achieving the highest semantic segmentation metrics on the source SAR dataset, with mIoU reaching 88.51% and mPA reaching 93.07%.
Abstract
Synthetic Aperture Radar (SAR) remains available through cloud and darkness, yet speckle and non-intuitive scattering make it hard to inspect and interpret. SAR-to-optical (S2O) translation renders radar observations as optical-like images for visual access, while semantic segmentation on SAR remains necessary since land-cover decisions cannot rely on synthesised pixels. We propose BMT, a hierarchical Vision Transformer coupling both tasks on a shared encoder via multi-scale one-hot conditioning, a gated LocalViTBlock, a colour-calibrating generation head, ControlNet-style zero-initialized injection, and uncertainty-based task weighting. The joint framework's value is not label-free training: it uses paired optical images and the same pixel annotations as a dedicated SAR segmentor, supervising the SAR decoder and teacher-forcing the generator with ground-truth one-hot maps. At test time, the model takes only SAR and returns an optical-like readout and a radar-decided land-cover map, with the translation objective supplying dense appearance supervision that speckle lacks. On WHU-OPT-SAR, the predicted-condition model achieves the best SSIM and PSNR among compared translators; its land-cover map surpasses the same backbone trained for labelling alone and is within about one point of DeepLabV3+.
Sources
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs