ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation

summary

The gist

SAR-to-EO image translation aims to generate interpretable electrooptical (EO) imagery from synthetic aperture radar (SAR) observations Because the latent codec affects the roundtrip preservation of…

In short

ReFlowSET generates interpretable electrooptical (EO) images from synthetic aperture radar (SAR) data by using a conditional latent flow-matching framework. It first audits multiple VAEs to select the best latent codec, then trains a smaller diffusion model from scratch. This model uses dual-stream SAR conditioning and aligns intermediate features with clean EO representations during training, leading to state-of-the-art results.

Key concepts

Codec Selection Strategy
The method systematically tests several pretrained Variational Autoencoders (VAEs) on both SAR and EO images. It selects the VAE that provides the best joint reconstruction fidelity for both modalities, ensuring the chosen latent space minimizes distortion when translating SAR to EO imagery.
Dual-Stream SAR Conditioning
The model uses two separate streams to process input: one stream handles the SAR condition, and another handles a noisy EO latent representation. These are processed independently before being fused later in the network, preserving modality-specific information during initial processing.
Training-Only EO Representation Alignment
During training, intermediate features from the model are aligned with clean target EO representations extracted by a frozen foundation model. This alignment minimizes a cosine distance loss without adding any inference cost, guiding the generative process toward realistic EO outputs.

Terminology used across episodes

This episode discusses

The paper

ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation · Read on arXiv

KAIST · Stellarvision Inc.

SAR-to-EO image translation aims to generate electro-optical (EO) imagery from synthetic aperture radar (SAR) observations. Existing latent diffusion approaches typically inherit a predetermined autoencoder, although reconstruction fidelity can vary substantially across codecs and modalities. Because the latent codec affects the round-trip preservation of both SAR conditions and EO targets, codec selection constitutes a fundamental design choice; nevertheless, existing methods largely rely on codecs pretrained on natural images. To remedy this, we introduce ReFlowSET, a conditional latent flow-matching framework that selects its codec through a joint SAR--EO reconstruction audit. Rather than inheriting a heavyweight pretrained generator, ReFlowSET trains a substantially smaller conditional DiT from scratch in the selected latent space, using dual-stream SAR conditioning followed by joint feature refinement. To provide semantic guidance for this from-scratch training, intermediate noisy-EO features are aligned with clean target-EO representations extracted by a frozen vision foundation model. This alignment is used only during training and introduces no additional inference cost. Experiments on QXS-SAROPT and SAR2Opt demonstrate state-of-the-art performance across diverse perceptual fidelity and distributional metrics. Code and pretrained weights are publicly available at https://github.com/KAIST-VICLab/ReFlowSET.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation".

Jane: Here's the gist of the paper. SAR-to-EO image translation aims to generate interpretable electrooptical (EO) imagery from synthetic aperture radar (SAR) observations Because the latent codec affects the roundtrip preservation of both SAR conditions and EO targets, codec selection constitutes a fundamental design choice ReFlowSET introduces a conditional latent flow-matching framework that selects its codec through a joint SAR–EO reconstruction audit Codec Selection Strategy The method first evaluates the modality-specific reconstruction ceilings of multiple pretrained VAEs to select the best latent space for SET The authors systematically evaluate multiple pretrained VAEs on SAR and EO imagery, revealing their modality-dependent reconstruction ceilings and identifying a suitable latent space for SET For instance, FLUX.2 provides the best reconstruction fidelity in six of eight dataset–modality settings and improves the mean EO reconstruction ceiling by 7.49 dB over SD2.1 This reconstruction audit motivates the selection of FLUX.2 as a less distortion-limited latent codec The authors retain only the frozen FLUX.2 autoencoder for both SAR and EO in all subsequent experiments Training Signal Construction ReFlowSET trains a substantially smaller conditional DiT from scratch using conditional flow matching The DiT employs dual-stream SAR conditioning that preserves modality-specific processing before channelwise fusion and joint refinement To provide semantic guidance for this from-scratch training, intermediate noisy-EO features are aligned with clean target-EO representations extracted by a frozen vision foundation model This alignment is used only during training and introduces no additional inference cost The total training objective is L = LFlow + λLRe, where LFlow minimizes the flow matching loss and λLRe minimizes the representation alignment loss Key Architectural Components The framework combines conditional latent flow matching, dual-stream SAR conditioning, and training-only EO representation alignment The conditional DiT receives the SAR latent z s as a persistent condition The dual-stream conditioning separately projects the SAR condition and noisy-EO latent, processing them using independently parameterized streams for the first r DiT blocks The training-only EO representation alignment minimizes the cosine distance LRe between aligned features and clean target representations Experimental Validation Extensive evaluations on QXS-SAROPT and SAR2Opt validate the proposed design, with ReFlowSET attaining leading performance in perceptual similarity and distributional realism On QXS-SAROPT, ReFlowSET obtains the best DISTS and ties the best FID On SAR2Opt, it improves the second-best FID from 71.8 to 66.3 and DISTS from 0.211 The results demonstrate that the from-scratch generator reaches or surpasses pretrained-generator baselines on perceptual and distributional metrics The alignment branch introduces no inference-time overhead ReFlowSET achieves the best FID and DISTS on both benchmarks and the best LPIPS on SAR2Opt among the evaluated methods ReFlowSET combines reconstruction-guided codec selection, conditional latent flow matching, delayed SAR–EO fusion, and training-only EO representation alignment Future work will examine cross-sensor generalization and the suppression of unsupported EO-like structures Page 1 --- ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation for SAR-to-EO Image Translation Jeonghyeok Do KAIST ehwjdgur0913@kaist.ac.kr Seungchul Lee Stellarvision Inc.

Tom: leesc@stellarvision.kr Munchurl Kim SAR-to-EO image translation aims to generate electrooptical (EO) imagery from synthetic aperture radar (SAR) observations Existing latent diffusion approaches typically inherit a predetermined autoencoder, although reconstruction fidelity can vary substantially across codecs and modalities Because the latent codec affects the roundtrip preservation of both SAR conditions and EO targets, codec selection constitutes a fundamental design choice nevertheless, existing methods largely rely on codecs pretrained on natural images To remedy this, we introduce ReFlowSET, a conditional latent flow-matching framework that selects its codec through a joint SAR–EO reconstruction audit Rather than inheriting a heavyweight pretrained generator, ReFlowSET trains a substantially smaller conditional DiT from scratch in the selected latent space, using dual-stream SAR conditioning followed by joint feature refinement To provide semantic guidance for this from-scratch training, intermediate noisy-EO features are aligned with clean target-EO representations extracted by a frozen vision foundation model This alignment is used only during training and introduces no additional inference cost Experiments on QXS-SAROPT and SAR2Opt demonstrate state-of-the-art performance across diverse perceptual fidelity and distributional metrics Code and pretrained weights are publicly available at https://github.com/KAIST-VICLab/ReFlowSET 1.

Jane: Introduction Synthetic aperture radar (SAR) provides reliable Earth observation regardless of illumination and under most weather conditions However, its coherent imaging mechanism introduces speckle and geometric distortions, making SAR imagery less intuitive to interpret than electro-optical (EO) imagery SAR-to-EO image translation (SET)

29: aims to generate an interpretable EO representation from a corresponding SAR observation, supporting visual analysis when optical measurements are unavailable This task remains challenging because SAR backscatter and EO reflectance describe fundamentally different physical properties, resulting in an inherently ambiguous cross-modal mapping Early SET approaches

6, 10, 11, 23: primarily relied on generative adversarial networks (GANs), incorporating multiscale context, scene information, or semantic supervision Although these methods enable direct translation, adversarial training may produce unstable textures and semantically incorrect objects Diffusion-based approaches

1, 4, 9, 18: provide a more stable generative objective and improved perceptual quality Recent latent methods

4, 9: further reduce computation but adopt the pretrained Stable Diffusion (SD) variational autoencoder (VAE)

19: as a fixed codec without examining its suitability for SAR and EO imagery Under latent-target training, the EO reconstruction represents a directly reachable decoded endpoint, while the SAR reconstruction indicates how much source structure is retained for conditioning Moreover, existing latent SET models commonly inject SAR through immediate channel concatenation

4: This position-wise fusion may prematurely entangle modalities with different statistics, particularly when paired SAR–EO observations contain local spatial misalignment We introduce ReFlowSET, a representation-aligned latent flow-matching framework for SET We first evaluate the modality-specific reconstruction ceilings of multiple pretrained VAEs

2, 5, 17, 19: and select the codec providing the strongest joint fidelity for SAR and EO imagery Rather than inheriting the selected codec’s heavyweight pretrained generator, we train a substantially smaller conditional DiT from scratch using conditional flow matching

13: The DiT employs dual-stream SAR conditioning that preserves modality-specific processing before channelwise fusion and joint refinement Because the from-scratch training, we align its intermediate noisy-EO features with clean-EO representations extracted by a frozen vision foundation model This alignment is used only during training and introduces no additional annotations nor inference cost Experiments on QXS-SAROPT and SAR2Opt demonstrate state-of-the-art performance across diverse perceptual fidelity and distributional metrics Page 2 --- 1 ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation for SAR to EO Image Translation 2.

Tom: Related Work 2.1.

Paper summary: Jane: So to wrap up on "ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation," the authors essentially introduced a method where you first audit multiple autoencoders to pick the best one, and then you train a smaller conditional diffusion model from scratch using flow matching in that specific space >

Tom: And they achieved better performance on key metrics like FID and DISTS on both QXS-SAROPT and SAR2Opt datasets compared to pretrained generator baselines >

Lu: It really boils down to combining reconstruction-guided codec selection with conditional latent flow matching, dual stream conditioning, and a training-only alignment step to get a more robust translation system >

Meng: For someone looking at deployment, the fact that they train from scratch instead of relying on a massive pre-trained generator is appealing because it means you're deploying something much smaller than the original model size >

Jane: And the limitation they admit is that while it does this well on these benchmarks, future work will look at how well this method works when trying to generalize it across different sensors or how to suppress those optical-like structures in the final output >

Tom: So, ReFlowSET gives us a way to select the right latent space and train an efficient generator that actually performs better than what we could get by just sticking with a pre-trained model >

Conclusion: Tom: So, we've been looking at ReFlowSET, which is this new framework for SAR to EO image translation.

Jane: Right, so after all that technical stuff about selecting the best autoencoder and training a model from scratch using flow matching... what's the actual takeaway here?

Tom: The main thing is that they picked a specific latent space—FLUX two in this case—and then trained their diffusion model from scratch. They didn't just use a pre-trained one.

Lu: From my angle, what’s interesting is how they used that training process to align the noisy features with clean target representations using a frozen foundation model during training only. It keeps it efficient for inference.

Meng: But how much smaller is this model compared to the ones you're usually shipping? That's what matters for deployment.

Jane: They claim it achieves top performance on two different benchmarks, QXS-SAROPT and SAR2Opt, beating existing methods in both perceptual similarity and distributional realism.

Tom: So we have a method that seems to solve the codec selection problem by auditing reconstructions and then trains a smaller model from scratch to get better results than what we can just take off the shelf.

Jane: It definitely sounds like they’ve tightened up the whole process, moving away from relying on fixed pre-trained components.

Lu: The paper emphasizes that this combination of reconstruction-guided codec selection and flow matching is what drives the gains, not just one single trick.

Meng: So for someone actually trying to deploy this, it means you can get a high-quality result without needing the massive infrastructure required by those pre-trained models.

Tom: Exactly. It seems like they’ve managed to make a lot of sense out of the latent space choice and the training signal construction.

Jane: And while they're really happy with these results, they do mention that future work will look at generalizing this across different sensors or maybe dealing with those optical-like structures in the final image.

More episodes

← Home