SphereFlow: Missing Modality Imputation via Geometric Transport on Hypersphere
cs.CV
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: Accepted to NeurIPS 2026. 23 pages, 7 figures. Code: https://github.com/WTHH031230/SphereFlow
Code: https://github.com/WTHH031230/SphereFlow
License: http://creativecommons.org/licenses/by/4.0/
The gist: Missing modality imputation aims to synthesize unobserved modalities from observed ones.
Terminology
Abstract
Missing modality imputation aims to synthesize unobserved modalities from observed ones. Existing methods treat each modality as living on its own latent manifold, forcing the generator to learn a long, unstructured jump between them. We present SphereFlow, which takes a fundamentally different view: by projecting all modalities into a shared hyperspherical latent space via a frozen self-supervised encoder, cross-modal imputation reduces to short-range geometric transport between nearby points on the sphere. Realizing this idea with flow matching, however, exposes two geometric conflicts: the standard Gaussian noise source is catastrophically mismatched with the unit-norm data manifold, and the linear interpolation path departs the sphere surface into a region where the decoder has no training signal. We resolve both with simple, geometry-aware modifications: replacing the noise source with the observed-modality latent so that the velocity field learns only the cross-modal displacement, and training the decoder with a time-weighted loss that extends its domain to the near-surface region traversed by the flow. We prove that this data-as-source formulation reduces the worst-case off-manifold deviation from a quantity that grows with the latent dimension to a small, dimension-independent bound governed by inter-modality similarity. Experiments on BraTS multi-modal MRI imputation and CT-MRI translation demonstrate state-of-the-art generation fidelity across all settings, while downstream brain tumor segmentation with our imputed data closes roughly eighty percent of the gap to the full-modality oracle. Code is available at https://github.com/WTHH031230/SphereFlow.
Sources
- The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
- Hyperspherical Autoencoder for High-Fidelity Image Reconstruction and Generation
- Flow Matching on General Geometries
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Auto-Encoding Variational Bayes
- Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoders
- Flow Matching for Generative Modeling
- Improving Reconstruction of Representation Autoencoder
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- DINOv2: Learning Robust Visual Features without Supervision
- Score-Based Generative Modeling through Stochastic Differential Equations
- Soft Masked Mamba Diffusion Model for CT to MRI Conversion
- Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised Code
- Making Reconstruction FID Predictive of Diffusion Generation FID
- Diffusion Transformers with Representation Autoencoders
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs