FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

summary

The gist

The gist The proposed framework FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals and complements this design with target-conditioned

In short

FuseAdapt is a parameter-efficient framework that fuses multi-modal data within a frozen vision foundation model's adaptation space. It uses lightweight, modality-specific branches and FFN low-rank residuals for cross-modal exchange. It also introduces Predictive Modality Modeling (PMM) to handle missing sensors during inference by predicting the missing modality's embedding based on available ones.

Key concepts

Adaptation Space Fusion: This method performs fusion inside the adaptation space of a frozen vision foundation model (VFM). Instead of adding new fusion layers, it learns modality-specific adjustments (like low-rank updates to Q/V projections) and routes only FFN low-rank residual activations across modalities. This keeps the core VFM fixed while learning how to combine different inputs effectively.
Predictive Modality Modeling (PMM): PMM is a training technique designed for missing modalities. During training, it masks a target modality and forces the model to predict its embedding using only the visible modalities. This prediction is supervised by a 'teacher' target computed when all modalities are present, creating a redundancy that helps the model handle incomplete sensor data during testing.
Low-Rank Adaptation (LoRA) in Fusion: The framework keeps Query (Q) and Value (V) low-rank adaptation private to each modality. This means only small, trainable updates are applied to these projections for each input stream. Cross-modal interaction is then achieved by routing only the FFN low-rank residual activations across modalities, ensuring that unavailable sources are masked before they affect the final output.
Missing Modality Segmentation (MISS): This refers to testing performance when one or more sensor modalities are absent during inference. FuseAdapt is specifically designed to improve segmentation accuracy in this 'missing modality' scenario, achieving significant gains over prior methods by leveraging the learned cross-modal interactions and the PMM predictive modeling objective.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Federated Learning — explained in 17 episodes
Flow Matching — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 17 episodes

This episode discusses

The paper

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities · Read on arXiv

Xin Zhang, Robby T. Tan

National University of Singapore

Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose FuseAdapt, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs adaptation-space fusion by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce Predictive Modality Modeling (PMM), a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +12.56 mIoU under missing modalities over prior state-of-the-art. Project page: https://devinxzhang.github.io/FuseAdapt-Page/.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities".

Jane: The gist The proposed framework FuseAdapt separates modality-private attention adaptation from cross-modal exchange through FFN low-rank residuals and complements this design with target-conditioned missing-modality latent prediction to improve both full…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, looking at "FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities," the authors are proposing this parameter-efficient framework that fuses information within the adaptation space of a frozen vision foundation model.

Jane: What they're saying is that this approach explores whether the foundation model's existing priors can actually serve as an effective shared backbone for multi-modal segmentation.

Lu: Essentially, they move away from adding standalone fusion networks and instead use lightweight adaptation branches for each modality and adaptation-space fusion by keeping the query and value low-rank adaptations private to each modality.

Meng: And the whole mechanism relies on routing only FFN low-rank residual activations across modalities inside that frozen backbone to get cross-modal exchange happening.

Tom: And they handle missing data with Predictive Modality Modeling, which is a direct representation-level objective for cross-modal redundancy during training, where they mask a target modality and predict its embedding from the visible ones.

Jane: The main implication here is that this method can significantly improve segmentation performance even when some sensors are unavailable at test time.

Lu: It suggests that representation learning from large foundation models can be used to build a backbone capable of handling the complexities of multi-modal input robustly.

Tom: So, in simple terms, they separated how each modality adapts privately from how they exchange information through those low-rank residuals and PMM complements this design with target-conditioned missing-modality latent prediction.

Jane: That’s a concise way to put it. The paper shows that this approach yields performance improvements across three benchmarks, MCubeS, DELIVER, and MUSES for both full and missing modality segmentation with a small trainable budget.

Conclusion: Tom: So, FuseAdapt is this parameter-efficient way to fuse multiple sensors when some of them are missing at test time. It’s been about a few weeks since this paper came out, and I want to talk about what it actually does.

Jane: It’s really smart because instead of building a huge fusion network, they keep the main vision model frozen and only train small "adaptation branches" for each sensor input. The core idea is that the foundation model already understands vision really well, so we just fine-tune it slightly to handle different inputs.

Lu: Exactly. They are doing this "adaptation-space fusion," which means they keep the query and value projections low-rank adaptation private to each modality, and then they only route FFN low-rank residual activations across modalities inside that frozen backbone. It’s keeping the heavy lifting in the frozen model while learning how to mix them intelligently.

Meng: From a deployment standpoint, that sounds promising for efficiency because we aren't adding a massive new module to run constantly. But what about the missing sensor part? How does it actually handle when one of those inputs is gone during real-time operation?

Jane: That’s where Predictive Modality Modeling, or PMM, comes in. During training, they mask one target modality and make the model predict that modality's embedding using only the ones that are visible. It’s a way to build redundancy directly into the representation learning process.

Tom: So it’s a direct way to learn cross-modal relationships even when data is missing, which is much better than relying on a separate fusion module that might just break if one stream goes dark. The results they show—up to plus three point zero one mIoU under full modalities and twelve point five six mIoU when modalities are missing—they are pretty strong.

Jane: That improvement in the missing modality case is really what caught my attention. It shows that this approach doesn't just work on perfect data; it actually gets better when things go wrong, which is crucial for real-world applications like autonomous driving or robotics.

Lu: The ablation studies confirm that routing those adaptation exchanges across the FFNs consistently improves performance, so it’s not just random mixing; the model learns *how* to mix based on what's available. It confirms the idea that different modalities have distinct roles in feature refinement.

Meng: I'm still watching how much train time this takes versus a standard multi-branch encoder setup. If the adaptation paths are too complex, it could slow down our inference pipeline at scale. What is the actual parameter count increase we’re looking at for these adaptation branches?

Jane: They keep the trainable budget small, which is the key word here. They show that you get significant gains in segmentation performance without needing a massive amount of new parameters to train or store. It’s very focused on being practical for existing vision foundation models.

Tom: So, looking at the title again—FuseAdapt—it really captures the essence: it’s parameter-efficient fusion that specifically tackles missing modalities. It moves away from brute-force fusion and instead uses smart adaptation within a frozen backbone, and PMM handles the missing data gracefully during inference.

Jane: Right. What this paper means for us is that we don't always need a completely new, massive architecture just to handle multi-sensor input; we can adapt existing powerful models in a much more targeted way to make them robust against real-world sensor failures. (Sound of gentle radio static swelling slightly)

Tom: This leads us perfectly into how this works technically, so let's see exactly how they built those adaptation branches next.

More episodes

← Home