Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles

arXiv:2607.05019 · cs.LG, cs.CV · Submitted 2026-07-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles".

Jane: Here's the gist of the paper. In this work, deep ensembles of unimodal networks can effectively classify multimodal data without explicit modality fusion The main contribution is that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance This advantage holds over intermediate-fusion techniques and hybrid methods as well.

Tom: Core Findings on Ensemble Superiority The study systematically compares deep ensembles to late-fusion networks at equal parameter count and shows that ensembles consistently outperform state-of-the-art late-fusion methods specifically designed to address modality imbalance This advantage also holds over the intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions.

Jane: The work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks.

Tom: Heuristic for Ensemble Selection The paper proposes and empirically validates a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search.

Paper summary: Tom: So we've seen the high level idea—this paper, "Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles," is arguing that you can classify multimodal data without any explicit modality fusion by using deep ensembles of unimodal networks.

Jane: They are specifically highlighting that when modality imbalance is pronounced, these ensembles consistently outperform late-fusion methods that are usually designed to fix that imbalance problem.

Lu: The core thesis is that multimodal data can be effectively classified simply by training deep neural networks independently on the complete dataset for each modality and then combining their independent predictions in this ensemble structure.

Meng: It's interesting because it means we don't have to worry about designing a fusion layer at all, which simplifies the architecture significantly when you think about deployment costs.

Lalam: They call these ensembles heterogeneous deep ensembles or HeDEs when the models are trained on different modalities, contrasting them with homogeneous deep ensembles or HoDEs that only use one modality.

Tom: And they did a lot of comparative work, showing this advantage holds even over intermediate fusion techniques and hybrid methods that try to combine unimodal and multimodal predictions.

Jane: They also presented a proposed method for selecting the optimal number of models per modality in an ensemble, which is empirically validated through experiments rather than just guesswork.

Lu: This selection process has a specific rule: under extreme imbalance, you start with unimodal models trained only on the stronger modality, and as the ensemble grows, incorporating models from the weaker modality becomes beneficial.

Meng: That heuristic to select model counts sounds practical for setting up initial training runs because it avoids having to test every possible configuration of model numbers.

Tom: And they established some scaling laws too, connecting how fast individual unimodal models lose accuracy to the asymptotic performance of the heterogeneous deep ensemble as you scale up.

Jane: This helps predict how much performance gains you can expect just by adding more unimodal components to your system over time.

Conclusion: Tom: Looking at "Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles," the authors are really pushing back on the idea that explicit fusion is the only way to handle multimodal data.

Jane: They're showing that deep ensembles of unimodal networks can effectively classify multimodal data even when there's significant imbalance between those modalities.

Lu: The title itself points to "Fusion-Free," meaning they are demonstrating classification without any explicit modality fusion happening in the network architecture.

Meng: For someone building a production system, this means you can keep your model structure cleaner by training unimodal models separately and then using this ensemble technique for the final decision.

Lalam: They also found that logit averaging is superior to probability averaging in these setups, leading to better calibrated predictions for the final output.

Tom: So what this means practically is that we might be able to achieve good multimodal results by focusing on training strong unimodal models and managing their ensemble composition smartly rather than building a complex fusion layer.

Ilya Burenko, Dmitry Vetrov

ScaDS.AI · Constructor University

cs.LG, cs.CV

Submitted: 2026-07-06

Updated: 2026-11-13

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: In this work, deep ensembles of unimodal networks can effectively classify multimodal data without explicit modality fusion The main contribution is that ensembles consistently outperform…

Key concepts

Deep Ensembles
This involves training multiple deep neural networks independently on different parts of the data or different modalities. Instead of combining their outputs directly (fusion), the final prediction is made by aggregating the predictions from all these individual models, leveraging their collective knowledge.
Late-Fusion Methods
These are fusion techniques where individual unimodal models make predictions first, and only then are those predictions combined at a later stage. The paper finds that deep ensembles consistently beat these methods when dealing with imbalanced data scenarios.
Heuristic for Ensemble Selection
This is a rule proposed to decide how many models of each modality should be in the ensemble. The rule suggests starting with models from the stronger modality and gradually adding models from the weaker one as the ensemble grows, predicting a shift toward heterogeneous ensembles.
Logit Averaging
Instead of averaging probabilities (which can be inaccurate), this technique averages the raw output scores (logits) from different models in a heterogeneous ensemble. This method is found to yield better calibrated predictions with lower Expected Calibration Error.

Terminology

Summary

In this work, deep ensembles of unimodal networks can effectively classify multimodal data without explicit modality fusion The main contribution is that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance This advantage holds over intermediate-fusion techniques and hybrid methods as well. p. 1

Where the paper says it

We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods specifically designed to address modality imbalance. This advantage also holds over the intermediate-fusion techniques we evaluated, for both CNN and Transformer architectures, as well as over hybrid approaches that combine unimodal and multimodal predictions.

Core Findings on Ensemble Superiority

The study systematically compares deep ensembles to late-fusion networks at equal parameter count and shows that ensembles consistently outperform state-of-the-art late-fusion methods specifically designed to address modality imbalance This advantage also holds over the intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions. The work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. p. 0

Where the paper says it

When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks.

Heuristic for Ensemble Selection

The paper proposes and empirically validates a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial. p. 0

Where the paper says it

We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial.

The heuristic predicts a transition to heterogeneous ensembles as the ensemble grows, which is consistently confirmed by experiments. p. 1
Where the paper says it

We propose and empirically validate a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance, the heuristic favors homogeneous ensembles at small sizes but predicts a transition to heterogeneous ensembles as the ensemble grows, which is consistently confirmed by our experiments.

Synthetic Data Construction

The authors propose a synthetic multimodal dataset construction that uses real-world data and provides control over the number of modalities and their predictive strength. This construction allows training overparameterized networks, revealing optimization challenges observed in practice. p. 1

Where the paper says it

We propose a synthetic multimodal dataset construction that uses real-world data and provides control over the number of modalities and their predictive strength. Unlike prior synthetic benchmarks, our construction allows training overparameterized networks, revealing optimization challenges observed in practice.

The findings are consistent across synthetic and real-world datasets of varying sizes and input modalities. p. 1
Where the paper says it

Unlike prior synthetic benchmarks, our construction allows training overparameterized networks, revealing optimization challenges observed in practice. Our findings are consistent across synthetic and real-world datasets of varying sizes and input modalities.

Scaling Laws and Asymptotic Performance

The authors fit scaling laws to both synthetic and real-world bimodal datasets, showing that the test losses of unimodal models reliably predict the asymptotic test loss of the heterogeneous deep ensemble. They establish that for any fixed proportion of unimodal models in HeDEs, a power law can be fitted to the test loss as a function of ensemble size. p. 7

Where the paper says it

7 Scaling laws In this section, we fit scaling laws to estimate how the test loss of heterogeneous deep ensembles changes as a function of ensemble size. Our goals are twofold: first, to show that for any fixed proportion of unimodal models in HeDEs, a power law can be fitted to the test loss as a function of ensemble size; second, to show that the asymptotic test loss of the ensemble can be reliably predicted from unimodal model losses.

Comparison with Other Fusion Techniques

The paper compares deep ensembles to intermediate-fusion networks and hybrid approaches that combine unimodal and multimodal models. Deep ensembles consistently outperform both intermediate-fusion networks and hybrid approaches on synthetic and real-world datasets at equal parameter count. p. 7

Where the paper says it

0 Table 3: Deep ensembles outperform hybrid approaches on synthetic and real-world datasets at equal parameter count. The choice of a vanilla late-fusion network as the multimodal component of I2M2 might be suboptimal.

Furthermore, logit averaging consistently outperforms probability averaging in heterogeneous deep ensembles

Reliability and Calibration

Logit-averaging heterogeneous deep ensembles yield well-calibrated predictions. The Expected Calibration Error (ECE) analysis shows that logits-averaging heterogeneous deep ensembles consistently achieves the lowest ECE, substantially improving over both unimodal models regardless of imbalance level or ensemble size. p. 3

Where the paper says it

We find it crucial to average output logits rather than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we present reliability diagrams and Expected Calibration Error analysis in Appendix F.

Limitations and Future Work

The work discusses limitations such as the synthetic dataset being class-aligned rather than instance-aligned It also notes that fusion networks can be designed to match deep ensemble performance, but a broader evaluation across fusion techniques is needed. The heuristic's theoretical basis remains unclear and needs further addressing.

Reproducibility and Resources

All training details, hyperparameters, and data splits are provided in Appendix C Baseline results were reproduced using the authors’ original code repositories. All datasets used are publicly available and properly cited. p. 5

Where the paper says it

We use three datasets: CREMA-D (small, audio-visual) [5], Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D The CREMA-D dataset is widely used to assess late-fusion networks under modality imbalance [43, 58, 26, 13, 22, 18, 19, 30].

p. 5
Where the paper says it

We use three datasets: CREMA-D (small, audio-visual) [5], Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D The CREMA-D dataset is widely used to assess late-fusion networks under modality imbalance [43, 58, 26, 13, 22, 18, 19, 30].

p. 5
Where the paper says it

We use three datasets: CREMA-D (small, audio-visual) [5], Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D The CREMA-D dataset is widely used to assess late-fusion networks under modality imbalance [43, 58, 26, 13, 22, 18, 19, 30].

p. 5
Where the paper says it

We use three datasets: CREMA-D (small, audio-visual) [5], Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D The CREMA-D dataset is widely used to assess late-fusion networks under modality imbalance [43, 58, 26, 13, 22, 18, 19, 30].

Wall-clock training times and peak RAM memory for all methods and datasets are reported in Appendix H. p. 3
Where the paper says it

A detailed comparison of training cost, memory usage, and parameter counts is provided in Appendix H. 6 Results We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8].

p. 3
Where the paper says it

A detailed comparison of training cost, memory usage, and parameter counts is provided in Appendix H. 6 Results We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8].

The research conforms with the NeurIPS Code of Ethics The work does not release pre-trained models or new datasets that pose risks for misuse. p. 1
Where the paper says it

Another line of research focuses on updating modality-specific weights independently of the weights used to extract features from other modalities [61, 18]. Additionally, some work uses unimodal pre-trained features [12] or updates them iteratively [13] to steer the multimodal learning process.

All experiments were conducted on H100 GPUs. p. 1
Where the paper says it

We propose and empirically validate a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance, the heuristic favors homogeneous ensembles at small sizes but predicts a transition to heterogeneous ensembles as the ensemble grows, which is consistently confirmed by our experiments.

p. 1
Where the paper says it

We propose and empirically validate a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance, the heuristic favors homogeneous ensembles at small sizes but predicts a transition to heterogeneous ensembles as the ensemble grows, which is consistently confirmed by our experiments.

Broader Impacts

The authors discuss potential societal impacts, noting that improved multimodal classifiers could be applied in sensitive domains such as surveillance or medical diagnosis. The work does not release new assets or models that pose risks for misuse. p. 8

Where the paper says it

Addressing these gaps could guide the design of improved ensemble selection and aggregation methods. As a methodological study on multimodal classification, this work does not introduce direct societal risks; however, improved multimodal classifiers could be applied in sensitive domains such as surveillance or medical diagnosis, where careful deployment considerations apply.

The paper does not involve crowdsourcing or research with human subjects. p. 2
Where the paper says it

These approaches construct synthetic data by sampling shared and modality-specific vectors from multivariate normal distributions and processing them through shallow MLPs. While these approaches allow control over mutual information between modalities, they do not involve training overparameterized networks on real-world inputs and cannot reproduce optimization challenges observed in practice, such as the faster convergence of the weaker modality [53].

The core methods in this research do not involve LLMs as any important, original, or non-standard components. All datasets used are standard research benchmarks. p. 1
Where the paper says it

Building on this, a combination of gradient modulation and loss regularization has also been explored in [60, 56]. Another line of research focuses on updating modality-specific weights independently of the weights used to extract features from other modalities [61, 18].

Licenses and Assets

All datasets used are publicly available and properly cited Baseline implementations were obtained from the authors’ publicly released code repositories. The name of the license (e.g., CC-BY 4.0) should be included for each asset. Code to reproduce all experiments is provided as supplementary material. p. 2

Where the paper says it

These approaches construct synthetic data by sampling shared and modality-specific vectors from multivariate normal distributions and processing them through shallow MLPs. While these approaches allow control over mutual information between modalities, they do not involve training overparameterized networks on real-world inputs and cannot reproduce optimization challenges observed in practice, such as the faster convergence of the weaker modality [53].

All datasets used are publicly available and properly cited. The authors should provide instructions on data access and preparation. p. 1
Where the paper says it

Regardless of the type of fusion, access to multiple modalities should improve classification. However, research has demonstrated that naive late-fusion networks, which classify based on concatenated features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal classifier [53].

p. 1
Where the paper says it

Regardless of the type of fusion, access to multiple modalities should improve classification. However, research has demonstrated that naive late-fusion networks, which classify based on concatenated features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal classifier [53].

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. All datasets used are publicly available and properly cited. The name of the license (e.g., CC-BY 4.0) should be included for each asset. All datasets used are publicly available and properly cited. The authors should provide instructions on data access and preparation.

Experimental Setting Details

Full training details including optimizers, learning rate schedules, batch sizes, augmentation strategies, and data preprocessing are provided in Appendix C. The synthetic dataset construction is described in Section 4. p. 2

Where the paper says it

In this section, we demonstrate that deep ensembles of unimodal networks outperform late-fusion classifiers across a range of modality imbalance levels. To illustrate this, we use the synthetic construction from Sec.

Computational costs are reported in Appendix H. p. 9
Where the paper says it

Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy, July 2019.

p. 9
Where the paper says it

Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy, July 2019.

All experiments were conducted on H100 GPUs. Wall-clock training times and peak RAM memory for all methods and datasets are reported in Appendix H. The paper does not include formal theoretical results All assumptions should be clearly stated or referenced in the statement of any theorems. p. 1
Where the paper says it

However, research has demonstrated that naive late-fusion networks, which classify based on concatenated features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal classifier [53]. To address this unexpected outcome, recent studies have examined the problem from both theoretical [20, 36, 62] and practical perspectives [43], seeking optimal methods for combining unimodal features in multimodal classification.

p. 1
Where the paper says it

However, research has demonstrated that naive late-fusion networks, which classify based on concatenated features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal classifier [53]. To address this unexpected outcome, recent studies have examined the problem from both theoretical [20, 36, 62] and practical perspectives [43], seeking optimal methods for combining unimodal features in multimodal classification.

p. 1
Where the paper says it

However, research has demonstrated that naive late-fusion networks, which classify based on concatenated features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal classifier [53]. To address this unexpected outcome, recent studies have examined the problem from both theoretical [20, 36, 62] and practical perspectives [43], seeking optimal methods for combining unimodal features in multimodal classification.

Statistical Significance

Results are reported as mean ± standard deviation across multiple runs (5 runs for CREMA-D, 10 ensembles for the ensemble-size-2 analysis). The factors of variability that the error bars are capturing should be clearly stated. p. 3

Where the paper says it

We find it crucial to average output logits rather than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we present reliability diagrams and Expected Calibration Error analysis in Appendix F.

The method for calculating the error bars should be explained. p. 3
Where the paper says it

We find it crucial to average output logits rather than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we present reliability diagrams and Expected Calibration Error analysis in Appendix F.

p. 3
Where the paper says it

We find it crucial to average output logits rather than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we present reliability diagrams and Expected Calibration Error analysis in Appendix F.

Experimental Result Reproducibility

All training details, hyperparameters, and data splits are provided in Appendix C Baseline results were reproduced using the authors’ original code repositories. Modifications to baseline implementations are documented in Appendix C. p. 4

Where the paper says it

approaches (single networks and HoDEs) operate on a single modality per model. All baseline results were reproduced using the authors’ original code repositories; details are in Appendix C.

Experimental Setting/Details

Full training details including optimizers, learning rate schedules, batch sizes, augmentation strategies, and data preprocessing are provided in Appendix C The synthetic dataset construction is described in Section 4. Computational costs are reported in Appendix H. All experiments were conducted on H100 GPUs Wall-clock training times and peak RAM memory for all methods and datasets are reported in Appendix H. p. 3

Where the paper says it

A detailed comparison of training cost, memory usage, and parameter counts is provided in Appendix H. 6 Results We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8].

The paper does not include formal theoretical results All assumptions should be clearly stated or referenced in the statement of any theorems.

Results are reported as mean ± standard deviation across multiple runs (5 runs for CREMA-D, 10 ensembles for the ensemble-size-2 analysis) The factors of variability that the error bars are capturing should be clearly stated. The method for calculating the error bars should be explained.

Full training details including optimizers, learning rate schedules, batch sizes, augmentation strategies, and data preprocessing are provided in Appendix C The synthetic dataset construction is described in Section 4. Computational costs are reported in Appendix H All experiments were conducted on H100 GPUs Wall-clock training times and peak RAM memory for all methods and datasets are reported in Appendix H. p. 3

Where the paper says it

A detailed comparison of training cost, memory usage, and parameter counts is provided in Appendix H. 6 Results We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8].

The paper does not include formal theoretical results All assumptions should be clearly stated or referenced in the statement of any theorems.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Modality imbalance handling using deep ensembles allows for effective classification without any explicit modality fusion by leveraging unimodal models trained on available modalities, which can be superior to late-fusion networks when imbalance is pronounced.

  2. A heuristic is proposed for selecting the number of models per modality in an ensemble, predicting a transition from homogeneous ensembles at small sizes to heterogeneous ensembles as the ensemble scales up.

  3. The system can estimate asymptotic test loss from unimodal model losses by fitting scaling laws, which complements the heuristic by suggesting that unimodal validation losses are sufficient to estimate both the composition and the expected performance of heterogeneous deep ensembles without training any.

  4. The AI system can be designed to select optimal modality allocation based on a formula derived from cross-entropy validation loss ratios: ni = round Nens / (1/l P i j 1/lj).

Abstract

In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance when trained from random initialization. This advantage also holds over intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal classifiers. We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable. As the ensemble scales up, incorporating models from the weaker modality becomes beneficial. Both predictions align with our empirical findings. However, the heuristic becomes less reliable when the diversity of ensemble members is low. To systematically explore the challenges of optimizing multimodal models, we propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength. Our findings are consistent across synthetic and real-world datasets. Finally, by fitting scaling laws to bimodal datasets, we estimate the asymptotic performance of ensembles.

Sources

Related papers