Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles".
Jane: Here's the gist of the paper. In this work, deep ensembles of unimodal networks can effectively classify multimodal data without explicit modality fusion The main contribution is that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance This advantage holds over intermediate-fusion techniques and hybrid methods as well.
Tom: Core Findings on Ensemble Superiority The study systematically compares deep ensembles to late-fusion networks at equal parameter count and shows that ensembles consistently outperform state-of-the-art late-fusion methods specifically designed to address modality imbalance This advantage also holds over the intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions.
Jane: The work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks.
Tom: Heuristic for Ensemble Selection The paper proposes and empirically validates a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search.
Paper summary: Tom: So we've seen the high level idea—this paper, "Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles," is arguing that you can classify multimodal data without any explicit modality fusion by using deep ensembles of unimodal networks.
Jane: They are specifically highlighting that when modality imbalance is pronounced, these ensembles consistently outperform late-fusion methods that are usually designed to fix that imbalance problem.
Lu: The core thesis is that multimodal data can be effectively classified simply by training deep neural networks independently on the complete dataset for each modality and then combining their independent predictions in this ensemble structure.
Meng: It's interesting because it means we don't have to worry about designing a fusion layer at all, which simplifies the architecture significantly when you think about deployment costs.
Lalam: They call these ensembles heterogeneous deep ensembles or HeDEs when the models are trained on different modalities, contrasting them with homogeneous deep ensembles or HoDEs that only use one modality.
Tom: And they did a lot of comparative work, showing this advantage holds even over intermediate fusion techniques and hybrid methods that try to combine unimodal and multimodal predictions.
Jane: They also presented a proposed method for selecting the optimal number of models per modality in an ensemble, which is empirically validated through experiments rather than just guesswork.
Lu: This selection process has a specific rule: under extreme imbalance, you start with unimodal models trained only on the stronger modality, and as the ensemble grows, incorporating models from the weaker modality becomes beneficial.
Meng: That heuristic to select model counts sounds practical for setting up initial training runs because it avoids having to test every possible configuration of model numbers.
Tom: And they established some scaling laws too, connecting how fast individual unimodal models lose accuracy to the asymptotic performance of the heterogeneous deep ensemble as you scale up.
Jane: This helps predict how much performance gains you can expect just by adding more unimodal components to your system over time.
Conclusion: Tom: Looking at "Fusion-Free Multimodal Classification with Heterogeneous Deep Ensembles," the authors are really pushing back on the idea that explicit fusion is the only way to handle multimodal data.
Jane: They're showing that deep ensembles of unimodal networks can effectively classify multimodal data even when there's significant imbalance between those modalities.
Lu: The title itself points to "Fusion-Free," meaning they are demonstrating classification without any explicit modality fusion happening in the network architecture.
Meng: For someone building a production system, this means you can keep your model structure cleaner by training unimodal models separately and then using this ensemble technique for the final decision.
Lalam: They also found that logit averaging is superior to probability averaging in these setups, leading to better calibrated predictions for the final output.
Tom: So what this means practically is that we might be able to achieve good multimodal results by focusing on training strong unimodal models and managing their ensemble composition smartly rather than building a complex fusion layer.
Ilya Burenko, Dmitry Vetrov
ScaDS.AI · Constructor University
cs.LG, cs.CV
Submitted: 2026-07-06
Updated: 2026-11-13
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: In this work, deep ensembles of unimodal networks can effectively classify multimodal data without explicit modality fusion The main contribution is that ensembles consistently outperform…
Key concepts
- Deep Ensembles
- This involves training multiple deep neural networks independently on different parts of the data or different modalities. Instead of combining their outputs directly (fusion), the final prediction is made by aggregating the predictions from all these individual models, leveraging their collective knowledge.
- Late-Fusion Methods
- These are fusion techniques where individual unimodal models make predictions first, and only then are those predictions combined at a later stage. The paper finds that deep ensembles consistently beat these methods when dealing with imbalanced data scenarios.
- Heuristic for Ensemble Selection
- This is a rule proposed to decide how many models of each modality should be in the ensemble. The rule suggests starting with models from the stronger modality and gradually adding models from the weaker one as the ensemble grows, predicting a shift toward heterogeneous ensembles.
- Logit Averaging
- Instead of averaging probabilities (which can be inaccurate), this technique averages the raw output scores (logits) from different models in a heterogeneous ensemble. This method is found to yield better calibrated predictions with lower Expected Calibration Error.
Terminology
Summary
In this work, deep ensembles of unimodal networks can effectively classify multimodal data without explicit modality fusion The main contribution is that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance This advantage holds over intermediate-fusion techniques and hybrid methods as well. p. 1 We systematically compare deep ensembles to late-fusion networks at equal parameter
count and show that ensembles consistently outperform state-of-the-art late-fusion methods
specifically designed to address modality imbalance. This advantage also holds over the
intermediate-fusion techniques we evaluated, for both CNN and Transformer architectures,
as well as over hybrid approaches that combine unimodal and multimodal predictions.Where the paper says it
Core Findings on Ensemble Superiority
The study systematically compares deep ensembles to late-fusion networks at equal parameter count and shows that ensembles consistently outperform state-of-the-art late-fusion methods specifically designed to address modality imbalance This advantage also holds over the intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions. The work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. p. 0 When modality imbalance
is pronounced, various regularization techniques have been proposed to balance the
learning process and overcome the inferior performance of late-fusion networks. In
contrast, this work demonstrates that multimodal data can be effectively classified
without any explicit modality fusion, using deep ensembles of unimodal networks.Where the paper says it
Heuristic for Ensemble Selection
The paper proposes and empirically validates a heuristic for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial. p. 0 We propose and empirically
validate a method for selecting the number of models per modality in an ensemble,
avoiding computationally expensive exhaustive search. Under extreme modality
imbalance and small ensemble sizes, the heuristic indicates that ensembles of
unimodal models trained solely on the stronger modality are preferable; as the
ensemble scales up, incorporating models from the weaker modality becomes
beneficial. We propose and empirically validate a heuristic for selecting the number of models per
modality in an ensemble, avoiding computationally expensive exhaustive search. Under
extreme modality imbalance, the heuristic favors homogeneous ensembles at small sizes
but predicts a transition to heterogeneous ensembles as the ensemble grows, which is
consistently confirmed by our experiments.Where the paper says it
Where the paper says it
Synthetic Data Construction
The authors propose a synthetic multimodal dataset construction that uses real-world data and provides control over the number of modalities and their predictive strength. This construction allows training overparameterized networks, revealing optimization challenges observed in practice. p. 1 We propose a synthetic multimodal dataset construction that uses real-world data and provides control over the number of modalities and their predictive strength. Unlike prior
synthetic benchmarks, our construction allows training overparameterized networks, revealing optimization challenges observed in practice. Unlike prior
synthetic benchmarks, our construction allows training overparameterized networks, revealing optimization challenges observed in practice. Our findings are consistent across
synthetic and real-world datasets of varying sizes and input modalities.Where the paper says it
Where the paper says it
Scaling Laws and Asymptotic Performance
The authors fit scaling laws to both synthetic and real-world bimodal datasets, showing that the test losses of unimodal models reliably predict the asymptotic test loss of the heterogeneous deep ensemble. They establish that for any fixed proportion of unimodal models in HeDEs, a power law can be fitted to the test loss as a function of ensemble size. p. 7 7 Scaling laws
In this section, we fit scaling laws to estimate how the test loss of heterogeneous deep ensembles
changes as a function of ensemble size. Our goals are twofold: first, to show that for any fixed
proportion of unimodal models in HeDEs, a power law can be fitted to the test loss as a function of
ensemble size; second, to show that the asymptotic test loss of the ensemble can be reliably predicted
from unimodal model losses.Where the paper says it
Comparison with Other Fusion Techniques
The paper compares deep ensembles to intermediate-fusion networks and hybrid approaches that combine unimodal and multimodal models. Deep ensembles consistently outperform both intermediate-fusion networks and hybrid approaches on synthetic and real-world datasets at equal parameter count. p. 7 0
Table 3: Deep ensembles outperform hybrid approaches on synthetic and real-world datasets at equal
parameter count. The choice of a vanilla late-fusion network as the multimodal component of I2M2 might be suboptimal.Where the paper says it
Reliability and Calibration
Logit-averaging heterogeneous deep ensembles yield well-calibrated predictions. The Expected Calibration Error (ECE) analysis shows that logits-averaging heterogeneous deep ensembles consistently achieves the lowest ECE, substantially improving over both unimodal models regardless of imbalance level or ensemble size. p. 3 We find it crucial to average output logits rather
than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving
accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we
present reliability diagrams and Expected Calibration Error analysis in Appendix F.Where the paper says it
Limitations and Future Work
The work discusses limitations such as the synthetic dataset being class-aligned rather than instance-aligned It also notes that fusion networks can be designed to match deep ensemble performance, but a broader evaluation across fusion techniques is needed. The heuristic's theoretical basis remains unclear and needs further addressing.
Reproducibility and Resources
All training details, hyperparameters, and data splits are provided in Appendix C Baseline results were reproduced using the authors’ original code repositories. All datasets used are publicly available and properly cited. p. 5 We use three datasets: CREMA-D (small, audio-visual) [5],
Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D
The CREMA-D dataset is widely used to assess late-fusion
networks under modality imbalance [43, 58, 26, 13, 22, 18, 19,
30]. We use three datasets: CREMA-D (small, audio-visual) [5],
Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D
The CREMA-D dataset is widely used to assess late-fusion
networks under modality imbalance [43, 58, 26, 13, 22, 18, 19,
30]. We use three datasets: CREMA-D (small, audio-visual) [5],
Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D
The CREMA-D dataset is widely used to assess late-fusion
networks under modality imbalance [43, 58, 26, 13, 22, 18, 19,
30]. We use three datasets: CREMA-D (small, audio-visual) [5],
Sarcasm (small, text-visual) [4], and Kinetics-400 (large, audiovisual) [24]. 1 CREMA-D
The CREMA-D dataset is widely used to assess late-fusion
networks under modality imbalance [43, 58, 26, 13, 22, 18, 19,
30]. A detailed comparison of training cost, memory usage, and parameter
counts is provided in Appendix H. 6 Results
We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8]. A detailed comparison of training cost, memory usage, and parameter
counts is provided in Appendix H. 6 Results
We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8]. Another line of research focuses on updating modality-specific weights independently of
the weights used to extract features from other modalities [61, 18]. Additionally, some work uses
unimodal pre-trained features [12] or updates them iteratively [13] to steer the multimodal learning
process. We propose and empirically validate a heuristic for selecting the number of models per
modality in an ensemble, avoiding computationally expensive exhaustive search. Under
extreme modality imbalance, the heuristic favors homogeneous ensembles at small sizes
but predicts a transition to heterogeneous ensembles as the ensemble grows, which is
consistently confirmed by our experiments. We propose and empirically validate a heuristic for selecting the number of models per
modality in an ensemble, avoiding computationally expensive exhaustive search. Under
extreme modality imbalance, the heuristic favors homogeneous ensembles at small sizes
but predicts a transition to heterogeneous ensembles as the ensemble grows, which is
consistently confirmed by our experiments.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Broader Impacts
The authors discuss potential societal impacts, noting that improved multimodal classifiers could be applied in sensitive domains such as surveillance or medical diagnosis. The work does not release new assets or models that pose risks for misuse. p. 8 Addressing
these gaps could guide the design of improved ensemble selection and aggregation methods. As a methodological study on multimodal classification, this work does not introduce direct societal
risks; however, improved multimodal classifiers could be applied in sensitive domains such as
surveillance or medical diagnosis, where careful deployment considerations apply. These approaches
construct synthetic data by sampling shared and modality-specific vectors from multivariate normal
distributions and processing them through shallow MLPs. While these approaches allow control over
mutual information between modalities, they do not involve training overparameterized networks
on real-world inputs and cannot reproduce optimization challenges observed in practice, such as the
faster convergence of the weaker modality [53]. Building on this, a combination of gradient modulation and loss regularization has also been explored
in [60, 56]. Another line of research focuses on updating modality-specific weights independently of
the weights used to extract features from other modalities [61, 18].Where the paper says it
Where the paper says it
Where the paper says it
Licenses and Assets
All datasets used are publicly available and properly cited Baseline implementations were obtained from the authors’ publicly released code repositories. The name of the license (e.g., CC-BY 4.0) should be included for each asset. Code to reproduce all experiments is provided as supplementary material. p. 2 These approaches
construct synthetic data by sampling shared and modality-specific vectors from multivariate normal
distributions and processing them through shallow MLPs. While these approaches allow control over
mutual information between modalities, they do not involve training overparameterized networks
on real-world inputs and cannot reproduce optimization challenges observed in practice, such as the
faster convergence of the weaker modality [53]. Regardless of the type of fusion, access to multiple modalities should improve classification. However,
research has demonstrated that naive late-fusion networks, which classify based on concatenated
features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal
classifier [53]. Regardless of the type of fusion, access to multiple modalities should improve classification. However,
research has demonstrated that naive late-fusion networks, which classify based on concatenated
features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal
classifier [53].Where the paper says it
Where the paper says it
Where the paper says it
Experimental Setting Details
Full training details including optimizers, learning rate schedules, batch sizes, augmentation strategies, and data preprocessing are provided in Appendix C. The synthetic dataset construction is described in Section 4. p. 2 In this
section, we demonstrate that deep ensembles of unimodal
networks outperform late-fusion classifiers across a range
of modality imbalance levels. To illustrate this, we use the synthetic construction from
Sec. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy,
July 2019. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual
Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy,
July 2019. However,
research has demonstrated that naive late-fusion networks, which classify based on concatenated
features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal
classifier [53]. To address this unexpected outcome, recent studies have examined the problem from
both theoretical [20, 36, 62] and practical perspectives [43], seeking optimal methods for combining
unimodal features in multimodal classification. However,
research has demonstrated that naive late-fusion networks, which classify based on concatenated
features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal
classifier [53]. To address this unexpected outcome, recent studies have examined the problem from
both theoretical [20, 36, 62] and practical perspectives [43], seeking optimal methods for combining
unimodal features in multimodal classification. However,
research has demonstrated that naive late-fusion networks, which classify based on concatenated
features from unimodal networks, can yield lower accuracy compared to the best-performing unimodal
classifier [53]. To address this unexpected outcome, recent studies have examined the problem from
both theoretical [20, 36, 62] and practical perspectives [43], seeking optimal methods for combining
unimodal features in multimodal classification.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Statistical Significance
Results are reported as mean ± standard deviation across multiple runs (5 runs for CREMA-D, 10 ensembles for the ensemble-size-2 analysis). The factors of variability that the error bars are capturing should be clearly stated. p. 3 We find it crucial to average output logits rather
than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving
accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we
present reliability diagrams and Expected Calibration Error analysis in Appendix F. We find it crucial to average output logits rather
than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving
accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we
present reliability diagrams and Expected Calibration Error analysis in Appendix F. We find it crucial to average output logits rather
than probabilities when training and testing deep ensembles (see Appendix E). Beyond improving
accuracy, logit averaging also yields well-calibrated predictions without post-hoc calibration; we
present reliability diagrams and Expected Calibration Error analysis in Appendix F.Where the paper says it
Where the paper says it
Where the paper says it
Experimental Result Reproducibility
All training details, hyperparameters, and data splits are provided in Appendix C Baseline results were reproduced using the authors’ original code repositories. Modifications to baseline implementations are documented in Appendix C. p. 4 approaches (single networks and HoDEs) operate on a single modality per model. All baseline results
were reproduced using the authors’ original code repositories; details are in Appendix C.Where the paper says it
Experimental Setting/Details
Full training details including optimizers, learning rate schedules, batch sizes, augmentation strategies, and data preprocessing are provided in Appendix C The synthetic dataset construction is described in Section 4. Computational costs are reported in Appendix H. All experiments were conducted on H100 GPUs Wall-clock training times and peak RAM memory for all methods and datasets are reported in Appendix H. p. 3 A detailed comparison of training cost, memory usage, and parameter
counts is provided in Appendix H. 6 Results
We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8].Where the paper says it
Results are reported as mean ± standard deviation across multiple runs (5 runs for CREMA-D, 10 ensembles for the ensemble-size-2 analysis) The factors of variability that the error bars are capturing should be clearly stated. The method for calculating the error bars should be explained.
Full training details including optimizers, learning rate schedules, batch sizes, augmentation strategies, and data preprocessing are provided in Appendix C The synthetic dataset construction is described in Section 4. Computational costs are reported in Appendix H All experiments were conducted on H100 GPUs Wall-clock training times and peak RAM memory for all methods and datasets are reported in Appendix H. p. 3 A detailed comparison of training cost, memory usage, and parameter
counts is provided in Appendix H. 6 Results
We compare deep ensembles with existing late-fusion methods on synthetic bimodal datasets generated from CIFAR-100 and ImageNet-1K [8].Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Modality imbalance handling using deep ensembles allows for effective classification
without any explicit modality fusion
by leveraging unimodal models trained on available modalities, which can be superior to late-fusion networks when imbalance is pronounced. -
A heuristic is proposed for selecting the number of models per modality in an ensemble, predicting a transition from homogeneous ensembles at small sizes to heterogeneous ensembles as the ensemble scales up.
-
The system can estimate asymptotic test loss from unimodal model losses by fitting scaling laws, which complements the heuristic by suggesting that
unimodal validation losses are sufficient to estimate both the composition and the expected performance of heterogeneous deep ensembles without training any.
-
The AI system can be designed to select optimal modality allocation based on a formula derived from cross-entropy validation loss ratios: ni = round Nens / (1/l P i j 1/lj).
Abstract
In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance when trained from random initialization. This advantage also holds over intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal classifiers. We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable. As the ensemble scales up, incorporating models from the weaker modality becomes beneficial. Both predictions align with our empirical findings. However, the heuristic becomes less reliable when the diversity of ensemble members is low. To systematically explore the challenges of optimizing multimodal models, we propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength. Our findings are consistent across synthetic and real-world datasets. Finally, by fitting scaling laws to bimodal datasets, we estimate the asymptotic performance of ensembles.
Sources
- Deep Ensembles: A Loss Landscape Perspective
- The Kinetics Human Action Video Dataset
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning