Clinical value claims in multimodal AI outpace supporting evidence

The gist The analysis audited 177 studies comparing multimodal with unimodal models and found that while multimodal models usually outperformed their unimodal comparators, claims of clinical value often outpace supporting evidence.

How it works

The study audited 177 studies comparing multimodal with unimodal models, including 81 from a systematic review and 96 from a probability sample of PubMed records The analysis focused on whether the added data source is worth acquiring for clinicians or health systems Multimodal prediction models ask more of a care pathway because someone has to order the extra scan or assay, and the result has to arrive before the decision the model is meant to inform

Key Findings on Model Performance

Multimodal models usually outperformed their unimodal comparators

The analysis showed that the multimodal model outperformed its strongest eligible unimodal comparator in 85.2% of legacy studies and an estimated 89.3% of update studies The pooled figure across both periods was 88.6% The median gain in AUROC was 0.042 (IQR, 0.020–0.100) for the pooled analysis However, gains were smaller when every data source was benchmarked, with a coefficient of -−-− -−-3.7 (95% CI, −0.056 to −0.005) for complete benchmarking

Evaluation of Evidence and Utility

A recent systematic review of 97 studies found that multimodal models outperformed unimodal ones in 91% of cases

The study examined various evidence items, such as calibration (C5), decision utility (C6), and external validation (C4) Reporting of these items showed no detectable change between the legacy and update periods Specifically, the share of studies reporting decision utility moved from 9.9% of legacy studies to an estimated 24.1% of update studies Furthermore, no eligible study evaluated a model prospectively

Claims of Clinical Value

Most claimed potential clinical usefulness (80.2% and an estimated 74.0%)

Claims of clinical value were close to universal, regardless of the evidence reported Most studies claimed potential clinical usefulness or improved care, with 83.6% of legacy studies and an estimated 85.9% of update studies still claiming potential clinical usefulness or improved care Explicit claims of improved care were more frequent in studies that had run a decision-curve analysis (an estimated 32.2% versus 6.5%) or had validated the gain externally (24.0% versus 8.7%)

Discussion and Limitations

The literature shows that adding a data source improves discrimination, but tests of the gain on external data or at a decision threshold are uncommon For a clinician deciding whether to order the additional scan or assay, this literature shows that the added data carry information and says little about whether acquiring them improves decisions The paper concludes that two changes would make these studies more useful to clinicians: a separately fitted model for every constituent data source, reported on the same test set, and prospective evaluation of clinical decisions before an added modality becomes part of routine care

Extended Data Insights

Had the strongest source lacked a model of its own, the gain against the best remaining source would have been larger by a design-weighted mean of 0.062

In completely benchmarked studies, leaving out the strongest source's model would have more than doubled the average reported gain The median regression analysis showed that the increase in reported gain was 0.127 when the comparator's AUROC was 0.70 or lower The analysis also quantified the incremental burden or cost of acquiring the added source, which ranged from 1.2% to an estimated 2.6%

Coverage Audit

The coverage audit of a second database suggested that adding more studies would not significantly change the prevalence estimates for most evidence items, with the largest change being for uncertainty of the gain (41.2% to 36.7%) The coefficient for complete benchmarking changed from-0.030 in the primary analysis to-0.041 when stratum E was added

Final Conclusion

The paper concludes that a gain in discrimination is the opening step of the case for an added test, but later steps concern decisions, and no prospective evaluation was found in either period The evidence suggests that two changes would make these studies more useful to clinicians: a separately fitted model for every constituent data source, reported on the same test set, and prospective evaluation of clinical decisions before an added modality becomes part of routine care

References

Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J., Multimodal biomedical AI Kline, A. et al., Multimodal machine learning in precision health: a scoping review Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P., Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines Moons, K. G., de Groot, J. A., Linnet, K., Reitsma, J. B. & Bossuyt, P. M., Quantifying the added value of a diagnostic test or marker Steyerberg, E. W. et al., Assessing the performance of prediction models: a framework for traditional and novel measures Pencina, M. J., D’Agostino, R. B. Sr & Demler, O. V., Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models Kerr, K. F., Bansal, A. & Pepe, M. S., Further insight into the incremental value of new markers: the interpretation of performance measures and the importance of clinical context Cook, N. R., Use and misuse of the receiver operating characteristic in risk prediction Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L. & Steyerberg, E. W., Calibration: the Achilles heel of predictive analytics Vickers, A. J. & Elkin, E. B., Decision curve analysis: a novel method for evaluating prediction models Vickers, A. J., Van Calster, B. & Steyerberg, E. W., Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests Vasey, B. et al., Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI Ferrante di Ruffano, L., Hyde, C. J., McCaffery, K. J., Bossuyt, P. M. & Deeks, J. J., Assessing the value of diagnostic tests: a framework for designing and evaluating trials Benani, A. et al., Is multimodal better? A systematic review of multimodal versus unimodal machine learning in clinical decision-making Liu, X. et al.

1. textbf Stronger Incremental Value Assessment for Added Modalities (C8): The AI system can quantify incremental burden or cost of acquiring the added source by explicitly measuring this metric to inform clinical decision-making, as reported in Table S7, which shows a quantified increase from 2.6 to 3.2 for acquisition burden when moving from legacy to update data.

2. textbf Robust Discrimination Benchmarking: The system should move beyond simply comparing multimodal models against a single unimodal comparator by implementing complete benchmarking, where every constituent modality has its own separately fitted unimodal model on the same evaluation set, as defined in Criterion C2, to avoid underestimating gains, since leaving out the strongest source's model would have raised the average gain from 0.046 to 0.108.

3. textbf Context-Aware Decision Utility and Calibration: The AI should incorporate decision utility (C6) and calibration (C5) into its assessment, as these factors are crucial for translating discrimination into clinical consequence, especially since Decision-curve analysis became more common between periods but often occurred in studies that had not benchmarked every source.

4. textbf Uncertainty Quantification for Clinical Use: When a multimodal model is deployed, the system must report uncertainty estimates ("C3) and external validation of the gain (C4), as these are essential for determining whether the added data carry information and says little about whether acquiring them improves decisions."