Multimodal Deep Learning for Uncertainty-Aware Radiation Pneumonitis Risk Prediction
summary
The gist
Radiation pneumonitis (RP) is a common and clinically significant toxicity of thoracic radiation therapy that can cause pulmonary morbidity and impair quality of life.
In short
MM-DiT is a Bayesian framework that combines planning CT scans and 3D radiation dose distributions to predict radiation pneumonitis risk. It goes beyond simple predictions by explicitly quantifying three types of uncertainty—aleatoric, epistemic, and label uncertainty. This allows clinicians to stratify patients into risk-uncertainty groups for more informed clinical decision-making.
Key concepts
- Multimodal Bayesian Diffusion Transformer (MM-DiT)
- This is the core framework that integrates planning CT images and 3D dose distributions. It uses a diffusion transformer structure to learn complex, cross-modal representations from these different data types without needing prior toxicity labels. This allows the model to understand how anatomy and dose interact.
- Aleatoric Uncertainty (Ualea)
- This uncertainty reflects the inherent randomness or stochasticity present in the input data itself. It captures how noisy or variable the patient's characteristics are, which directly impacts the prediction's variance. High aleatoric uncertainty means the model is inherently less certain about that specific prediction.
- Epistemic Uncertainty (Uepst)
- This uncertainty measures limitations in the model itself, stemming from insufficient or unrepresentative training data. It indicates that a patient's characteristics fall outside what the model has seen before. High epistemic uncertainty suggests the model is unsure because it hasn't encountered similar cases.
- Label Uncertainty (Ulabel)
- This component explicitly models inconsistencies and subjectivity in the toxicity annotations used for training. It quantifies how much the true label might fluctuate based on the predicted risk, providing a measure of confidence in the ground truth data.
Terminology used across episodes
This episode discusses
The paper
Multimodal Deep Learning for Uncertainty-Aware Radiation Pneumonitis Risk Prediction · Read on arXiv
Jin Yang, Tian Liu, Jing Wang, Robert Samstein, Kenneth Rosenzweig, Julie Bloom
Department of Radiation Oncology, Icahn School of Medicine at Mount Sinai
Radiation pneumonitis (RP) is a potentially serious complication of thoracic radiotherapy, making reliable risk prediction important for identifying high-risk patients and guiding effective toxicity management. Although deep learning models can predict RP risk, uncertainty quantification remains underdeveloped in radiation toxicity assessment. We developed a Multimodal Bayesian Diffusion Transformer (MM-DiT) that integrates pretreatment computed tomography (CT) and three-dimensional planning dose distributions for RP risk prediction and uncertainty quantification. MM-DiT combines self-supervised multimodal representation learning with Bayesian classification to estimate individual risk and characterize aleatoric, epistemic, and label uncertainty. In independent Mount Sinai Health System (MSHS) and RTOG cohorts, MM-DiT achieved areas under the receiver operating characteristic curve (AUCs) of 0.89 and 0.86, respectively, with expected calibration errors of 0.02 and 0.03. Incorporating post-treatment CT increased AUC to 0.95 in MSHS and reduced aleatoric and epistemic uncertainty by approximately 40-45% and 40%, respectively. These findings establish a framework for uncertainty-aware radiation toxicity prediction that complements risk estimates with information about predictive reliability. By integrating multimodal learning and uncertainty quantification, MM-DiT may advance more interpretable and trustworthy AI-assisted toxicity assessment and support individualized radiotherapy planning.
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Multimodal Deep Learning for Uncertainty-Aware Radiation Pneumonitis Risk Prediction".
Dev: Here's the gist of the paper. Radiation pneumonitis (RP) is a common and clinically significant toxicity of thoracic radiation therapy that can cause pulmonary morbidity and impair quality of life.
Rosa: MM-DiT provides individualized RP risk estimates together with complementary measures of aleatoric, epistemic, and label uncertainty.
Dev: Problem Statement Conventional dose-volume histogram–based metrics and normal tissue complication probability models are widely used for RP risk assessment but inadequately capture the complex spatial, anatomical, and patient-specific factors underlying radiation-induced lung injury.
Rosa: These traditional approaches compress complex three-dimensional (3D) dose distributions into one-dimensional summary statistics, thereby discarding spatial information regarding regional dose deposition and tissue heterogeneity.
Paper summary: Rosa: So, to get into what this paper is really saying, they start by pointing out that conventional dose-volume histogram based metrics and normal tissue complication probability models just can't handle the complexity of radiation lung injury >
Dev: They argue these traditional models struggle because they compress everything into one dimension when there's so much spatial information about the dose distribution that gets lost >
Rosa: The thesis here is that by using a Multimodal Bayesian Diffusion Transformer, they can jointly estimate RP risk and characterize the sources of predictive uncertainty >
Taro: Why does characterizing uncertainty matter for us? Because right now, most AI methods just give you a single number—a point estimate—without telling you how much to trust it >
Dev: The paper claims that this approach offers a more reliable and transparent way to do AI assisted RP risk assessment by providing that explicit modeling of label noise >
Rosa: They’re trying to support more informed individualized treatment planning and safer clinical decision making in thoracic radiation therapy using this framework >
Taro: It's about giving clinicians a tool where they can stratify patients into risk uncertainty groups, like high risk plus high confidence, which guides immediate intervention decisions >
Dev: The paper is essentially showing how integrating heterogeneous sources of information—patient demographics, clinical characteristics, dose distributions, and radiomic features from imaging—can capture the complex interactions contributing to RP development >
Rosa: It’s about moving away from just a deterministic prediction to something that acknowledges the reliability of every individual prediction >
Taro: That's what makes it relevant because in real-world clinical settings, you can't afford to make a decision without knowing the risk and how certain you are about that risk >
Dev: So, this framework is designed to be more reliable and transparent for those who need to use AI in their treatment planning process >
Rosa: The authors of this paper are Jin Yang, Tian Liu, Jing Wang, Robert Samstein, Kenneth Rosenzweig, Julie Bloom and Ming Chao at the Department of Radiation Oncology at Icahn School of Medicine at Mount Sinai in New York City >
Taro: It’s interesting that they’re focusing on lung cancer and other thoracic malignancies as the primary context for this work because RP is such a significant constraint on safe dose delivery >
Dev: The paper also notes that the clinical translation of AI methods remains limited by challenges like generalizability, calibration, interpretation, and reliable assessment of individual patient risk >
Rosa: So, even with improved performance numbers like AUC values reaching eighty-five point five nine percent in the RTOG six hundred seventeen cohort and eighty-nine point one one percent in the MSHS cohort >
Taro: Those performance numbers are good, but they're just one piece of a puzzle when you consider how robust the framework handles those uncertainties when the world misbehaves >
Dev: It shows that integrating multimodal information can improve predictive performance over models that only look at one modality, like just dose distribution or just imaging features >
Rosa: So, this paper is essentially proposing a way to get better predictions by making sure we also understand the uncertainty around those predictions >
Conclusion: Taro: To wrap this up, the Multimodal Deep Learning for Uncertainty-Aware Radiation Pneumonitis Risk Prediction framework is a Bayesian multimodal approach that integrates planning CT imaging and three dee radiation dose distributions for individualized RP risk prediction and explicit uncertainty quantification > <ref:2608.28937#pg2>
Dev: It's about providing a more transparent and reliable approach to AI assisted RP risk assessment by giving us that explicit modeling of label noise >
Rosa: It moves us toward more informed individualized treatment planning because it allows clinicians to see not just the predicted risk but also how certain the model is in its own assessment >
Taro: This framework extends conventional risk stratification toward uncertainty-aware clinical interpretation, allowing you to categorize patients into groups like high risk plus high confidence for immediate intervention considerations >
Dev: It’s a tool that helps identify predictions that are sufficiently supported to inform clinical consideration without necessarily prescribing specific treatment modifications based solely on the model output >
Taro: It’s a framework that gives us a mechanism for identifying predictions that may be sufficiently supported to inform clinical consideration >
Dev: Ultimately, it's about providing a more transparent and reliable approach to AI assisted RP risk assessment >
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G