Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability

arXiv:2508.16000 · eess.IV, cs.CV, cs.LG · Submitted 2025-08-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis".

Dev: Here's the gist of the paper. A precise assessment of the risk of breast lesions can greatly lower it and assist physicians in choosing the best course of action The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM) Pruning redundant reasoning tokens The study examines a number of multimodal deep networks grounded on feature concatenation, co-attention, and crossattention to integrate mammography and categorical clinical characteristics The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM) How the training signal is built The embedding of mammograms is achieved by employing a ResNet-50 backbone pretrained on ImageNet and fine-tuned on the CBIS-DDSM dataset The output of the ResNet-50 global average pooling (GAP) layer yields a de-dimensional embedding: e i = ResNet50 (x i(m)), e i ∈ Rde, de = 2048 To obtain a compact latent representation, the authors project e i into a d-dimensional space using an affine transformation followed by a nonlinear activation: ̃i = F(WeT∙ ei + be), We ∈ Rde×d, be ∈ Rd, (2) Structured clinical features are encoded as one-hot vectors with dimensions corresponding to their categorical granularity The total of the dimensions of five feature vectors will yield 36 dimensions for our clinical embedding Where the gains come from The authors evaluate the model in combination with either co-attention or cross-attention to address missing clinical feature data, which frequently occurs in real-world settings Co-attention is a symmetric attention mechanism where both modalities attend to each other simultaneously, producing mutually informed representations Cross-attention is asymmetric, allowing the image features to be selectively enhanced based on only the most relevant clinical features The proposed cross-attention fusion achieved the highest performance across all metrics, with an AUC-ROC of 0.980, an F1-score of 0.940, an accuracy of 0.960, a precision of 0.920, and a recall of 0.95 Pruning redundant reasoning tokens Explainable artificial intelligence (XAI) is defined as "How well a system can describe the actions of AI-powered objects in a simulation game" The paper proposes an explainable multimodal model which helps improve the pathology classification performance by leveraging multimodal data and provide interpretability for the model diagnosis using the accessible labels of clinical features that correspond with their agreeing mammogram in the CBIS-DDSM catalogue Where the gains come from The authors apply SHAP, LIME, and Grad-CAM to interpret predictions at both the image and feature levels to enhance transparency and support clinical decision-making Grad-CAM highlights regions of the mammogram most influential for a given prediction SHAP assigns each input feature a Shapley value from cooperative game theory, representing its average contribution to the prediction across all possible feature coalitions LIME explains individual predictions by training a simple, interpretable surrogate model locally around the prediction instance Where the gains come from The comparative analysis showed that incorporating both clinical and imaging data through a crossattention fusion strategy substantially enhanced classification performance across all metrics The proposed model achieved an AUC-ROC of 0.980, an F1-score of 0.940, an accuracy of 0.960, a precision of 0.920, and a recall of 0.95 The combination provided both global visualization of salient regions and local feature-level attribution Pruning redundant reasoning tokens The authors performed case-level analyses to better understand the strengths and limitations of our proposed multimodal breast cancer detection model using explainable AI (XAI) techniques In malignant cases, Grad-CAM highlighted high-intensity areas concentrated around spiculated and irregular masses findings that are consistent with established radiological markers of malignancy Where the gains come from The results demonstrated that our model’s correct predictions are often supported by strong alignment between XAI explanations and clinical reasoning Errors can be traced to specific misinterpretations, highlighting areas for future model refinement The convergence between LIME and Grad-CAM results validated the internal consistency of our model’s decision-making process Pruning redundant reasoning tokens Future work will focus on optimizing fusion mechanisms and exploring automated extraction of clinical features directly from imaging data as well as investigate the application of reinforcement learning and continual learning paradigms The paper is 2508.16000.

Paper summary: Rosa: To recap, this paper, "Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability," looks at how to combine mammography images with clinical data using cross-attention to improve pathology classification performance while also providing interpretability.

Dev: They propose a model that leverages multimodal data and provides interpretability for the diagnosis using the accessible labels of clinical features that correspond with their agreeing mammogram in the CBIS-DDSM catalogue <ref:2508.16000#pg2>.

Taro: The core idea is addressing missing clinical feature data, which happens a lot in real-world settings, by evaluating both co-attention and cross-attention strategies for fusion <ref:2508.16000#pg3>.

Rosa: They mention that the embedding of mammograms comes from a ResNet-fifty backbone pretrained on ImageNet and fine-tuned on the CBIS-DDSM dataset, which yields a de-dimensional embedding with dimension two thousand forty-eight <ref:2508.16000#pg3>.

Dev: They then project this embedding into a d-dimensional space using an affine transformation followed by a nonlinear activation to get the final representation, and they encode structured clinical features as one-hot vectors, resulting in thirty-six dimensions for the clinical embedding <ref:2508.16000#pg3>.

Taro: The paper shows that their proposed cross-attention fusion achieved the highest performance across all metrics, with an AUC-ROC of zero point nine eight zero, an F1-score of zero point nine four zero, an accuracy of zero point nine six zero, a precision of zero point nine two zero, and a recall of zero point nine five <ref:2508.1600#pg3>.

Dev: So they’re not just building a classifier; they’re building one that offers some kind of transparency into its decision-making process, which is key for clinical use <ref:2508.16000#pg2>.

Conclusion: Rosa: Now we're wrapping up this look at "Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability." The authors are essentially saying they’ve put together a model that uses cross-attention to merge the image data and the clinical facts.

Dev: That fusion is what makes it work, and they're pointing out that explainability methods like Grad-CAM highlight regions of the mammogram most influential for a given prediction <ref:2508.16000#pg2>.

Taro: And they also use SHAP to assign each input feature a Shapley value from cooperative game theory, representing its average contribution to the prediction across all possible feature coalitions <ref:2508.16000#pg2>.

Rosa: The comparative analysis showed that incorporating both clinical and imaging data through a cross-attention fusion strategy substantially enhanced classification performance across all metrics, hitting an AUC-ROC of zero point nine eight zero, an F1-score of zero point nine four zero, an accuracy of zero point nine six zero, a precision of zero point nine two zero, and a recall of zero point nine five <ref:2538#pg1>.

Dev: That combination provided both global visualization of salient regions and local feature-level attribution <ref:2538#pg1>.

Rosa: The case-level analyses showed that in malignant cases, Grad-CAM highlighted high-intensity areas concentrated around spiculated and irregular masses findings that are consistent with established radiological markers of malignancy <ref:2538#pg1>.

Taro: The results demonstrated that our model’s correct predictions are often supported by strong alignment between XAI explanations and clinical reasoning <ref:2538#pg1>.

Dev: They also found that the convergence between LIME and Grad-CAM results validated the internal consistency of their model’s decision-making process <ref:2538#pg1>.

Rosa: So, they’ve shown how this multimodal approach gets really high scores across all their tests, and the implication is that this method could help lower the risk assessment for breast lesions by giving physicians a more robust tool to choose the best next steps.

Dev: Right. It moves beyond just building a classifier; it builds one that offers some kind of transparency into its decision-making process <ref:2538#pg1>.

University for Development Studies

eess.IV, cs.CV, cs.LG

Submitted: 2025-08-21

Updated: 2025-08-21

Comments: 11 pages, 9 figures

Journal ref: International Journal of Computer Applications. 187, 147 (October 2026), 1-10

DOI: 10.5120/ijcacc129d4311fa

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: A precise assessment of the risk of breast lesions can greatly lower it and assist physicians in choosing the best course of action The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score

Key concepts

Cross-Attention Fusion
This is a method where the model allows both the mammogram data and clinical features to 'attend' to each other simultaneously. It creates mutually informed representations by selectively focusing on the most relevant parts of one modality based on information from the other, which boosts overall diagnostic accuracy.
ResNet-50 Backbone
This is a deep neural network structure pre-trained on a massive image dataset (ImageNet) and then fine-tuned specifically for breast cancer images. It acts as the primary tool for extracting rich visual features from the raw mammogram data before they are combined with clinical information.
Grad-CAM
Grad-CAM is an Explainable AI technique that generates a heatmap overlaid on the original image. This heatmap shows exactly which regions in the mammogram most influenced the model's final prediction, helping clinicians visually verify if the model is focusing on suspicious areas like irregular masses.

Terminology

Summary

A precise assessment of the risk of breast lesions can greatly lower it and assist physicians in choosing the best course of action The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM)

Pruning redundant reasoning tokens

The study examines a number of multimodal deep networks grounded on feature concatenation, co-attention, and crossattention to integrate mammography and categorical clinical characteristics The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM)

How the training signal is built

The embedding of mammograms is achieved by employing a ResNet-50 backbone pretrained on ImageNet and fine-tuned on the CBIS-DDSM dataset The output of the ResNet-50 global average pooling (GAP) layer yields a de-dimensional embedding: e i = ResNet50 (x i(m)), e i ∈ Rde, de = 2048 To obtain a compact latent representation, the authors project e i into a d-dimensional space using an affine transformation followed by a nonlinear activation: ̃i = F(WeT∙ ei + be), We ∈ Rde×d, be ∈ Rd, (2) Structured clinical features are encoded as one-hot vectors with dimensions corresponding to their categorical granularity The total of the dimensions of five feature vectors will yield 36 dimensions for our clinical embedding

Where the gains come from

The authors evaluate the model in combination with either co-attention or cross-attention to address missing clinical feature data, which frequently occurs in real-world settings Co-attention is a symmetric attention mechanism where both modalities attend to each other simultaneously, producing mutually informed representations Cross-attention is asymmetric, allowing the image features to be selectively enhanced based on only the most relevant clinical features The proposed cross-attention fusion achieved the highest performance across all metrics, with an AUC-ROC of 0.980, an F1-score of 0.940, an accuracy of 0.960, a precision of 0.920, and a recall of 0.95

Explainable artificial intelligence (XAI) is defined as How well a system can describe the actions of AI-powered objects in a simulation game The paper proposes an explainable multimodal model which helps improve the pathology classification performance by leveraging multimodal data and provide interpretability for the model diagnosis using the accessible labels of clinical features that correspond with their agreeing mammogram in the CBIS-DDSM catalogue

The authors apply SHAP, LIME, and Grad-CAM to interpret predictions at both the image and feature levels to enhance transparency and support clinical decision-making Grad-CAM highlights regions of the mammogram most influential for a given prediction SHAP assigns each input feature a Shapley value from cooperative game theory, representing its average contribution to the prediction across all possible feature coalitions LIME explains individual predictions by training a simple, interpretable surrogate model locally around the prediction instance

The comparative analysis showed that incorporating both clinical and imaging data through a crossattention fusion strategy substantially enhanced classification performance across all metrics The proposed model achieved an AUC-ROC of 0.980, an F1-score of 0.940, an accuracy of 0.960, a precision of 0.920, and a recall of 0.95 The combination provided both global visualization of salient regions and local feature-level attribution

The authors performed case-level analyses to better understand the strengths and limitations of our proposed multimodal breast cancer detection model using explainable AI (XAI) techniques In malignant cases, Grad-CAM highlighted high-intensity areas concentrated around spiculated and irregular masses findings that are consistent with established radiological markers of malignancy

The results demonstrated that our model’s correct predictions are often supported by strong alignment between XAI explanations and clinical reasoning Errors can be traced to specific misinterpretations, highlighting areas for future model refinement The convergence between LIME and Grad-CAM results validated the internal consistency of our model’s decision-making process

Future work will focus on optimizing fusion mechanisms and exploring automated extraction of clinical features directly from imaging data as well as investigate the application of reinforcement learning and continual learning paradigms The paper is 2508.16000.

Improvements for AI systems

  1. Cross-modal fusion for breast cancer diagnosis will achieve an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 on publicly accessible datasets (TCGA and CBIS-DDSM). This provides a precise assessment of the risk of breast lesions to lower it and assist physicians in choosing the best course of action.

  2. Adaptive fusion mechanisms using co-attention or cross-attention will allow the model to intelligently adapt to clinical scenarios where clinical characteristics are occasionally absent or insufficient in real-world settings. This enables the model to be readily integrated into any clinical workflow by addressing missing value issues.

  3. The system will provide interpretability for diagnosis using SHAP, LIME, and Grad-CAM techniques applied at both the image and feature levels. This allows clinicians to validate the model’s reasoning by highlighting critical imaging regions and influential clinical attributes.

  4. The improved AI system can perform targeted conditioning, where the image features are selectively enhanced based on only the most relevant clinical features, without reciprocally altering the clinical representations. This is particularly effective when one modality is consistently more informative than the other.

  5. The model will utilize Grad-CAM to generate saliency heatmaps over mammography inputs that visually identify regions most influential for a prediction, such as spiculation and irregular masses findings. This confirms that the model’s focus aligns with medically relevant evidence.

  6. SHAP analysis will quantify the contribution of individual clinical and imaging features to predictions, offering a global interpretability perspective. Positive SHAP values will correspond to features pushing the decision toward malignancy, such as mass shape and breast density.

  7. LIME will produce case-specific explanations by training a simple surrogate model locally around a prediction instance. This provides an intuitive linear approximation of the black-box decision boundary in the local neighborhood of x.

Abstract

A precise assessment of the risk of breast lesions can greatly lower it and assist physicians in choosing the best course of action. To categorise breast lesions, the majority of current computer-aided systems only use characteristics from mammograms. Although this method is practical, it does not completely utilise clinical reports' valuable information to attain the best results. When compared to utilising mammography alone, will clinical features greatly enhance the categorisation of breast lesions? How may clinical features and mammograms be combined most effectively? In what ways may explainable AI approaches improve the interpretability and reliability of models used to diagnose breast cancer? To answer these basic problems, a comprehensive investigation is desperately needed. In order to integrate mammography and categorical clinical characteristics, this study examines a number of multimodal deep networks grounded on feature concatenation, co-attention, and cross-attention. The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM).

Sources

Related papers