Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis".

Dev: Here's the gist of the paper. A precise assessment of the risk of breast lesions can greatly lower it and assist physicians in choosing the best course of action The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM) Pruning redundant reasoning tokens The study examines a number of multimodal deep networks grounded on feature concatenation, co-attention, and crossattention to integrate mammography and categorical clinical characteristics The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM) How the training signal is built The embedding of mammograms is achieved by employing a ResNet-50 backbone pretrained on ImageNet and fine-tuned on the CBIS-DDSM dataset The output of the ResNet-50 global average pooling (GAP) layer yields a de-dimensional embedding: e i = ResNet50 (x i(m)), e i ∈ Rde, de = 2048 To obtain a compact latent representation, the authors project e i into a d-dimensional space using an affine transformation followed by a nonlinear activation: ̃i = F(WeT∙ ei + be), We ∈ Rde×d, be ∈ Rd, (2) Structured clinical features are encoded as one-hot vectors with dimensions corresponding to their categorical granularity The total of the dimensions of five feature vectors will yield 36 dimensions for our clinical embedding Where the gains come from The authors evaluate the model in combination with either co-attention or cross-attention to address missing clinical feature data, which frequently occurs in real-world settings Co-attention is a symmetric attention mechanism where both modalities attend to each other simultaneously, producing mutually informed representations Cross-attention is asymmetric, allowing the image features to be selectively enhanced based on only the most relevant clinical features The proposed cross-attention fusion achieved the highest performance across all metrics, with an AUC-ROC of 0.980, an F1-score of 0.940, an accuracy of 0.960, a precision of 0.920, and a recall of 0.95 Pruning redundant reasoning tokens Explainable artificial intelligence (XAI) is defined as "How well a system can describe the actions of AI-powered objects in a simulation game" The paper proposes an explainable multimodal model which helps improve the pathology classification performance by leveraging multimodal data and provide interpretability for the model diagnosis using the accessible labels of clinical features that correspond with their agreeing mammogram in the CBIS-DDSM catalogue Where the gains come from The authors apply SHAP, LIME, and Grad-CAM to interpret predictions at both the image and feature levels to enhance transparency and support clinical decision-making Grad-CAM highlights regions of the mammogram most influential for a given prediction SHAP assigns each input feature a Shapley value from cooperative game theory, representing its average contribution to the prediction across all possible feature coalitions LIME explains individual predictions by training a simple, interpretable surrogate model locally around the prediction instance Where the gains come from The comparative analysis showed that incorporating both clinical and imaging data through a crossattention fusion strategy substantially enhanced classification performance across all metrics The proposed model achieved an AUC-ROC of 0.980, an F1-score of 0.940, an accuracy of 0.960, a precision of 0.920, and a recall of 0.95 The combination provided both global visualization of salient regions and local feature-level attribution Pruning redundant reasoning tokens The authors performed case-level analyses to better understand the strengths and limitations of our proposed multimodal breast cancer detection model using explainable AI (XAI) techniques In malignant cases, Grad-CAM highlighted high-intensity areas concentrated around spiculated and irregular masses findings that are consistent with established radiological markers of malignancy Where the gains come from The results demonstrated that our model’s correct predictions are often supported by strong alignment between XAI explanations and clinical reasoning Errors can be traced to specific misinterpretations, highlighting areas for future model refinement The convergence between LIME and Grad-CAM results validated the internal consistency of our model’s decision-making process Pruning redundant reasoning tokens Future work will focus on optimizing fusion mechanisms and exploring automated extraction of clinical features directly from imaging data as well as investigate the application of reinforcement learning and continual learning paradigms The paper is 2508.16000.

Paper summary: Rosa: To recap, this paper, "Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability," looks at how to combine mammography images with clinical data using cross-attention to improve pathology classification performance while also providing interpretability.

Dev: They propose a model that leverages multimodal data and provides interpretability for the diagnosis using the accessible labels of clinical features that correspond with their agreeing mammogram in the CBIS-DDSM catalogue <ref:2508.16000#pg2>.

Taro: The core idea is addressing missing clinical feature data, which happens a lot in real-world settings, by evaluating both co-attention and cross-attention strategies for fusion <ref:2508.16000#pg3>.

Rosa: They mention that the embedding of mammograms comes from a ResNet-fifty backbone pretrained on ImageNet and fine-tuned on the CBIS-DDSM dataset, which yields a de-dimensional embedding with dimension two thousand forty-eight <ref:2508.16000#pg3>.

Dev: They then project this embedding into a d-dimensional space using an affine transformation followed by a nonlinear activation to get the final representation, and they encode structured clinical features as one-hot vectors, resulting in thirty-six dimensions for the clinical embedding <ref:2508.16000#pg3>.

Taro: The paper shows that their proposed cross-attention fusion achieved the highest performance across all metrics, with an AUC-ROC of zero point nine eight zero, an F1-score of zero point nine four zero, an accuracy of zero point nine six zero, a precision of zero point nine two zero, and a recall of zero point nine five <ref:2508.1600#pg3>.

Dev: So they’re not just building a classifier; they’re building one that offers some kind of transparency into its decision-making process, which is key for clinical use <ref:2508.16000#pg2>.

Conclusion: Rosa: Now we're wrapping up this look at "Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability." The authors are essentially saying they’ve put together a model that uses cross-attention to merge the image data and the clinical facts.

Dev: That fusion is what makes it work, and they're pointing out that explainability methods like Grad-CAM highlight regions of the mammogram most influential for a given prediction <ref:2508.16000#pg2>.

Taro: And they also use SHAP to assign each input feature a Shapley value from cooperative game theory, representing its average contribution to the prediction across all possible feature coalitions <ref:2508.16000#pg2>.

Rosa: The comparative analysis showed that incorporating both clinical and imaging data through a cross-attention fusion strategy substantially enhanced classification performance across all metrics, hitting an AUC-ROC of zero point nine eight zero, an F1-score of zero point nine four zero, an accuracy of zero point nine six zero, a precision of zero point nine two zero, and a recall of zero point nine five <ref:2538#pg1>.

Dev: That combination provided both global visualization of salient regions and local feature-level attribution <ref:2538#pg1>.

Rosa: The case-level analyses showed that in malignant cases, Grad-CAM highlighted high-intensity areas concentrated around spiculated and irregular masses findings that are consistent with established radiological markers of malignancy <ref:2538#pg1>.

Taro: The results demonstrated that our model’s correct predictions are often supported by strong alignment between XAI explanations and clinical reasoning <ref:2538#pg1>.

Dev: They also found that the convergence between LIME and Grad-CAM results validated the internal consistency of their model’s decision-making process <ref:2538#pg1>.

Rosa: So, they’ve shown how this multimodal approach gets really high scores across all their tests, and the implication is that this method could help lower the risk assessment for breast lesions by giving physicians a more robust tool to choose the best next steps.

Dev: Right. It moves beyond just building a classifier; it builds one that offers some kind of transparency into its decision-making process <ref:2538#pg1>.