ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation
summary
The gist
ScaleFusionNet is a hybrid model that integrates a Cross-Attention Transformer Module (CATM) and an adaptive fusion block (AFB) to enhance feature extraction and fusion by capturing both local and…
In short
ScaleFusionNet is a hybrid model for skin lesion segmentation that uses a Cross-Attention Transformer Module (CATM) and an Adaptive Fusion Block (AFB). It combines Swin Transformers for multi-scale feature extraction and cross-attention to refine features at skip connections. This approach significantly improves lesion boundary accuracy, achieving high Dice scores on standard datasets.
Key concepts
- Swin Transformer Blocks
- These are transformer blocks used in the encoder to extract features at different scales. They allow the model to capture both fine details and broader contextual information from the skin lesion images simultaneously, which is crucial for understanding irregular shapes.
- Cross-Attention Fusion (CAF)
- This mechanism is used within the CATM to adaptively align hierarchical features between different parts of the network. It helps precisely match features from different resolutions, reducing semantic gaps and improving how the model combines local and global information.
- Deformable Convolution
- Used in the Adaptive Fusion Block (AFB), this technique allows the convolution operation to adapt its shape based on the input image. This enables the block to focus its feature extraction precisely on target areas, which is key for accurately refining lesion boundaries and preserving fine details.
Terminology used across episodes
This episode discusses
- ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation · Paper Radio
- U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation
- SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation
The paper
ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation · Read on arXiv
Saqib Qamar, Syed Furqan Qadri, Roobaea Alroobaea, Goram Mufarah M Alshmrani, Richard Jiang
Division of Robotics, Perception and Learning at KTH Royal Institute of Technology · Faculty of Computing and Information Technology at Sohar University · BGI Research in Hangzhou, China · Department of Computer Science at College of Computers and Information Technology at Taif University · School of Computing and Commutations at Lancaster University
Melanoma is a malignant tumor that originates from skin cell lesions. Accurate and efficient segmentation of skin lesions is essential for quantitative analysis but remains a challenge due to blurred lesion boundaries, gradual color changes, and irregular shapes. To address this, we propose ScaleFusionNet, a hybrid model that integrates a Cross-Attention Transformer Module (CATM) and adaptive fusion block (AFB) to enhance feature extraction and fusion by capturing both local and global features. We introduce CATM, which utilizes Swin transformer blocks and Cross Attention Fusion (CAF) to adaptively refine feature fusion and reduce semantic gaps in the encoder-decoder to improve segmentation accuracy. Additionally, the AFB uses Swin Transformer-based attention and deformable convolution-based adaptive feature extraction to help the model gather local and global contextual information through parallel pathways. This enhancement refines the lesion boundaries and preserves fine-grained details. ScaleFusionNet achieves Dice scores of 92.94% and 91.80% on the ISIC-2016 and ISIC-2018 datasets, respectively, demonstrating its effectiveness in skin lesion analysis. Simultaneously, independent validation experiments were conducted on the PH squared dataset using the pretrained model weights. The results show that ScaleFusionNet demonstrates significant performance improvements compared with other state-of-the-art methods. Our code implementation is publicly available at GitHub.
DOI: 10.1038/s41598-025-17300-x
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation".
Dev: Here's the gist of the paper. ScaleFusionNet is a hybrid model that integrates a Cross-Attention Transformer Module (CATM) and an adaptive fusion block (AFB) to enhance feature extraction and fusion by capturing both local and global features The results show that ScaleFusionNet demonstrates significant performance improvements compared with other state-of-the-art methods Model Architecture Overview ScaleFusionNet employs a hierarchical Swin Transformer-based encoder, where patch embedding and Swin Transformer blocks extract the multi-scale features The decoder utilizes AFBs, which combine Swin Transformer and deformable convolution features to refine feature integration and improve lesion boundary preservation The encoder progressively extracts the multi-scale features at resolutions 64x64x96, 256x256x3, 32x32x192, and 8x8x768 Feature Fusion Mechanisms The CATM is introduced at skip connection points to refine encoder-decoder feature fusion It utilizes Swin transformer blocks and cross-attention fusion to adaptively align hierarchical features The AdaptiveFusionBlock further enhances multi-scale feature extraction and fusion by integrating deformable convolutions with Swin transformer-based attention This three-way fusion mechanism enhances the refinement of the lesion boundary while preserving fine-grained details through complementary feature representations Experimental Validation and Results ScaleFusionNet achieves Dice scores of 92.94% and 91.80% on the ISIC-2016 and ISIC-2018 datasets, respectively On the ISIC-2016 dataset, ScaleFusionNet achieved a DSC score of 92.94% and an IOU score of 87.35% Independent validation experiments on the PH2 dataset show that ScaleFusionNet demonstrates significant performance improvements compared with other state-of-the-art methods Ablation Study Findings The structural ablation study on the ISIC-2016 dataset showed that Method 0, using only the hybrid architecture, resulted in a DSC of 91.76% Method 1, incorporating the CATM, achieved a DSC of 92.54% Method 2, introducing only the AdaptiveFusionBlock on top of the hybrid architecture, achieved a DSC performance of 92.62% ScaleFusionNet, combining all three proposed improvement methods, achieved a DSC of 92.94% The results highlight the importance of the CATM and AFB for ScaleFusionNet’s performance ScaleFusionNet is an important advancement in skin lesion segmentation, as it uses adaptive multi-scale fusion and cross-attention mechanism to achieve accurate and strong results However, future work should address computational efficiency and incorporate uncertainty quantification to further enhance its clinical applicability and reliability The model’s ability to preserve fine-grained details and refine the limits of the injury further underscores its potential to improve melanoma diagnosis and treatment in real-world clinical settings The authors also suggest that better feature allocation, like self-selective routing, could solve the issue of computational complexity The paper concludes by stating that ScaleFusionNet is a leading solution for skin lesion segmentation tasks The model’s ability to reduce misdiagnosis and improve lesion boundary delineation makes it a highly effective tool for clinical applications The authors also mention that deformable convolutions might worsen the issue, suggesting simpler methods should be explored in future work The paper is available on GitHub and the datasets are available in the 2016 and 2018 repositories The source code implementation can be found by clicking on this link The authors also express their appreciation to Taif University, Saudi Arabia, for supporting this work through project number (TU-DSPP-2024-229) The study did not involve animals or human participants The authors declare no competing interests All authors contributed to writing the manuscript and approved the final version The paper is published on arXiv with the identifier 2503.03327v4.
Rosa: The paper was submitted to arXiv on 2 Jul 2025.
Dev: The authors' email addresses are sqama@kth.se, r.jiang2@lancaster.ac.uk.
Rosa: The paper is indexed under the category eess-IV.
Paper summary: Rosa: So we're looking at this paper today, "ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation." Basically, they're tackling the problem of segmenting skin lesions in images. They say that getting accurate segmentation is really hard because of those blurred boundaries and the way colors change gradually and shapes are irregular <ref:2503.03327#pg1>.
Dev: Right, so they propose this hybrid model called ScaleFusionNet to fix that by integrating a Cross-Attention Transformer Module, or CATM, with an adaptive fusion block, or AFB. They claim this approach helps by grabbing both local and global features simultaneously <ref:2503.03327#pg1>.
Taro: I'm curious about the mechanism they mentioned. The CATM uses Swin transformer blocks and cross-attention fusion to adaptively refine how the features are fused, which they say helps reduce semantic gaps between the encoder and decoder <ref:2503.03327#pg3>.
Rosa: And then you have this AFB, which uses Swin Transformer-based attention combined with deformable convolution based adaptive feature extraction to gather local and global contextual information through parallel pathways <ref:2503.03327#pg1>. It sounds like they're trying to get better boundary refinement and keep those fine details intact <ref:2503.03327#pg1>.
Dev: I see the architecture described, too. They use a hierarchical Swin Transformer based encoder for multi-scale feature extraction, and the decoder uses these AFB blocks to combine that multi-scale information Page six. The encoder progressively extracts features at resolutions like sixty-four by sixty-four by ninety-six and then down to eight by eight by seven hundred sixty-eight Page seven.
Taro: So, when they talk about the CATM being at skip connection points, that means they are refining the feature fusion between the encoder and decoder using cross-attention mechanisms for precise alignment Page seven. How does that translate to actually seeing better boundaries on an image?
Rosa: Well, according to their findings on ISIC-two thousand sixteen and ISIC-two thousand eighteen datasets, ScaleFusionNet achieved Dice scores of ninety-two point nine four percent and ninety-one point eight zero percent, respectively Page two. They also did some independent validation experiments on the PH2 dataset using the pretrained model weights, showing significant performance improvements compared to other methods Page two <ref:2503.03327#pg1>.
Dev: That's a pretty strong set of numbers, especially coming from those ISIC datasets. But I want to know what they mean for us in terms of real-world deployment. They're focused on segmentation accuracy in skin lesion analysis Page two <ref:2503.03327#pg1>.
Paper summary: Taro: From my side, I wonder about the limitations they mentioned. They acknowledge that Vision Transformers can only output single-scale feature representations, which means a pure Transformer model struggles to capture multiscale information in two dimensional images Page three <ref:2503.03327#pg1>. That's a key constraint for the whole design.
Rosa: Exactly. And they also pointed out that even with U-Net skip connections, there can still be a semantic gap if there's a big difference between the encoder and decoder stages Page three <ref:2503.03327#pg1>. ScaleFusionNet tries to address that gap with their CATM and AFB mechanisms Page one <ref:2503.03327#pg1>.
Dev: So, the core idea is using these two components—the CATM for feature fusion refinement and the AFB for adaptive extraction—to handle those multi-scale issues better than previous Transformer U-shaped models Page three <ref:2503.03327#pg1>. It's about making sure the model pays more attention to the right parts of the image at different scales.
Taro: And I'm interested in how that adaptation plays out when things go wrong in a real scenario. The paper suggests that this focus on target areas becomes much stronger after going through the CATM Page eighteen <ref:2503.03327#pg3>. Does that mean it's more robust when the lesion is oddly shaped?
Rosa: It seems to be designed to handle those irregular shapes better by refining the boundaries and preserving fine-grained details through these three-way fusion mechanisms Page one <ref:2503.03327#pg1>. The ablation study supports this, showing that adding both the CATM and the AFB together leads to a Dice score of ninety-two point nine four percent, which is their best result Ablation Study Findings.
Dev: It's interesting how they broke it down. Method zero with just the hybrid architecture got a DSC of ninety-one point seven six percent, while adding the CATM boosted it to ninety-two point five four percent, and then adding only the AFB brought it up to ninety-two point six two percent Ablation Study Findings. It shows that both parts are necessary for that peak performance Ablation Study Findings.
Taro: So, if we think about this for autonomy, it means the system isn't just seeing a lesion once; it's using a sophisticated fusion strategy to understand the context across different scales to make a better decision Page eighteen <ref:2503.03327#pg3>. That level of contextual understanding is what I need when something unexpected happens.
Paper summary: Rosa: It definitely suggests that for practical applications, especially in clinical settings, this kind of feature refinement is what makes the difference in segmentation accuracy Page two <ref:2503.03327#pg1>. It moves beyond just finding a big object to precisely delineating the exact edge of a small lesion.
Dev: So, to wrap up on this paper "ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation," they've shown how combining cross-attention and adaptive fusion can effectively tackle the multiscale challenges in skin lesion segmentation Page one <ref:2503.03327#pg1,ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation>. They achieved Dice scores of ninety-two point nine four percent on ISIC-two thousand sixteen and ninety-one point eight zero percent on ISIC-two thousand eighteen with validation on PH2 showing improvements over other state-of-the-art methods Page two.
Taro: It's a solid result for segmentation accuracy, but we have to remember the limitation they mentioned: the reliance on the Swin Transformer structure means it's inherently constrained by how single-scale representations are generated Page three <ref:2503.03327#pg1>. They also suggested that exploring simpler methods for feature allocation might be needed later to handle computational complexity Conclusion.
Rosa: So, what this means for us is that when we build these systems, we need to think about both high accuracy and efficiency. This paper gives us a detailed blueprint for using transformer mechanisms not just for classification, but specifically for the complex task of fusing features across different scales in medical imaging Page one <ref:2503.03327#pg1>.
Dev: It's a good framework because it explicitly addresses those semantic gaps and provides parallel pathways for local and global information gathering through the AFB Page nine. For real-time loops, getting that feature fusion right is critical for maintaining a low latency and ensuring a reliable safety case Dev persona.
Taro: I think the implication is that future work needs to focus on uncertainty quantification. Knowing when the model isn't sure about a boundary is just as important as getting an accurate number Conclusion.
Rosa: Exactly. It moves us toward more reliable tools for diagnosis and treatment, which is where this kind of high-precision segmentation matters most in the long run Conclusion.
Conclusion: Rosa: So, we're wrapping up this look at ScaleFusionNet, titled "ScaleFusionNet: Transformer-Guided Multi-Scale Feature Fusion for Skin Lesion Segmentation." Basically, the authors are showing how they used a hybrid model to segment skin lesions better than just standard methods.
Dev: Yeah, it’s about using a Cross-Attention Transformer Module and an adaptive fusion block to grab both local and global features at the same time. That’s what I mean when I say they’re trying to handle those different scales of information.
Taro: From where I sit, it means they are getting better at figuring out the shape of the lesion because they aren't just looking at one scale, but trying to fuse things across different resolutions. That should help when a lesion is really irregular.
Rosa: Exactly. The authors achieved Dice scores of ninety-two point ninety-four percent on the ISIC-two thousand sixteen dataset and ninety-one point eighty percent on the ISIC-two thousand eighteen dataset, which shows they are performing quite well in practice.
Dev: Those numbers are solid, but we have to remember that this is based on pretraining and validation against other state-of-the-art methods. It’s not just some perfect score in a vacuum.
Taro: Right, and the ablation study showed that adding both the CATM and the AFB together brought them up to ninety-two point ninety-four percent, which proves both components are actually working together for that performance gain.
Rosa: The implication for me is that this approach is a strong tool for clinical applications because it focuses on refining those boundaries and preserving those fine details we talked about earlier.
Dev: It definitely moves beyond just finding a big object; it’s about precisely delineating the edge of the injury, which makes the segmentation much more reliable for diagnostic purposes.
Taro: I think what this suggests is that for real-world use, you need that kind of multi-scale context to handle messy inputs without failing when things get unexpected.
Rosa: So, while it's a strong model, we still have to keep an eye on computational efficiency and how well it handles those extreme failure modes when the input image is completely different from what it was trained on.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G