Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation

arXiv:2202.12295 · eess.IV, cs.CV, cs.LG · Submitted 2022-02-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation".

Paper summary: Rosa: To get into the specifics of this paper Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation they present a new way to do end-to-end deep model building using matrix factorization layers specifically for medical image segmentation >

Dev: The core thesis is that they built a differentiable matrix factorization layer using a block coordinate descent solver to efficiently model contextual information in the network >

Rosa: They also introduced this Shifted Window operation called SW Matricize and combined it with the NMF to fully utilize local contexts effectively >

Taro: That part about SW Matricize seems important for making sure they still capture those fine, local details even when you're factoring things out >

Dev: They show that the components from the matrix factorization are actually highly meaningful which gives a big advantage in terms of interpretability compared to standard CNNs and Transformers >

Rosa: And they found this distinctive interesting feature where you can speed up inference for a trained Factorizer model with no extra steps without losing much accuracy >

Dev: That’s the practical side of it, showing that their method isn't just theoretically better but also faster to run in practice >

Conclusion: Rosa: So looking at the title Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation and seeing these results from BraTS and ISLES’twenty-two we see they’ve proposed a scalable method based on NMF > <ref:2202.12295#pg3>

Dev: It boils down to using matrix factorization as an alternative attention mechanism that handles context modeling without the quadratic complexity issue that plagues traditional Transformer models >

Taro: What this means for us is that we can build these powerful segmentation models while keeping them interpretable because those NMF components give us a way to see what the model is actually focusing on >

Rosa: It suggests a path where we can combine the spatial efficiency of local context with a structured way to model global relationships without hitting those attention complexity walls >

Dev: The implication is that we might be able to deploy these models more easily in clinical settings because they are scalable and you have this clear understanding of how the context is being modeled >

Department of Electrical Engineering (ESAT), STADIUS Center, KU Leuven Department of Electrical Engineering (ESAT), icometrix Research and Development, Group Science, Engineering and Technology, KU Leuven Kulak Group Science, Engineering and Technology Department of Electrical Engineering (ESAT), PSI

eess.IV, cs.CV, cs.LG

Submitted: 2022-02-24

Updated: 2022-12-13

Comments: 31 pages, 8 figures, 4 tables

DOI: 10.1016/j.media.2022.102706

Code: https://github.com/pashtari/factorizer

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: Convolutional Neural Networks (CNNs) with U-shaped architectures have dominated medical image segmentation, which is crucial for various clinical purposes <ref:2202.12295#pg2> However…

Key concepts

U-shaped architecture
This is the overall structure of the model, similar to U-Net. It involves a main path with skip connections that allow information from earlier layers to be directly passed to later layers. This design helps the model effectively capture both local details and broader spatial context in medical images.
NMF-based modules
Instead of standard self-attention, Factorizer uses Non-negative Matrix Factorization (NMF) modules within its blocks. NMF is a mathematical technique used to decompose complex data matrices into simpler, interpretable factors, which helps the model learn meaningful patterns from image features.
Matricize Operations
These are specific techniques used to transform multi-dimensional image data (like 3D volumes) into a batch of matrices. They include Global Matricize for reshaping the entire volume, Local Matricize for creating small patches, and Shifted Window (SW) Matricize to ensure smooth feature maps at boundaries.

Terminology

Summary

Convolutional Neural Networks (CNNs) with U-shaped architectures have dominated medical image segmentation, which is crucial for various clinical purposes <ref:2202. p. 012295#pg2> However, the quadratic complexity of attention makes existing Transformer-based models use self-attention layers only after somehow reducing the image resolution, which limits the ability to capture global contexts present at higher resolutions This work introduces a family of models, dubbed Factorizer, which leverages the power of low-rank matrix factorization for constructing an end-to-end segmentation model Factorizers compete favorably with CNNs and Transformers in terms of accuracy, scalability, and interpretability, achieving state-of-the-art results on the BraTS dataset for brain tumor segmentation and ISLES’22 dataset for stroke lesion segmentation. p. 0

Introduction to the Problem

Medical image segmentation is an essential prerequisite for the analysis of anatomical structures for various clinical purposes, including diagnosis and treatment planning Currently, U-Net models mostly rely on convolution operations with small receptive fields, which are capable of exploiting only local context at each resolution Hence, they generally fail to effectively model long-range spatial dependencies, often necessary for example, for better recognition of brain lesions. Transformers have recently proven promising performance on vision tasks, including semantic segmentation, mainly due to their capability of modeling long-range dependencies. p. 0 Nevertheless, the quadratic complexity of attention makes existing Transformer-based models use self-attention layers only after somehow reducing the image resolution, which limits the ability to capture global contexts present at higher resolutions. p. 0

Factorizer Architecture

The Factorizer block is constructed by replacing the selfattention layer of a ViT block with our NMF-based modules and then integrated into a U-shaped architecture with skip connections. p. 1 A Factorizer block comprises NMF module and MLP, each of which comes after Layer Normalization and before a residual connection, that is, Y = WrappedNMF(LayerNorm(X)) + X, Z = MLP(LayerNorm(Y)) + Y. p. 6 The number of input and output channels are the same, but the number of inner channels is double that of input channels. p. 6

Matrix Factorization for Context Modeling

The attention mechanism can be viewed as a special case of nonparametric regression, and we propose a linearly scalable alternative to self-attention by formulating an NMF algorithm as a differentiable layer. The Factorizer block is constructed by replacing the selfattention layer of a ViT block with our NMF-based modules and then integrated into a U-shaped architecture with skip connections. The WrappedNMF can be computed as follows, X1 = PointwiseConv(X), X2 = Matricize(X1), X3 = NMF(ReLU(X2)), X4 = Dematricize(X3), WrappedNMF(X) = PointwiseConv(X4). p. 7

Matricization Operations

Before applying any matrix factorization method, an input batch of multi-dimensional images, denoted by X ∈ R B×C×H×W×D, must be turned into a batch of matrices, say Z ∈ R B0×M×N. This operation is called Matricize. p. 7 We propose three Matricize operations: i) Global Matricize, ii) Local Matricize, and iii) Shifted Window (SW) Matricize. p. 7 Global Matricize reshapes a batch of C-channel 3D images X ∈ R B×C×H×W×D into a batch of matrices denoted by a 3D tensor Z ∈ R B(C/E)×E×HW D, where E is called the head dimension. p. 7 Local Matricize splits an input X ∈ R B×C×H×W ×D into a grid of non-overlapping patches of size (E, P, P, P). p. 7 Shifted Window Matricize introduces a Shifted Window (SW) Matricize operation to make the output feature maps smoother around boundaries. p. 8

Results and Interpretability

Swin Factorizer is the clear overall winner in the brain tumor segmentation task with an average Dice score of 84.21% and HD95 of 6.89 mm. Highly meaningful NMF components give an additional interpretability advantage to Factorizers over CNNs and Transformers. p. 0 For instance, as observed in Figure 6b, the first (row 1) and second (row 2) components discriminate roughly WT while the third (row 3) and fourth (row 4) components capture TC. p. 2

Inference Speed-up

Our ablation studies reveal a distinctive interesting feature of Factorizers that enables us to easily speed up the inference for a trained Factorizer model with no extra steps and without sacrificing much accuracy. In practice, we use a rank-one approximation (R = 1) with T = 5 outer iterations of HALS, which is equivalent to MU for R = 1. p. 9 When it comes to model speed-up, this brings a great advantage to Factorizers over CNNs and Transformers, which require much more complex mechanisms to achieve their faster versions. p. 2

Conclusion and Future Work

Factorizer, with an average Dice score of 83.24% and an average HD95 of 9. p. 871 mm, marginally outperformed Performer. p. 1 One possible extension of this work is to customize the NMF objective to simultaneously exploit both local and global contexts. p. 1 Moreover, it would be useful to explore some ideas for automated selection of the rank hyperparameter. p. 3 We will explore the effectiveness of other NMF variants, such as Semi-NMF and Convex-NMF, which work on mixed-signed data matrices and relax the need for the ReLU activation function before factorization. p. 5 Finally, while this paper focuses on the segmentation of 3D medical images, Factorizer may also potentially serve as an effective approach for efficiently processing high-resolution 2D medical or natural images. p. 5

Ablation Summary

When we removed NMF subblocks from a Factorizer model, the performance substantially dropped while Factorizer models without MLP subblocks demonstrated less deterioration in results. Adding positional embedding to the bridge of a network improved the performance, particularly for Global Factorizer. p. 5 We observed that most of the performance is achieved via the encoder, which includes the first five blocks. p. 3 The NMF layer of the bridge block (layer 5) makes the greatest contribution to the performance. p. 8 Removing an NMF layer, especially those at higher-resolution stages of the network, can significantly reduce the computational complexity and speed up the inference time.

Data Augmentation Pipeline

The MONAI library [43] was used for data augmentation. The following random transforms were applied on the fly during training in the following order: 1. p. 7 Affine transform. p. 7 An affine transform was applied with a probability of 0. p. 715, where the angles of rotation (in degrees) and the scaling factor were drawn from U(−30, 30) and U(0. p. 47, 1.3), respectively. p. 2 2. Flip. All patches were flipped with a probability of 0. p. 15 along each spatial dimension. p. 7 3. Gaussian Noise. p. 6 Each voxel of a patch was independently subject to additive white Gaussian noise with a variance drawn from U(0, 0. p. 81). This augmentation was applied with a probability of 0. p. 1 p. 115. 4. Gaussian Smoothing. p. 6 Each modality of a patch was filtered independently via a symmetric Gaussian kernel whose width (in voxels) was drawn from U(0. p. 85, 1.5). This augmentation was applied with a probability of 0.15. 5. Scale Intensity. p. 1 Voxel intensities were scaled with s ∼ U(0. p. 67, 1.3) with a probability of 0.15. 6. Shift Intensity. p. 8 Voxel intensities were shifted with offsets drawn from U(−0. p. 61, 0.1) with a probability of 0.15. 7. Adjust Contrast. p. 3 Gamma correction with γ ∼ U(0.7, 1.5) was applied voxel-wise with a probability of 0. p. 615.

Matricize Implementation Details

Algorithm 3 presents PyTorch implementations of Local and SW Matricize modules. The SWMatricize module first applies a local matricization and then concatenates the output with the matricized result of a rolled tensor. p. 6

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Matrix factorization models can be used to model local or global contexts by formulating Nonnegative Matrix Factorization as a differentiable layer integrated into a U-shaped architecture. This allows for an end-to-end medical image segmentation model that leverages low-rank matrix approximation for context modeling, achieving state of the art results on datasets like BraTS and ISLES’22.

  2. The Factorizer block can be constructed by replacing the selfattention layer of a ViT block with a Wrapped NMF module, which uses matricization and NMF to model contextual information. This provides a linearly scalable alternative to self-attention, enabling the exploitation of global context at the highest-resolution stage of a 3D network.

  3. The Shifted Window (SW) Matricize operation can be combined with NMF to effectively aggregate local information, mitigating issues where voxels close to boundaries are not represented effectively by regular patches. This leads to smoother feature maps around boundaries in the Factorizer block compared to Local Matricize alone.

  4. The Factorizer model can achieve a significant speed-up in inference for a trained model without extra steps and without sacrificing much accuracy, as revealed by ablation studies. This is achieved by short-circuiting some NMF layers or reducing the number of outer iterations in the Hierarchical Alternating Least Squares (HALS) solver.

  5. The NMF components within Factorizers are highly meaningful, giving an interpretability advantage over CNNs and Transformers because each component represents specific image semantics. This allows researchers to gain insight into what specific regions like tumor core or whole tumor are being detected by the model.

Sources

Related papers