Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements

arXiv:2111.13630 · eess.IV, cs.CV, cs.LG · Submitted 2021-11-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements".

Dev: Here's the gist of the paper. Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements is a work that proposes an efficient multi-organ segmentation model based on SpatialConfiguration-Net (SCN) to improve generalization and reduce computational requirements for clinical use Even though many semantic segmentation methods exist that are able to perform well on many medical datasets, often, they are not designed for direct use in clinical practice This work employs a multi-organ segmentation model based on the SpatialConfiguration-Net (SCN), which integrates prior knowledge of the spatial configuration among the labelled organs to resolve spurious responses in the network outputs Furthermore, it modifies the architecture of the segmentation model to reduce its memory footprint as much as possible without drastically impacting the quality of the predictions Lastly, a minimal inference script is implemented for which both execution time and required GPU memory are optimized.

Rosa: The proposed approach involves a two-stage process where a localization model determines the region of interest (ROI) before the segmentation model based on SCN is employed The localization model, Mloc, is implemented as a coarse binary segmentation CNN that treats all labelled organs as the foreground label and is trained using the full image content after greatly downsampling to reduce memory footprint The segmentation model, Mseg, utilizes the ROI predicted by the localization model to reduce image memory by cropping it, allowing content within the ROI to retain a larger resolution compared to the image used by the localization model.

Dev: Model Architecture and Training The segmentation model Mseg is based on SCN, which was originally proposed for landmark localization and later adapted to multi-label segmentation of the whole heart The architecture of the localization model and the local network of the segmentation model follow a similar structure, consisting of a sequence of contracting blocks followed by expanding blocks, similar to U-Net The loss function for training is formulated as Lseg = LGD(yseg, ˆyseg) + λ l segLCE(yseg, ˆy l seg) + λ s segLCE(yseg, ˆy 2 seg), where LGD and LCE represent the generalized dice loss and cross-entropy loss functions respectively.

Rosa: The proposed segmentation model uses 1,270,090 parameters and requires 8,797,627,020 FLOPS for an input size of 32 × 32 × 32 and up to 879,762,672,300 FLOPS when using an input size of 160 × 128 × 160.

Paper summary: Rosa: So, we're looking at this paper called "Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements". Essentially, the main idea is that there are a lot of segmentation methods out there for medical images, but they often aren't set up for real clinical use because they struggle with new data or just take way too much computing power.

Dev: Right. And this work proposes a multi-organ segmentation model based on something called the SpatialConfiguration-Net, which they claim helps improve generalization to unseen data by using prior knowledge about where the organs are located in space.

Rosa: Exactly, and they also focus heavily on making it efficient. They modified the architecture of this network to shrink its memory footprint without significantly hurting how accurate the predictions are. Plus, they optimized the script for inference so it runs fast and uses less GPU memory when you actually try to use it in a clinical setting.

Taro: I'm interested in how they handle that spatial knowledge integration. They hypothesize that because this Spatial Configuration Network has both a local and a spatial network, the spatial part can integrate prior knowledge about the arrangement of organs which helps resolve those weird, spurious responses you see when the network gets confused by new images.

Dev: That's what they're aiming for with their design. They use a two-stage process where first, a localization model identifies the region of interest or ROI using all the image content at a low resolution, and then they use that ROI to crop out a high-resolution piece of the original image for their main segmentation model.

Rosa: So, to summarize what we've covered so far, this paper by Thaler et al. introduces the SpatialConfiguration-Net approach to multi-organ segmentation, claiming it offers better generalization by using spatial configuration knowledge and significantly reduces computational needs through architectural modifications and inference script optimization.

Dev: It really boils down to balancing that quality improvement with the need for efficiency in a practical medical environment where resources are limited. This sets up a good expectation for how this method might be applied outside of just lab testing.

Taro: When you look at the results they report, they see very strong performance for organs like liver and kidney, achieving average Dice Scores around ninety-five percent or higher on the validation set <ref:2111.13630#pg1>. But then there's a dip with the pancreas, which got an average Dice Score of eighty-two point three percent <ref:2111.13630#pg1>.

Rosa: That gap in performance across different organs is something they point out when they talk about their qualitative findings. They observed that even healthy organs can get mislabeled if there are very large lesions present in those individual organs.

Dev: So, the authors suggest that to solve those specific issues with large lesions, maybe they need to think about an imbalanced training scheme or a different data augmentation strategy where they artificially inject images with very large lesions into their training set.

Taro: It sounds like they're acknowledging that the model is good generally, but it needs more examples of those extreme cases before it can handle them reliably.

Rosa: And that brings us to the conclusion of this paper, "Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements". The core message is about making complex segmentation models usable in clinical practice by focusing on both spatial awareness and efficiency.

Dev: They demonstrate that they can achieve high Dice Scores, like ninety-five percent for the liver and kidney, while keeping the computational demands relatively low through their architectural adjustments <ref:2111.13630#pg1>. It's about getting good results without requiring massive GPU resources.

Rosa: The paper suggests that for organs like liver and kidney, performance is quite robust, but the limitations they acknowledge are related to data imbalance when dealing with very large lesions in those specific regions.

Taro: So, it’s not a complete fix for every organ or every scenario, but it's a solid step toward creating models that are more practical for real-world deployment where you can't just run these massive models on demand.

Dev: And the title itself speaks to the whole point: efficiency and low GPU memory requirements alongside multi-organ segmentation using SpatialConfiguration-Net. It’s a direct address to the practical constraints of deploying deep learning in healthcare settings.

Conclusion: Rosa: So we've been looking at this paper about efficient multi-organ segmentation using something called SpatialConfiguration-Net, and now we’re getting to what they actually concluded about it.

Dev: They’re talking about how they managed to pack spatial knowledge into the network without blowing up the memory requirements too much.

Rosa: Exactly. It’s not just another segmentation method; it's trying to make these complex models practical for real clinical use by being smart about how they organize their data internally.

Dev: They show good results on organs like liver and kidney, getting those high Dice Scores, but they also point out where the system struggles with things like very large lesions in the pancreas.

Rosa: That's a key part of it. It means that while the model is generally pretty strong for healthy structures, if you have something really big and messy in an organ, it can still get confused or mislabeled.

Dev: And they suggest that to fix those specific problems with large lesions, the training data needs to be different—maybe artificially adding more of those hard cases into the training set.

Rosa: So when you think about this for the real world, it means we’re getting tools that are lighter on compute and better at recognizing where things *should* be based on their arrangement in space.

Dev: It sets up a good baseline for what efficient multi-organ segmentation looks like today, but those limitations with large lesions are definitely the next thing to test out.

Gottfried Schatz Research Center: Biophysics Medical University of Graz Department of Computer Graphics and Vision Graz University of Technology Institute of Computer Graphics and Vision

eess.IV, cs.CV, cs.LG

Submitted: 2021-11-26

Updated: 2026-11-12

Code: https://github.com/franzthaler/FLARE21-EfficientSCN

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 80/100

The gist: Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements is a work that proposes an efficient multi-organ segmentation model based on…

Key concepts

SpatialConfiguration-Net (SCN)
This network integrates prior knowledge about how different organs are spatially arranged. It helps the model correct incorrect predictions by understanding the expected spatial relationships between labeled organs, leading to more accurate segmentation.
Two-Stage Process
The system uses two sequential models: first, a localization model (Mloc) identifies areas of interest (ROIs) by treating all organs as foreground labels on downsampled images. Second, the main segmentation model (Mseg) then crops these ROIs for higher-resolution processing, saving memory.
Memory Optimization
The architecture is modified to minimize the model's memory footprint without severely harming prediction quality. This is achieved through a two-stage approach and a minimal inference script, ensuring the system runs efficiently on devices with limited GPU resources.
Multi-Organ Segmentation
The model aims to segment multiple organs simultaneously, such as liver, kidney, and spleen. The SCN architecture helps resolve errors specific to these complex tasks by leveraging spatial context among the various labeled organs.

Terminology

Summary

Efficient Multi-Organ Segmentation Using SpatialConfiguration-Net with Low GPU Memory Requirements is a work that proposes an efficient multi-organ segmentation model based on SpatialConfiguration-Net (SCN) to improve generalization and reduce computational requirements for clinical use Even though many semantic segmentation methods exist that are able to perform well on many medical datasets, often, they are not designed for direct use in clinical practice This work employs a multi-organ segmentation model based on the SpatialConfiguration-Net (SCN), which integrates prior knowledge of the spatial configuration among the labelled organs to resolve spurious responses in the network outputs Furthermore, it modifies the architecture of the segmentation model to reduce its memory footprint as much as possible without drastically impacting the quality of the predictions Lastly, a minimal inference script is implemented for which both execution time and required GPU memory are optimized. p. 0

Where the paper says it

In this work, we employ a multi-organ segmentation model based on the SpatialConfiguration-Net (SCN), which integrates prior knowledge of the spatial configuration among the labelled organs to resolve spurious responses in the network outputs. Furthermore, we modified the architecture of the segmentation model to reduce its memory footprint as much as possible without drastically impacting the quality of the predictions.

Methodology Overview

The proposed approach involves a two-stage process where a localization model determines the region of interest (ROI) before the segmentation model based on SCN is employed The localization model, Mloc, is implemented as a coarse binary segmentation CNN that treats all labelled organs as the foreground label and is trained using the full image content after greatly downsampling to reduce memory footprint The segmentation model, Mseg, utilizes the ROI predicted by the localization model to reduce image memory by cropping it, allowing content within the ROI to retain a larger resolution compared to the image used by the localization model.

Model Architecture and Training

The segmentation model Mseg is based on SCN, which was originally proposed for landmark localization and later adapted to multi-label segmentation of the whole heart The architecture of the localization model and the local network of the segmentation model follow a similar structure, consisting of a sequence of contracting blocks followed by expanding blocks, similar to U-Net The loss function for training is formulated as Lseg = LGD(yseg, ˆyseg) + λ l segLCE(yseg, ˆy l seg) + λ s segLCE(yseg, ˆy 2 seg), where LGD and LCE represent the generalized dice loss and cross-entropy loss functions respectively. The proposed segmentation model uses 1,270,090 parameters and requires 8,797,627,020 FLOPS for an input size of 32 × 32 × 32 and up to 879,762,672,300 FLOPS when using an input size of 160 × 128 × 160. p. 2

Where the paper says it

The terms LGD and LCE represent the generalized dice loss and cross-entropy loss functions respectively and λ 1 seg and λ 2 seg are hyperparameters used to weight the individual loss functions. Overall, our segmentation model uses 1,270,090 parameters and requires 8,797,627,020 FLOPS for an input size of 32 × 32 × 32 and 879,762,672,300 FLOPS when using an input size of 160 × 128 × 160.

Optimization for Efficiency

To optimize inference time and GPU memory, a minimal script is implemented where the same preprocessing strategy as during training is performed before generating predictions to compute the ROI The training protocols involved 4-fold cross-validation where each segmentation model was trained for 100,000 iterations for Mloc and 500,000 iterations for Mseg. The training times observed were Mloc at 11. p. 3

Where the paper says it

Training protocols. Data augmentation methods Rotation, translation, scaling, elastic deformation, intensity shift and scaling Initialization of the network He normal initialization Batch size 1 Patch sampling strategy Mloc full image Patch size Mloc [32 × 32 × 32] to [80 × 80 × 256] Image spacing Mloc 6 × 6 × 6 mm Total iterations Mloc 1,000,000 Patch sampling strategy Mseg ROI from ground truth (training) or localization model (inference) Patch size Mseg [32 × 32 × 32] to [160 × 128 × 160] Image spacing Mseg 2 × 2 × 2 mm Total iterations Mseg 500,000 Optimizer Adam (β1 = 0.

0 hours and Mseg at 49. p. 3
Where the paper says it

Training time Mloc 11. 0 hours Training time Mseg 49.

0 hours.

Performance Evaluation

Quantitative results from the 4-fold cross-validation show good performance, with the average Dice Score (DSC) for liver, kidney, and spleen being 97.8%, 95.0%, and 97.2% respectively. p. 2

Where the paper says it

Defining the input and output of the local network as x 1 seg and ˆy 1 seg and of the spatial network as x 2 seg and ˆy 2 seg allows formulate the loss function as: Lseg = LGD(yseg, ˆyseg) + λ 1 segLCE(yseg, ˆy 1 seg) + λ 2 segLCE(yseg, ˆy 2 seg), (2) where the ground truth yseg was resampled identically to the image xseg. The terms LGD and LCE represent the generalized dice loss and cross-entropy loss functions respectively and λ 1 seg and λ 2 seg are hyperparameters used to weight the individual loss functions.

The pancreas achieved the lowest DSC of 82. p. 4
Where the paper says it

The average Dice Score (DSC) of liver, kidney and spleen are 97. 2%, respectively, while the pancreas has the lowest DSC of 82.

3%. On the validation set, liver, kidney, and spleen achieved average DSC scores of 95. p. 4
Where the paper says it

The quantitative results of the submitted model on the validation set given in Table 5 follow the same trend with liver, kidney and spleen performing a lot better in terms of DSC and NSD on average compared to the pancreas. The segmentation of both, liver and spleen, achieved a DSC of over 90% closely followed by the kidney with 89.

43%, 89.74%, and 93.88% respectively The performance gap between cross-validation and the validation set is attributed to the data used for training versus evaluation, as the validation set includes images from nine different centers with various phases. p. 5
Where the paper says it

In contrast, the validation set contains images from 9 different centers of which 8 have been unseen during training as well as images with various phases, including many samples with phases also unseen during training. Consequently, while the cross-validation set can be used to evaluate how well the model performs on the task, the evaluation on the validation set shows how well the model generalizes to data from different centers with various phases.

Qualitative Findings

Qualitative results illustrate that while healthy organs are segmented very well, large lesions in individual organs can lead to incorrect labeling or mislabeling by the segmentation model For instance, difficult cases from the validation set showed large lesions in the liver and spleen that were not labeled correctly. This observation suggests that additional data containing very large lesions would be beneficial to solve problems observed in these regions. p. 6

Where the paper says it

In contrast, large lesions of individual organs are only present in a small number of images which can lead to lacking segmentation predictions for unseen images if this type of pathology was never or not sufficiently encountered during training. This indicates, that additional data containing very large lesions would be beneficial to solve the problems observed in these regions.

The authors suggest using an imbalanced training scheme or a more sophisticated data augmentation strategy where large lesions are injected artificially into the image. p. 6
Where the paper says it

This indicates, that additional data containing very large lesions would be beneficial to solve the problems observed in these regions. Other potential solutions, which are not based on acquiring additional data, are to either use an imbalanced training scheme, where images with large lesions are selected more frequently during training, or a more sophisticated data augmentation strategy, where large lesions are injected artificially into the image using, e.

Conclusion

The segmentation for liver, kidney, and spleen performs very well with each achieving a DSC of at least 95%, while the segmentation of the pancreas only reached 82.3% The results on the validation set confirm that liver and spleen achieved a DSC over 90% closely followed by the kidney with 89. p. 4

Where the paper says it

The quantitative results of the submitted model on the validation set given in Table 5 follow the same trend with liver, kidney and spleen performing a lot better in terms of DSC and NSD on average compared to the pancreas. The segmentation of both, liver and spleen, achieved a DSC of over 90% closely followed by the kidney with 89.

74%, whereas the pancreas only managed to reach a DSC of 75. p. 4
Where the paper says it

The segmentation of both, liver and spleen, achieved a DSC of over 90% closely followed by the kidney with 89. 74%, however, the pancreas only managed to reach a DSC of 75.

15%. The performance gap is caused by the data used to train and evaluate the respective model, as validation set images are from different centers and phases than those used in training. p. 5
Where the paper says it

This gap in performance is caused by the data used to train and evaluate the respective model, as the cross-validation models were trained and evaluated only on subsets of the training set, while the submitted model was trained on the full training set and evaluated on the validation set. As can be seen in Table 1, the training set only consists of images obtained from two centers with the same phase.

Acknowledgment

The authors of this paper declare that the segmentation method they implemented for participation in the FLARE challenge has not used any pre-trained models nor additional datasets other than those provided by the organizers. This work was supported by the Austrian Research Promotion Agency (FFG): 871262. p. 6

Where the paper says it

Acknowledgment The authors of this paper declare that the segmentation method they implemented for participation in the FLARE challenge has not used any pre-trained models nor additional datasets other than those provided by the organizers. This work was supported by the Austrian Research Promotion Agency (FFG): 871262.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Spatial Configuration Network integration resolves spurious responses in network outputs by integrating prior knowledge of the spatial configuration among the labelled organs. This allows for more robust segmentation predictions on unseen data acquired from different scanners.

  2. The localization model Mloc, adapted from U-Net, determines a Region of Interest (ROI) to reduce memory footprint before employing the SCN segmentation model. This two-stage approach improves efficiency and accuracy by focusing the computationally intensive segmentation on relevant areas.

  3. The proposed loss function Lseg combines generalized dice loss with cross-entropy loss between the ground truth and intermediate predictions from both the local and spatial networks. This multi-loss approach ensures that both subnetworks contribute to a plausible final prediction, as formulated in Equation 2 of the paper.

  4. Inference time and required GPU memory are optimized through a minimal inference script that performs preprocessing identically to training but minimizes computational overhead. This optimization allows for deployment on resource-constrained clinical systems while maintaining prediction quality.

Abstract

Even though many semantic segmentation methods exist that are able to perform well on many medical datasets, they are often not designed for direct use in clinical practice. The two main concerns are generalization to unseen data with a different visual appearance, e.g., images acquired using a different scanner, and efficiency in terms of computation time and required Graphics Processing Unit (GPU) memory. In this work, we employ a multi-organ segmentation model based on the SpatialConfiguration-Net (SCN), which integrates prior knowledge of the spatial configuration among the labelled organs to resolve spurious responses in the network outputs. Furthermore, we modified the architecture of the segmentation model to reduce its memory footprint as much as possible without drastically impacting the quality of the predictions. Lastly, we implemented a minimal inference script for which we optimized both execution time and required GPU memory.

Sources

Related papers