NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classification

arXiv:2408.01797 · eess.IV, cs.CV · Submitted 2024-08-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classification".

Dev: Here's the gist of the paper. In this work, NuLite introduces a lightweight and fast model for nuclei instance segmentation and classification in Hematoxylin and Eosin (H&E) stained histopathological images <ref:2408.01797#pg2>.

Rosa: The experimental results prove that our models are equivalent to CellViT (SOTA) in terms of panoptic quality and F-score <ref:2408.01797#pg2>.

Dev: Model Architecture and Design NuLite is a U-Net-like architecture designed explicitly to be lightweight and fast <ref:2408.01797#pg3>.

Rosa: The architecture consists of the FastViT

13: encoder, one decoder, and three segmentation heads purpose-built to perform one of the tasks identified in the HoVer-Net: nuclei prediction, horizontal and vertical map prediction, and nuclei classification <ref:2408.01797#pg4>.

Paper summary: Dev: Thinking about the title, NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classification—it really tells you exactly what they set out to accomplish zero. It’s not trying to be the biggest model possible; it’s focused on being fast and light while still delivering state of the art performance on nuclei tasks.

Rosa: And that’s where the authors are pushing, showing that you don't always need massive computational power for high-quality results in this specific domain zero. They proved their models are equivalent to CellViT in terms of panoptic quality and F-score across several benchmarks like PanNuke zero.

Taro: So, what’s the implication for a researcher who is trying to build a diagnostic tool? Are they suggesting that for routine analysis, you can probably use something much lighter than the heaviest models currently proposed in the literature zero?

Dev: They are suggesting that efficiency isn't just an afterthought; it's built into the architecture from the start by designing a U-Net like structure with specific heads and a simplified decoder zero. This makes it practical for real-world use where latency matters.

Rosa: It’s about making these advanced techniques accessible to more users, not just researchers with access to massive GPU clusters zero. The paper's mention of NuLite-T being fifty eight times smaller in parameters is a concrete reason why this matters for deployment.

Taro: So, the main point is that you can get performance that rivals the most advanced models while drastically cutting down on computational requirements zero. It’s about moving from theoretical potential to something that’s actually runnable.

Dev: Yeah, it moves us toward models where we can actually discuss real-time loops or integrate these analyses into faster workflows without waiting too long for a result zero. The whole point of the work is showing that high quality and high speed can coexist in this context.

Conclusion: Rosa: So, NuLite is this new model that they put out, trying to do nuclei segmentation and classification in H andE images, and they call it lightweight and fast zero.

Dev: Right, so it’s about making a model that runs without needing a super powerful computer. They claim it's fast enough for real-time use.

Rosa: They compared it against CellViT, which is the state of the art thing right now, and they said NuLite is on par with CellViT in terms of quality metrics like panoptic quality and F-score zero.

Dev: That’s a big claim if it means you can get that level of performance without the massive computational cost. I need to know what the actual trade-off is there.

Rosa: The paper shows that NuLite is way smaller, about fifty eight times fewer parameters and ten times less GFlops than CellViT zero. That’s a huge difference when you're thinking about deployment on smaller hardware.

Dev: But what about the training? How did they get it to match those results when the underlying architecture is so simplified compared to something like CellViT?

Rosa: They used a specific set of loss functions for training, combining things like Focal Tversky loss and Dice loss for segmentation, and Mean Square Error and Mean Square Gradient Error for the orientation maps zero. It’s a customized recipe.

Dev: So it's not just throwing a small network at the problem; they tuned the training process to make those different parts work together well. That sounds like a lot of fine-tuning work on top of the design itself.

Rosa: Exactly, and when you look at external datasets, like CoNseP or MoNuSeg, NuLite still hits state of the art results and beats CellViT in precision and recall zero. It holds up in more varied testing.

Dev: So it seems they’ve managed to keep the performance high while aggressively cutting down on the hardware needs. The implication for me is that we might be able to run these kinds of analyses where latency is critical, like on a field robot or a portable device.

Rosa: That’s the thing—it moves these powerful segmentation tools out of just huge labs and into places where you actually need them in the field zero.

Dev: And that brings us to how it handles unexpected data, Taro needs to look at this next. When the world misbehaves, does this lightweight model still keep up gracefully?

Department of Electrical Engineering and Information Technology University of Naples Federico II

eess.IV, cs.CV

Submitted: 2024-08-03

Updated: 2024-08-09

DOI: 10.1016/j.bspc.2025.109333

Code: https://github.com/CosmoIknosLab/NuLite

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 87/100

The gist: In this work, NuLite introduces a lightweight and fast model for nuclei instance segmentation and classification in Hematoxylin and Eosin (H&E) stained histopathological images <ref:2408.01797#pg2>.

Key concepts

NuLite Architecture
NuLite is a U-Net variant specifically designed to be lightweight and fast. It uses a FastViT encoder, one decoder, and three distinct segmentation heads: one for nucleus segmentation, one for horizontal/vertical map prediction, and one for nucleus classification.
Segmentation Heads
The model has three specialized output heads. The NP-HEAD segments nuclei against the background. The HV-HEAD predicts horizontal and vertical orientation maps of the nuclei. Finally, the NC-HEAD performs the classification task to determine what kind of nucleus it is.
Loss Functions (LNP/LHV)
Training uses a combination of loss functions for different tasks. LNP combines Focal Tversky loss and Dice loss for segmentation, while LHV combines Mean Square Error and Mean Square Gradient Error for orientation prediction. The total loss sums these individual losses.
Efficiency Metrics
NuLite-T is significantly smaller than CellViT, being about 58 times smaller in parameters and 10 times smaller in GFlops. This results in up to 13 times faster inference speed on GPU latency, making it highly efficient.

Terminology

Summary

In this work, NuLite introduces a lightweight and fast model for nuclei instance segmentation and classification in Hematoxylin and Eosin (H&E) stained histopathological images <ref:2408. p. 0

Where the paper says it

IV] 9 Aug 2024 NuLite - Lightweight and Fast Model for Nuclei Instance Segmentation and Classification Cristian Tommasinoa,∗ , Cristiano Russoa , Antonio M. Rinaldia aDepartment of Electrical Engineering and Information Technology University of Naples Federico II, Via Claudio, 21, Naples 80125, Italy Abstract In pathology, accurate and efficient analysis of Hematoxylin and Eosin (H&E) slides is crucial for timely and effective cancer diagnosis.

01797#pg2>. The experimental results prove that our models are equivalent to CellViT (SOTA) in terms of panoptic quality and F-score <ref:2408. p. 0
Where the paper says it

We obtained three versions of our model, NuLite-S, NuLite-M, and NuLite-H, trained on the PanNuke dataset. The experimental results prove that our models are equivalent to CellViT (SOTA) in terms of panoptic quality and F-score.

01797#pg2>.

Model Architecture and Design

NuLite is a U-Net-like architecture designed explicitly to be lightweight and fast <ref:2408.01797#pg3>. The architecture consists of the FastViT [13] encoder, one decoder, and three segmentation heads purpose-built to perform one of the tasks identified in the HoVer-Net: nuclei prediction, horizontal and vertical map prediction, and nuclei classification <ref:2408. p. 1

Where the paper says it

architecture designed for segmenting and classifying nuclei instances in Hematoxylin and Eosin (H&E) images. Our architecture consists of the FastViT [13] encoder, one decoder, and three segmentation heads purpose-built to perform one of the tasks identified in the HoVer-Net: nuclei prediction, horizontal and vertical map prediction, and nuclei classification [11].

01797#pg4>. The network comprises three heads: the NP-HEAD for nuclei segmentation against the background, the HV-HEAD for predicting horizontal and vertical orientation maps, and the NC-HEAD for nuclei classification <ref:2408. p. 2
Where the paper says it

Therefore, our model predicts nuclei maps, type maps, and horizontal and vertical maps, followed by a watershed algorithm to perform nuclei instance segmentation in a postprocessing step. Thus, our network comprises three heads: the NP-HEAD for nuclei segmentation against the background, the HV-HEAD for predicting horizontal and vertical orientation maps, and the NC-HEAD for nuclei classification, as illustrated in Figure 1.

01797#pg5>. The decoder consists of five main layers and three segmentation heads <ref:2408. p. 2
Where the paper says it

We focused on the decoder design and built it to work with the FastViT [13] encoder. Therefore, the decoder consists of five main layers and three segmentation heads, as detailed in Table 1.

01797#pg6>.

Training Methodology

To train NuLite, a combination of different loss functions is used for each network output <ref:2408.01797#pg7>. The total loss is defined as the sum of a loss for each segmentation head, as shown in Equation 1 <ref:2408. p. 3

Where the paper says it

Loss fuction To train NuLite, we use a combination of different loss functions for each network output, as also suggested in [11, 12]. Therefore, the total loss is defined as the sum of a loss for each segmentation head, as shown in Equation 1.

01797#pg7>. Specifically, LNP is defined as a linear combination of Focal Tversky loss (FTL) and Dice loss (DICE) <ref:2408. p. 3
Where the paper says it

Therefore, the total loss is defined as the sum of a loss for each segmentation head, as shown in Equation 1. Ltotal = LNP + LHV + LNT + LTC (1) LNP is the loss for the NP-HEAD, defined as a linear combination of Focal Tversky loss (FTL) and Dice loss (DICE), as shown in Equation 2.

01797#pg7>. LHV is defined as a linear combination of Mean Square Error (MSE) and Mean Square Gradient Error (MSGE) <ref:2408. p. 3
Where the paper says it

Ltotal = LNP + LHV + LNT + LTC (1) LNP is the loss for the NP-HEAD, defined as a linear combination of Focal Tversky loss (FTL) and Dice loss (DICE), as shown in Equation 2. LNP = λ FTL NP LFTL + λ DICE NP LDICE (2) LHV is the loss for the HV-HEAD, defined as a linear combination of Mean Square Error (MSE) and Mean Square Gradient Error (MSGE), as shown in Equation 3.

01797#pg7>.

Performance Evaluation and Comparison

The model's efficacy is proven through rigorous testing on benchmark datasets such as PanNuke [14] <ref:2408.01797#pg3>. NuLite consistently achieved SOTA results in these tests, outperforming advanced models like CellViT in various metrics, including precision, recall, and F1-score <ref:2408. p. 1

Where the paper says it

Furthermore, we conducted comprehensive evaluations on additional datasets such as MoNuSeg [15], CoNSeP [11], and GlySAC [16]. NuLite consistently achieved SOTA results in these tests, outperforming advanced models like CellViT in various metrics, including precision, recall, and F1-score.

01797#pg4>. The model is publicly available at https://github. p. 0
Where the paper says it

Lastly, to prove the effectiveness of our solution, we provide a robust comparison of external datasets, namely CoNseP, MoNuSeg, and GlySAC. Our model is publicly available at https://github.

com/CosmoIknosLab/NuLite <ref:2408. p. 0
Where the paper says it

Our model is publicly available at https://github. com/CosmoIknosLab/NuLite.

01797#pg2>.

Efficiency and Generalization

The lightest model, NuLite-T, is about 58 times smaller in terms of parameters and about 10 times smaller in terms of GFlops <ref:2408.01797#pg2>. Considering the GPU latency, our model is up to about 13 times faster than CellViT <ref:2408. p. 0

Where the paper says it

In comparison, our heaviest model is about 15 times smaller in terms of parameters and about 7 times smaller in terms of GFlops. Moreover, considering the GPU latency, our model is up to about 13 times faster than CellViT.

01797#pg2>. The performance evaluations on external datasets such as CoNseP, MoNuSeg, and GlySAC show that NuLite consistently achieved SOTA results in these tests, outperforming advanced models like CellViT in various metrics <ref:2408. p. 1
Where the paper says it

Furthermore, we conducted comprehensive evaluations on additional datasets such as MoNuSeg [15], CoNSeP [11], and GlySAC [16]. NuLite consistently achieved SOTA results in these tests, outperforming advanced models like CellViT in various metrics, including precision, recall, and F1-score.

01797#pg3>. In terms of complexity analysis, all versions of NuLite are significantly lower than CellViT in terms of GFLOPS and parameters <ref:2408. p. 8
Where the paper says it

Models complexity analysis To prove that our model, NuLite, has a lower complexity than CellViT, in Table 6, we report an exhaustive comparison between NuLite and CellViT in terms of parameters count and GFlops, estimated size, and latency on GPU using an input shape of 256 and 1024. In particular, we consider all FastViT models (T8, T12, S12, SA24, SA36, and MA36) and the reparameterized versions in the inference step.

01797#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved model architecture for efficiency by using one decoder instead of multiple decoders to avoid parameter redundancy among the decoders and further reduce the parameters and GFLOPS. This makes NuLite a faster, lighter alternative with state-of-the-art (SOTA) panoptic quality and detection performance.

  2. Enhanced postprocessing pipeline by leveraging three specialized heads: NP-HEAD for nuclei segmentation against the background, the HV-HEAD for predicting horizontal and vertical orientation maps, and the NC-HEAD for nuclei classification. This allows for a robust comparison of external datasets by performing instance segmentation with a watershed algorithm using outputs from these heads.

  3. Optimized training process using a combined loss function: Ltotal = LNP + LHV + LNT + LTC (1), which sums losses for nuclei prediction, orientation maps, nucleus types, and tissue classification. This allows for fine-grained control over different aspects of the segmentation and classification tasks simultaneously.

  4. Robust performance across diverse pathological data by evaluating NuLite on external datasets like CoNseP, MoNuSeg, and GlySAC. The paper demonstrates that NuLite consistently achieved SOTA results in these tests, proving its robustness and generalizability across different types of histopathological images.

  5. Optimized model selection strategy using a tiered approach based on performance metrics. The authors selected NuLite versions by grouping them by GFLOPS and choosing the best performer in each group, leading to the selection of FastViT-S12 as the backbone for NuLite-T(iny), FastViT-SA24 as the backbone of NuList-M(medium), and FastViT-SA36 for NuLite-H(uge).

  6. Improved inference speed by utilizing a reparameterized version of the Vision Transformer encoder. The paper shows that reparameterization does not significantly affect performance but results in superior computational efficiency, leading to NuLite models being up to about 13 times faster than CellViT.

  7. Multi-scale analysis capability by comparing performance across different input patch sizes, specifically 256 × 256 pixels with 64-pixel overlap vs. 1024 × 1024 pixels. This demonstrates that using larger patches (1024 × 1024 pixels) does not negatively impact the performance and may even slightly improve it in some cases.

  8. Versatile analysis across complex tissue types by achieving superior results in specific categories on external datasets. For instance, NuLite-M and NuLite-H show notable strengths in specific categories, with NuLite-H showing superior recall and F1 scores for the CoNSeP dataset, indicating its ability to handle complex tissue classification tasks.

  9. Streamlined Whole Slide Imaging (WSI) analysis capability. The research suggests that the ability to effectively use larger patches (1024 × 1024 pixels) could significantly streamline the process of analyzing large WSI data, reducing the need for extensive patch overlap and accelerating the segmentation process.

Abstract

In pathology, accurate and efficient analysis of Hematoxylin and Eosin (H&E) slides is crucial for timely and effective cancer diagnosis. Although many deep learning solutions for nuclei instance segmentation and classification exist in the literature, they often entail high computational costs and resource requirements, thus limiting their practical usage in medical applications. To address this issue, we introduce a novel convolutional neural network, NuLite, a U-Net-like architecture designed explicitly on Fast-ViT, a state-of-the-art (SOTA) lightweight CNN. We obtained three versions of our model, NuLite-S, NuLite-M, and NuLite-H, trained on the PanNuke dataset. The experimental results prove that our models equal CellViT (SOTA) in terms of panoptic quality and detection. However, our lightest model, NuLite-S, is 40 times smaller in terms of parameters and about 8 times smaller in terms of GFlops, while our heaviest model is 17 times smaller in terms of parameters and about 7 times smaller in terms of GFlops. Moreover, our model is up to about 8 times faster than CellViT. Lastly, to prove the effectiveness of our solution, we provide a robust comparison of external datasets, namely CoNseP, MoNuSeg, and GlySAC. Our model is publicly available at https://github.com/CosmoIknosLab/NuLite

Sources

Related papers