NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classification
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classification".
Dev: Here's the gist of the paper. In this work, NuLite introduces a lightweight and fast model for nuclei instance segmentation and classification in Hematoxylin and Eosin (H&E) stained histopathological images <ref:2408.01797#pg2>.
Rosa: The experimental results prove that our models are equivalent to CellViT (SOTA) in terms of panoptic quality and F-score <ref:2408.01797#pg2>.
Dev: Model Architecture and Design NuLite is a U-Net-like architecture designed explicitly to be lightweight and fast <ref:2408.01797#pg3>.
Rosa: The architecture consists of the FastViT
13: encoder, one decoder, and three segmentation heads purpose-built to perform one of the tasks identified in the HoVer-Net: nuclei prediction, horizontal and vertical map prediction, and nuclei classification <ref:2408.01797#pg4>.
Paper summary: Dev: Thinking about the title, NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classification—it really tells you exactly what they set out to accomplish zero. It’s not trying to be the biggest model possible; it’s focused on being fast and light while still delivering state of the art performance on nuclei tasks.
Rosa: And that’s where the authors are pushing, showing that you don't always need massive computational power for high-quality results in this specific domain zero. They proved their models are equivalent to CellViT in terms of panoptic quality and F-score across several benchmarks like PanNuke zero.
Taro: So, what’s the implication for a researcher who is trying to build a diagnostic tool? Are they suggesting that for routine analysis, you can probably use something much lighter than the heaviest models currently proposed in the literature zero?
Dev: They are suggesting that efficiency isn't just an afterthought; it's built into the architecture from the start by designing a U-Net like structure with specific heads and a simplified decoder zero. This makes it practical for real-world use where latency matters.
Rosa: It’s about making these advanced techniques accessible to more users, not just researchers with access to massive GPU clusters zero. The paper's mention of NuLite-T being fifty eight times smaller in parameters is a concrete reason why this matters for deployment.
Taro: So, the main point is that you can get performance that rivals the most advanced models while drastically cutting down on computational requirements zero. It’s about moving from theoretical potential to something that’s actually runnable.
Dev: Yeah, it moves us toward models where we can actually discuss real-time loops or integrate these analyses into faster workflows without waiting too long for a result zero. The whole point of the work is showing that high quality and high speed can coexist in this context.
Conclusion: Rosa: So, NuLite is this new model that they put out, trying to do nuclei segmentation and classification in H andE images, and they call it lightweight and fast zero.
Dev: Right, so it’s about making a model that runs without needing a super powerful computer. They claim it's fast enough for real-time use.
Rosa: They compared it against CellViT, which is the state of the art thing right now, and they said NuLite is on par with CellViT in terms of quality metrics like panoptic quality and F-score zero.
Dev: That’s a big claim if it means you can get that level of performance without the massive computational cost. I need to know what the actual trade-off is there.
Rosa: The paper shows that NuLite is way smaller, about fifty eight times fewer parameters and ten times less GFlops than CellViT zero. That’s a huge difference when you're thinking about deployment on smaller hardware.
Dev: But what about the training? How did they get it to match those results when the underlying architecture is so simplified compared to something like CellViT?
Rosa: They used a specific set of loss functions for training, combining things like Focal Tversky loss and Dice loss for segmentation, and Mean Square Error and Mean Square Gradient Error for the orientation maps zero. It’s a customized recipe.
Dev: So it's not just throwing a small network at the problem; they tuned the training process to make those different parts work together well. That sounds like a lot of fine-tuning work on top of the design itself.
Rosa: Exactly, and when you look at external datasets, like CoNseP or MoNuSeg, NuLite still hits state of the art results and beats CellViT in precision and recall zero. It holds up in more varied testing.
Dev: So it seems they’ve managed to keep the performance high while aggressively cutting down on the hardware needs. The implication for me is that we might be able to run these kinds of analyses where latency is critical, like on a field robot or a portable device.
Rosa: That’s the thing—it moves these powerful segmentation tools out of just huge labs and into places where you actually need them in the field zero.
Dev: And that brings us to how it handles unexpected data, Taro needs to look at this next. When the world misbehaves, does this lightweight model still keep up gracefully?
Department of Electrical Engineering and Information Technology University of Naples Federico II
eess.IV, cs.CV
Submitted: 2024-08-03
Updated: 2024-08-09
DOI: 10.1016/j.bspc.2025.109333
Code: https://github.com/CosmoIknosLab/NuLite
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 87/100
The gist: In this work, NuLite introduces a lightweight and fast model for nuclei instance segmentation and classification in Hematoxylin and Eosin (H&E) stained histopathological images <ref:2408.01797#pg2>.
Key concepts
- NuLite Architecture
- NuLite is a U-Net variant specifically designed to be lightweight and fast. It uses a FastViT encoder, one decoder, and three distinct segmentation heads: one for nucleus segmentation, one for horizontal/vertical map prediction, and one for nucleus classification.
- Segmentation Heads
- The model has three specialized output heads. The NP-HEAD segments nuclei against the background. The HV-HEAD predicts horizontal and vertical orientation maps of the nuclei. Finally, the NC-HEAD performs the classification task to determine what kind of nucleus it is.
- Loss Functions (LNP/LHV)
- Training uses a combination of loss functions for different tasks. LNP combines Focal Tversky loss and Dice loss for segmentation, while LHV combines Mean Square Error and Mean Square Gradient Error for orientation prediction. The total loss sums these individual losses.
- Efficiency Metrics
- NuLite-T is significantly smaller than CellViT, being about 58 times smaller in parameters and 10 times smaller in GFlops. This results in up to 13 times faster inference speed on GPU latency, making it highly efficient.
Terminology
Summary
In this work, NuLite introduces a lightweight and fast model for nuclei instance segmentation and classification in Hematoxylin and Eosin (H&E) stained histopathological images <ref:2408. p. 0 IV] 9 Aug 2024
NuLite - Lightweight and Fast Model for Nuclei Instance Segmentation and Classification
Cristian Tommasinoa,∗
, Cristiano Russoa
, Antonio M. Rinaldia
aDepartment of Electrical Engineering and Information Technology University of Naples Federico II, Via Claudio, 21, Naples 80125, Italy
Abstract
In pathology, accurate and efficient analysis of Hematoxylin and Eosin (H&E) slides is crucial for timely and effective cancer
diagnosis. We obtained three versions of our model, NuLite-S, NuLite-M, and NuLite-H, trained on the PanNuke dataset. The experimental
results prove that our models are equivalent to CellViT (SOTA) in terms of panoptic quality and F-score.Where the paper says it
Where the paper says it
Model Architecture and Design
NuLite is a U-Net-like architecture designed explicitly to be lightweight and fast <ref:2408.01797#pg3>. The architecture consists of the FastViT [13] encoder, one decoder, and three segmentation heads purpose-built to perform one of the tasks identified in the HoVer-Net: nuclei prediction, horizontal and vertical map prediction, and nuclei classification <ref:2408. p. 1 architecture designed for segmenting and classifying nuclei instances in Hematoxylin and Eosin (H&E) images. Our architecture consists of the FastViT [13] encoder, one decoder,
and three segmentation heads purpose-built to perform one of
the tasks identified in the HoVer-Net: nuclei prediction, horizontal and vertical map prediction, and nuclei classification
[11]. Therefore, our
model predicts nuclei maps, type maps, and horizontal and vertical maps, followed by a watershed algorithm to perform nuclei instance segmentation in a postprocessing step. Thus, our
network comprises three heads: the NP-HEAD for nuclei segmentation against the background, the HV-HEAD for predicting horizontal and vertical orientation maps, and the NC-HEAD
for nuclei classification, as illustrated in Figure 1. We focused on the decoder design and built it to work with
the FastViT [13] encoder. Therefore, the decoder consists of
five main layers and three segmentation heads, as detailed in
Table 1.Where the paper says it
Where the paper says it
Where the paper says it
Training Methodology
To train NuLite, a combination of different loss functions is used for each network output <ref:2408.01797#pg7>. The total loss is defined as the sum of a loss for each segmentation head, as shown in Equation 1 <ref:2408. p. 3 Loss fuction
To train NuLite, we use a combination of different loss functions for each network output, as also suggested in [11, 12]. Therefore, the total loss is defined as the sum of a loss for each
segmentation head, as shown in Equation 1. Therefore, the total loss is defined as the sum of a loss for each
segmentation head, as shown in Equation 1. Ltotal = LNP + LHV + LNT + LTC (1)
LNP is the loss for the NP-HEAD, defined as a linear combination of Focal Tversky loss (FTL) and Dice loss (DICE), as
shown in Equation 2. Ltotal = LNP + LHV + LNT + LTC (1)
LNP is the loss for the NP-HEAD, defined as a linear combination of Focal Tversky loss (FTL) and Dice loss (DICE), as
shown in Equation 2. LNP = λ
FTL
NP LFTL + λ
DICE
NP LDICE (2)
LHV is the loss for the HV-HEAD, defined as a linear combination of Mean Square Error (MSE) and Mean Square Gradient
Error (MSGE), as shown in Equation 3.Where the paper says it
Where the paper says it
Where the paper says it
Performance Evaluation and Comparison
The model's efficacy is proven through rigorous testing on benchmark datasets such as PanNuke [14] <ref:2408.01797#pg3>. NuLite consistently achieved SOTA results in these tests, outperforming advanced models like CellViT in various metrics, including precision, recall, and F1-score <ref:2408. p. 1 Furthermore, we conducted comprehensive evaluations on additional
datasets such as MoNuSeg [15], CoNSeP [11], and GlySAC
[16]. NuLite consistently achieved SOTA results in these tests,
outperforming advanced models like CellViT in various metrics, including precision, recall, and F1-score. Lastly, to prove the effectiveness of our
solution, we provide a robust comparison of external datasets, namely CoNseP, MoNuSeg, and GlySAC. Our model is publicly
available at https://github. Our model is publicly
available at https://github. com/CosmoIknosLab/NuLite.Where the paper says it
Where the paper says it
Where the paper says it
Efficiency and Generalization
The lightest model, NuLite-T, is about 58 times smaller in terms of parameters and about 10 times smaller in terms of GFlops <ref:2408.01797#pg2>. Considering the GPU latency, our model is up to about 13 times faster than CellViT <ref:2408. p. 0 In comparison,
our heaviest model is about 15 times smaller in terms of parameters and about 7 times smaller in terms of GFlops. Moreover,
considering the GPU latency, our model is up to about 13 times faster than CellViT. Furthermore, we conducted comprehensive evaluations on additional
datasets such as MoNuSeg [15], CoNSeP [11], and GlySAC
[16]. NuLite consistently achieved SOTA results in these tests,
outperforming advanced models like CellViT in various metrics, including precision, recall, and F1-score. Models complexity analysis
To prove that our model, NuLite, has a lower complexity than
CellViT, in Table 6, we report an exhaustive comparison between NuLite and CellViT in terms of parameters count and
GFlops, estimated size, and latency on GPU using an input
shape of 256 and 1024. In particular, we consider all FastViT
models (T8, T12, S12, SA24, SA36, and MA36) and the reparameterized versions in the inference step.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved model architecture for efficiency by using
one decoder
instead of multiple decoders toavoid parameter redundancy among the decoders and further reduce the parameters and GFLOPS.
This makes NuLite afaster, lighter alternative with state-of-the-art (SOTA) panoptic quality and detection performance.
-
Enhanced postprocessing pipeline by leveraging three specialized heads:
NP-HEAD for nuclei segmentation against the background, the HV-HEAD for predicting horizontal and vertical orientation maps, and the NC-HEAD for nuclei classification.
This allows for arobust comparison of external datasets
by performing instance segmentation with a watershed algorithm using outputs from these heads. -
Optimized training process using a combined loss function:
Ltotal = LNP + LHV + LNT + LTC (1),
which sums losses for nuclei prediction, orientation maps, nucleus types, and tissue classification. This allows for fine-grained control over different aspects of the segmentation and classification tasks simultaneously. -
Robust performance across diverse pathological data by evaluating NuLite on external datasets like CoNseP, MoNuSeg, and GlySAC. The paper demonstrates that
NuLite consistently achieved SOTA results in these tests,
proving itsrobustness and generalizability across different types of histopathological images.
-
Optimized model selection strategy using a tiered approach based on performance metrics. The authors selected NuLite versions by grouping them by GFLOPS and choosing the best performer in each group, leading to the selection of
FastViT-S12 as the backbone for NuLite-T(iny), FastViT-SA24 as the backbone of NuList-M(medium), and FastViT-SA36 for NuLite-H(uge).
-
Improved inference speed by utilizing a reparameterized version of the Vision Transformer encoder. The paper shows that
reparameterization does not significantly affect performance
but results insuperior computational efficiency,
leading to NuLite models beingup to about 13 times faster than CellViT.
-
Multi-scale analysis capability by comparing performance across different input patch sizes, specifically
256 × 256 pixels with 64-pixel overlap vs. 1024 × 1024 pixels.
This demonstrates thatusing larger patches (1024 × 1024 pixels) does not negatively impact the performance and may even slightly improve it in some cases.
-
Versatile analysis across complex tissue types by achieving superior results in specific categories on external datasets. For instance,
NuLite-M and NuLite-H show notable strengths in specific categories,
with NuLite-H showingsuperior recall and F1 scores
for the CoNSeP dataset, indicating its ability to handlecomplex tissue classification tasks.
-
Streamlined Whole Slide Imaging (WSI) analysis capability. The research suggests that the ability to effectively use larger patches (1024 × 1024 pixels) could
significantly streamline the process of analyzing large WSI data, reducing the need for extensive patch overlap and accelerating the segmentation process.
Abstract
In pathology, accurate and efficient analysis of Hematoxylin and Eosin (H&E) slides is crucial for timely and effective cancer diagnosis. Although many deep learning solutions for nuclei instance segmentation and classification exist in the literature, they often entail high computational costs and resource requirements, thus limiting their practical usage in medical applications. To address this issue, we introduce a novel convolutional neural network, NuLite, a U-Net-like architecture designed explicitly on Fast-ViT, a state-of-the-art (SOTA) lightweight CNN. We obtained three versions of our model, NuLite-S, NuLite-M, and NuLite-H, trained on the PanNuke dataset. The experimental results prove that our models equal CellViT (SOTA) in terms of panoptic quality and detection. However, our lightest model, NuLite-S, is 40 times smaller in terms of parameters and about 8 times smaller in terms of GFlops, while our heaviest model is 17 times smaller in terms of parameters and about 7 times smaller in terms of GFlops. Moreover, our model is up to about 8 times faster than CellViT. Lastly, to prove the effectiveness of our solution, we provide a robust comparison of external datasets, namely CoNseP, MoNuSeg, and GlySAC. Our model is publicly available at https://github.com/CosmoIknosLab/NuLite
Sources
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PanNuke Dataset Extension, Insights and Baselines
- HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation
- TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Related papers
- HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- FDIR: Fidelity-Anchored Decompressed Image Restoration for Human and Machine Perception
- A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer
- CADS: A Comprehensive Anatomical Dataset and Segmentation for Whole-Body Anatomy in Computed Tomography
- Task-Aware Bitrate Thresholds for Codec-Robust Ultrasound Segmentation