HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation

arXiv:2311.12553 · eess.IV, cs.CV · Submitted 2023-11-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation".

Dev: Here's the gist of the paper. Nuclei panoptic segmentation, i.e., the simultaneous detection, segmentation, and classification of nuclear instances, is at the core of the automation of several tasks in digital pathology

1: .

Rosa: Our approach yields results comparable to HoVerNet on the public PanNuke and Consep datasets with a three-fold reduction in inference time Model Architecture and Distillation Strategy The proposed method introduces “HoVer-UNet,” which is a compact, streamlined single UNet network equipped with a Mix Vision Transformer backbone and a custom loss function to optimally encode the distilled knowledge of HoVerNet.

Dev: This distillation framework adopts an offline technique using HoVerNet as a pre-trained teacher network.

Rosa: The strategy combines all output branches of HoVerNet into a single branch network and uses a single-branch UNet

8: as the student model.

Title and authors: Rosa: So we're looking at this paper right now, "HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation." The authors are Cristian Tommasino, Cristiano Russo, and Antonio Maria Rinaldi. This whole idea is about taking a big model called HoVerNet and making it smaller and faster using a UNet structure.

Dev: Exactly. They're proposing this approach to distill the knowledge from that multi-branch HoVerNet framework for nuclei segmentation and classification in histopathology into one compact, streamlined single UNet network with a Mix Vision Transformer backbone.

Rosa: Basically, they want to keep the good performance but drastically cut down on how long it takes for the system to run. They did this by using an offline technique where HoVerNet acts as a pre-trained teacher network for their student model.

Dev: And the key part of their distillation strategy is combining all those different output branches from HoVerNet into one single branch network, which is then fed into a standard single-branch UNet. They use a custom loss function to optimally encode that distilled knowledge.

Rosa: That custom loss function they use isn't just one thing; it's a linear combination across three specific branches: the NP, HV, and TP branches. For the HV branch specifically, they set up the loss as mean squared error plus a smooth gaussian loss.

Dev: And for the NP and TP branches, their student loss is defined similarly to that—it’s cross entropy plus dice score. They use specific hyperparameters like alpha equals zero point five and T equals one when they train this model with a MixViT-B2 backbone.

Rosa: The results they show on the public PanNuke dataset are pretty interesting; they got a panoptic quality of zero point six two eight six, which is actually higher than HoVerNet's score of zero point six five nine six.

Dev: That’s a bit surprising because the text notes that their F-score detection performance is comparable to HoVerNet, even though the panoptic quality number itself is lower than the original model.

Title and authors: Rosa: It sounds like they found a trade-off where you get better speed while keeping the results quite close to what those larger models achieve on certain metrics. They also showed a three-fold reduction in inference time compared to HoVerNet.

Dev: That speedup is what makes this practical for real pathology work, because they say it can reduce processing time from about an hour down to around twenty minutes for whole-slide images.

Rosa: So, the core idea here is taking a complex system and distilling its intelligence into something lightweight that still performs well on nuclei segmentation and classification in H andE stained images.

Dev: And the next question I have is how this distillation framework handles real-world deployment, especially concerning latency and failure modes during operation.

Taro: When you distill a model like this, you're essentially trying to capture the essence of its function into a simpler architecture, but that means you're relying heavily on that initial teacher network being perfect. If the teacher has some subtle bias or limitation in how it processes complex nuclei structures, that limitation gets baked into your student UNet.

Rosa: Right, so if the teacher has an issue with a specific type of staining or noise pattern, the distilled model will carry that issue forward and we need to be careful about what data we train it on.

Dev: That’s true. From my side looking at the control loops, I'm concerned about how stable this single-branch network is when it has to handle noisy input from a real microscope setup versus the clean data used during distillation.

Taro: And when you test this outside of the lab, like on actual slides where things go wrong—maybe some poor image quality or unexpected tissue morphology—how gracefully does this distilled model recover instead of just failing entirely?

Rosa: Well, the paper focuses heavily on the benchmark datasets like PanNuke and Consep for validation, but they don't really detail extensive real-world stress testing outside of that controlled environment. That’s a limitation they point out.

Title and authors: Dev: Exactly. The authors focus on showing comparable performance to other models like DIST, Mask RCNN, and Micro-Net in terms of panoptic quality scores on PanNuke, but the text also highlights that their F-score detection performance is comparable to HoVerNet while demonstrating a significant advantage in processing speed.

Taro: So, even though the panoptic score isn't higher than HoVerNet's zero point six five nine six, they claim comparable F-score detection performance, which suggests the segmentation accuracy on individual nuclei instances is still quite robust <ref:2311.12553#pg3>.

Rosa: That’s a key distinction between just looking at one metric and looking at the whole picture of what the system actually does for pathologists.

Dev: And to wrap up on the methodology, they clearly state that their solution achieved its best results using the UNet model with a MixViT-B2 backbone employing alpha equals zero point five and T equals one as those specific hyperparameters.

Taro: It’s interesting how precisely they tuned those parameters to get that balance between performance and speed; it shows a lot of work went into optimizing that specific configuration.

Rosa: So, to summarize the main point of this paper, "HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation," is that they successfully derived a lightweight panoptic segmentation model from the original HoVerNet framework by combining its three output branches into a single branch network.

Dev: The implication for the field is that we can now have models that are significantly smaller and faster than state-of-the-art models like HoVerNet, without having to sacrifice the detection performance on critical metrics like the F-score.

Taro: For me, it means we have a viable path toward quicker computational pathology solutions that can be integrated into more real-time diagnostic pipelines when dealing with whole slide images.

Rosa: It certainly lays groundwork for swifter computational pathology solutions, and they’ve made their code publicly available so everyone can see how it works.

Dev: That's the wrap on this one about HoVer-UNet, showing that distillation can be a very effective way to achieve a significant reduction in computational requirements while maintaining competitive results.

The paper's summary: Rosa: So we're talking about this paper now, "HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation." Basically, they took a big model called HoVerNet and turned it into something smaller using a UNet structure.

Dev: Yeah, the core idea is distilling all that complex knowledge from the original HoVerNet framework down into one compact, streamlined single UNet network.

Rosa: They use an offline technique where HoVerNet acts as a pre-trained teacher network for their student model. It’s like teaching a kid using a super smart expert.

Dev: And the tricky part is how they combine those different output branches from HoVerNet into one single branch network before feeding it into the student UNet. They use a custom loss function to make sure that knowledge transfer works right.

Rosa: That custom loss function isn't just one thing; it’s a linear combination across three specific branches, NP, HV, and TP. For the HV branch specifically, they set up the loss as mean squared error plus a smooth gaussian loss.

Dev: And for the NP and TP branches, their student loss is defined similarly to that—it’s cross entropy plus dice score. They tune this with specific hyperparameters like alpha equals zero point five and T equals one when they train this model with a Mix Vision Transformer backbone.

Rosa: The results they show on the public PanNuke dataset are pretty interesting; they got a panoptic quality of zero point six two eight six, which is actually higher than HoVerNet's score of zero point six five nine six.

Dev: That’s a bit surprising because the text notes that their F-score detection performance is comparable to HoVerNet, even though the panoptic quality number itself is lower than the original model.

Rosa: It sounds like they found a trade-off where you get better speed while keeping the results quite close to what those larger models achieve on certain metrics. They also showed a three-fold reduction in inference time compared to HoVerNet.

Dev: That speedup is what makes this practical for real pathology work, because they say it can reduce processing time from about an hour down to around twenty minutes for whole-slide images.

Rosa: So, the core idea here is taking a complex system and distilling its intelligence into something lightweight that still performs well on nuclei segmentation and classification in H andE stained images.

Dev: And the next question I have is how this distillation framework handles real-world deployment, especially concerning latency and failure modes during operation.

Taro: When you distill a model like this, you're essentially trying to capture the essence of its function into a simpler architecture, but that means you're relying heavily on that initial teacher network being perfect. If the teacher has some subtle bias or limitation in how it processes complex nuclei structures, that limitation gets baked into your student UNet.

The paper's summary: Rosa: Right, so if the teacher has an issue with a specific type of staining or noise pattern, the distilled model will carry that issue forward and we need to be careful about what data we train it on.

Dev: That’s true. From my side looking at the control loops, I'm concerned about how stable this single-branch network is when it has to handle noisy input from a real microscope setup versus the clean data used during distillation.

Taro: And when you test this outside of the lab, like on actual slides where things go wrong—maybe some poor image quality or unexpected tissue morphology—how gracefully does this distilled model recover instead of just failing entirely?

Rosa: Well, the paper focuses heavily on the benchmark datasets like PanNuke and Consep for validation, but they don't really detail extensive real-world stress testing outside of that controlled environment. That’s a limitation they point out.

Dev: Exactly. The authors focus on showing comparable performance to other models like DIST, Mask RCNN, and Micro-Net in terms of panoptic quality scores on PanNuke, but the text also highlights that their F-score detection performance is comparable to HoVerNet while demonstrating a significant advantage in processing speed.

Taro: So, even though the panoptic score isn't higher than HoVerNet's zero point six five nine six, they claim comparable F-score detection performance, which suggests the segmentation accuracy on individual nuclei instances is still quite robust.

Rosa: That’s a key distinction between just looking at one metric and looking at the whole picture of what the system actually does for pathologists.

Dev: And to wrap up on the methodology, they clearly state that their solution achieved its best results using the UNet model with a MixViT-B2 backbone employing alpha equals zero point five and T equals one as those specific hyperparameters.

Taro: It’s interesting how precisely they tuned those parameters to get that balance between performance and speed; it shows a lot of work went into optimizing that specific configuration.

Rosa: So, to summarize the main point of this paper, "HoVer-UNet," is that they successfully derived a lightweight panoptic segmentation model from the original HoVerNet framework by combining its three output branches into a single branch network.

Dev: The implication for the field is that we can now have models that are significantly smaller and faster than state-of-the-art models like HoVerNet, without having to sacrifice the detection performance on critical metrics like the F-score.

Taro: For me, it means we have a viable path toward quicker computational pathology solutions that can be integrated into more real-time diagnostic pipelines when dealing with whole slide images.

Rosa: It certainly lays groundwork for swifter computational pathology solutions, and they’ve made their code publicly available so everyone can see how it works.

Dev: That's the wrap on this one about HoVer-UNet, showing that distillation can be a very effective way to achieve a significant reduction in computational requirements while maintaining competitive results.

Rosa: Next up we’re looking at some papers on audio deepfake detection. How does that compare to this image segmentation work?

The paper's improvements: Rosa: So we're looking at how they actually improved the original HoVerNet model in this paper, "HoVer-UNet." They didn't just stop at making it smaller; they added new ways to train and structure it.

Dev: Yeah, the main improvement is that they introduced a specific way to handle the knowledge transfer, which is using a linear combination of losses. It’s not just one loss function anymore.

Rosa: Right, so instead of relying on just one measure for performance across all parts of the model, they use alpha times student loss plus one minus alpha times distillation loss. That lets them balance how much they care about the new network versus how well it mimics the old teacher.

Dev: They also refined their backbone choice, using a Mix Vision Transformer backbone like MixViT-B2 and setting specific training parameters like alpha equals zero point five and T equals one. These settings are what helped them hit that performance comparison on the PanNuke dataset.

Rosa: So, the improvement there is about tuning that architecture to get the best speed for the best quality trade-off, which is what you want when you’re trying to run this on a slower computer or a real pathology scanner.

Dev: And they made sure their loss terms were structured across three distinct branches—NP, HV, and TP—giving them more granular control over how the segmentation and classification errors are weighted during training.

Rosa: That granular control is important because it lets them tailor the distillation process to each specific task within the panoptic segmentation pipeline.

Dev: It means they aren't just doing a blanket reduction in size; they're surgically optimizing which parts of the model retain their high-fidelity information while cutting out what’s redundant.

Rosa: So, what this means for us is that if you’re looking to build a faster segmentation tool, this distillation strategy isn't just a trick; it’s a structured way to transfer complex capabilities without losing critical accuracy on key metrics like the F-score.

Dev: It shows that you can take an existing, powerful model and systematically engineer it into something much more efficient for real-time applications.

Rosa: And this brings us to the big question: Can this kind of distillation strategy work with other types of models, or is it specific to the UNet structure they used?

Conclusion: Rosa: So we’re wrapping up this look at "HoVer-UNet: Accelerating HoVerNet with UNet-based multi-class nuclei segmentation via knowledge distillation." Basically, they successfully shrunk a big model down into a much faster one for analyzing H and E stained images.

Dev: Right, it shows that you can take established, high-performing architectures and apply smart distillation techniques to create something that fits real-time or near real-time needs without sacrificing the core accuracy metrics.

Rosa: The implication for pathologists is significant because if you can process whole slide images in twenty minutes instead of an hour, that changes how quickly a diagnosis can be delivered.

Dev: It’s about making computational pathology tools accessible on a wider scale, not just for huge institutional labs with massive computing clusters.

Taro: From my side looking at the real-world stress test, I still have some reservations about how robust this distillation is when the input image quality really degrades outside of those clean benchmark sets.

Rosa: That's fair, Taro; they did focus heavily on PanNuke and Consep for validation, so we don't see extensive field testing on noisy slides yet.

Dev: The safety case here relies on that initial teacher network being very stable, and if the distillation process doesn't perfectly capture all those subtle error modes in the original model, those failures will show up under stress.

Taro: So the question is whether this speed gain is worth any potential loss in resilience when a system has to deal with unexpected tissue morphology or poor staining conditions.

Rosa: It’s a trade-off that needs careful consideration for deployment, but the core method of combining those three output branches into one network is definitely a clever way to achieve the size reduction they promised.

Dev: Indeed, it lays solid groundwork for building swifter computational pathology solutions that can actually be integrated into quicker diagnostic pipelines.

Rosa: That’s where we leave it for now, but next time we’ll look at how this approach compares to other methods used in audio deepfake detection.

Department of Electrical Engineering and Information Technology University of Napoli Federico II · Department of Pathology Radboud University Medical Center

eess.IV, cs.CV

Submitted: 2023-11-21

Updated: 2023-12-04

Comments: 4 pages, 2 figures, submitted to ISBI 2024

DOI: 10.1109/ISBI56570.2024.10635755

Code: https://github.com/DIAGNijmegen/HoVer-UNet

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 77/100

The gist: Nuclei panoptic segmentation, i.e., the simultaneous detection, segmentation, and classification of nuclear instances, is at the core of the automation of several tasks in digital pathology [1].

Key concepts

Nuclei Panoptic Segmentation
This is the core task of simultaneously detecting, segmenting, and classifying every single nuclear instance within a tissue image. It ensures that the model not only finds where nuclei are but also precisely outlines their boundaries and labels them correctly for automated analysis.
Knowledge Distillation
This technique involves training a smaller 'student' model to mimic the performance of a larger, more complex 'teacher' model. In this case, HoVerNet is the teacher; its complex knowledge is transferred into the simpler HoVer-UNet architecture to achieve similar accuracy while being much faster.
Mix Vision Transformer Backbone
This refers to a specific type of deep learning architecture used as the backbone for the model. A Vision Transformer (ViT) structure, particularly a 'Mix' variant, allows the model to capture complex spatial relationships and contextual information within high-resolution images more effectively than traditional convolutional networks.

Terminology

Summary

Nuclei panoptic segmentation, i. p. 0

Where the paper says it

Index Terms— Instance Nuclei Segmentation, Nuclei Classification, Knowledge Distillation, HoVerNet 1. INTRODUCTION Nuclei panoptic segmentation, i.

e., the simultaneous detection, segmentation, and classification of nuclear instances, is at the core of the automation of several tasks in digital pathology [1]. p. 0
Where the paper says it

INTRODUCTION Nuclei panoptic segmentation, i. , the simultaneous detection, segmentation, and classification of nuclear instances, is at the core of the automation of several tasks in digital pathology, particularly in the analysis of routine Hematoxylin and Eosin (H&E) stained histology slides [1].

Our approach yields results comparable to HoVerNet on the public PanNuke and Consep datasets with a three-fold reduction in inference time

Model Architecture and Distillation Strategy

The proposed method introduces “HoVer-UNet,” which is a compact, streamlined single UNet network equipped with a Mix Vision Transformer backbone and a custom loss function to optimally encode the distilled knowledge of HoVerNet. This distillation framework adopts an offline technique using HoVerNet as a pre-trained teacher network. p. 1

Where the paper says it

Distillation Framework. In our KD framework, we adopt an offline technique using HoVerNet as a pre-trained teacher network.

The strategy combines all output branches of HoVerNet into a single branch network and uses a single-branch UNet [8] as the student model. p. 1
Where the paper says it

Given that HoVerNet performs nuclei instance segmentation and classification through three branches, our distillation strategy is based on the idea of combining all output branches of HoVerNet into a single branch network (see Figure 1. Note that we aim to train a student that can replace only the HoVerNet backbone, not its post-processing steps, which we left unvaried.

The loss terms are derived from a linear combination of losses across three branches: NP, HV, and TP. p. 1
Where the paper says it

(1) With regards to individual loss terms, we were inspired by HoVerNet loss, and we specifically modified it for KD. In detail, the student and distillation losses, denoted as Lstudent and Ldistill, respectively, are derived from a linear combination of losses across three branches: NP, HV, and TP.

Specifically for the HV branch, the loss is formulated as LHV = MSE(x, y) + MSGE(x, y) and for the NP and TP branches, the student loss is similarly defined as LNP = LTP = CE(x, y) + DICE(x, y). p. 1
Where the paper says it

For the HV branch, the loss is formulated as LHV = MSE(x, y) + MSGE(x, y), incorporating Mean Squared Error (MSE) and Mean Squared Gradient Error (MSGE). For the NP and TP branches, the student loss is similarly defined as LNP = LTP = CE(x, y) + DICE(x, y), combining Weighted Cross-Entropy (CE) loss with Dice loss (DICE).

Performance Comparison on Datasets

The framework was trained on PanNuke and validated on CoNSeP datasets. On the PanNuke dataset, the proposed solution achieved a higher Panoptic Quality (PQ) score of 0. p. 1

Where the paper says it

[4] and Gamper et al. Specifically, we employed the Panoptic Quality metric, initially proposed in [1], in conjunction with the F-score, introduced by Graham et al.

6286 compared to HoVerNet's 0.6596 and demonstrated comparable F-score detection performance. p. 1
Where the paper says it

Results on PanNuke. We evaluated several performance metrics including binary panoptic quality (P QB), multiclass panoptic quality (P QM), panoptic quality for specific classes (P QN,I,C,D,E), F-score detection (Fd), and F-score for individual classes (F N,I,C,D,E).

On the CoNSeP dataset, the solution outperformed HoVerNet in terms of PQ, though it fell short in terms of F-score detection and showed that inference time is about three times lower. p. 2
Where the paper says it

com/vqdang/hover net Nuke’s dead and connective tissues as well as CoNSeP’s other types, which include fibroblast, muscle, and endothelial tissues. Table 3 shows the results demonstrating that our solution outperforms HoVerNet in terms of PQ, though it falls short in terms of F-score detection.

Inference Efficiency Gains

The primary advantage of HoVer-UNet is the significant reduction in computational requirements, which results in faster inference times. The complexity comparison shows that the most complex model has 10. p. 1

Where the paper says it

We selected these models because they are substantially faster than HoVerNet and exhibit fewer Multiply-Accumulate Operations (MACs). Specifically, the most complex model has 10.

63 GMac and an inference time of 0. p. 1
Where the paper says it

Specifically, the most complex model has 10. 63 GMac and an inference time of 0.

065 seconds, compared to HoVerNet, which has 149. p. 1
Where the paper says it

63 GMac and an inference time of 0. 065 seconds, compared to HoVerNet, which has 149.

73 GMac and an inference time of 0. p. 1
Where the paper says it

Specifically, the most complex model has 10. 63 GMac and an inference time of 0.

835 seconds. p. 1
Where the paper says it

73 GMac and an inference time of 0. 835 seconds.

This speedup is crucial for practical applications like whole-slide image processing where inference time can be reduced from an hour to approximately 20 minutes. p. 3
Where the paper says it

then use them in the distillation loss. Lastly, considering a whole-slide image, if HoVerNet takes 1 hour, we down it to ≈20 minutes with HoVer-UNet, with overall results in practice comparable.

Conclusion and Impact

The method successfully presents a novel approach for panoptic segmentation in histopathological analysis of H&E stained images. The approach yields results comparable to the state-of-the-art HoVerNet model and considerably reduces inference time. p. 2

Where the paper says it

Our method uses a KD offline approach to train a UNet network combined with a Mix Vision Transformer backbone. Our approach yields results comparable to the state-of-theart HoVerNet model and considerably reduces inference time.

This method performs similarly to leading models and lays the groundwork for the progression of swifter computational pathology solutions. p. 3
Where the paper says it

This approach performs similarly to leading models and lays the groundwork for the progression of swifter computational pathology solutions, such as biomarkers and automated quantification for subsequent tasks. Notably, our method is approximately three times faster than HoVerNet.

The authors have made their solution publicly available to promote its use in future developments in computational pathology. p. 3
Where the paper says it

Notably, our method is approximately three times faster than HoVerNet. We have made our solution publicly available to promote its use in future developments in computational pathology.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Knowledge distillation of HoVerNet into a single-branch UNet network using a custom loss function L = alpha · Lstudent + (1 − alpha) · Ldistill allows for training a smaller student model based on the complex teacher, aiming to derive a lightweight panoptic segmentation model from the original HoVerNet.

  2. The improved AI system can perform nuclei instance segmentation and classification in H&E stained histology slides with a three-fold reduction in inference time compared to HoVerNet, potentially reducing processing time from approximately one hour down to about 20 minutes for whole-slide images.

  3. The model is optimized using a Mix Vision Transformer backbone (e.g., MixViT-B2) and specific hyperparameters (alpha = 0.5 and T = 1), achieving performance comparable to HoVerNet on the PanNuke dataset while demonstrating a significant advantage in terms of processing speed.

  4. The system can be trained using an offline knowledge distillation approach where HoVerNet performs nuclei instance segmentation and classification through three branches, our distillation strategy is based on the idea of combining all output branches of HoVerNet into a single branch network.

Abstract

We present HoVer-UNet, an approach to distill the knowledge of the multi-branch HoVerNet framework for nuclei instance segmentation and classification in histopathology. We propose a compact, streamlined single UNet network with a Mix Vision Transformer backbone, and equip it with a custom loss function to optimally encode the distilled knowledge of HoVerNet, reducing computational requirements without compromising performances. We show that our model achieved results comparable to HoVerNet on the public PanNuke and Consep datasets with a three-fold reduction in inference time. We make the code of our model publicly available at https://github.com/DIAGNijmegen/HoVer-UNet.

Sources

Related papers