Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift".

Dev: Here's the gist of the paper. Deep learning models can be calibrated to provide more reliable probability estimates and uncertainty estimations for thyroid nodule classification, which is important when automated image-based triage suggestions may be unreliable, especially under dataset shift The proposed calibrated deep-ensemble framework demonstrated strong internal discrimination and calibration on TN5000 and supported conservative selective image-based triage using ensemble disagreement However, external evaluation on TN3K showed reduced discrimination, degraded calibration, and limited transportability of the TN5000-derived operating thresholds under dataset shift Framework Development The framework combines a ConvNeXt-Tiny backbone with squeeze-and-excitation attention, five independently trained ensemble members, member-wise vector-scaling calibration, and mutual information as an ensemble-disagreement score for deferral The development workflow comprised lesion centred ROI preparation, preliminary backbone screening, fold specific deterministic ensemble training, member wise probability calibration, mutual information guided deferral, and strict probability based triage The selected architecture was ConvNeXt-Tiny after being evaluated using validation AUC ROC as the primary screening criterion A squeeze-and-excitation (SE) channel attention module was incorporated into the selected classification architecture to recalibrate the pooled channel representation before classification Calibration and Uncertainty Signals Probability calibration assesses whether predicted probabilities correspond to observed outcome frequencies and is particularly important when probabilities are used to define conservative rule-out or rule-in thresholds Uncertainty estimation provides an additional signal for identifying predictions that should not be treated as stand alone recommendations Deep ensembles estimate prediction reliability through disagreement among independently trained models Mutual information was selected as the primary deferral score because the central study question was whether disagreement among independently trained deep learning models could identify thyroid nodule images for which autonomous image based triage should be withheld Triage Policy and Performance The strict policy implemented two explicit deferral conditions: an MI gate for ensemble disagreement and a probability gray zone for cases that did not satisfy the rule-out or rule-in criteria The triage policy comprised two sequential decision stages: first, images with MI above a fold specific threshold were assigned to radiologist review; second, among images that passed the MI gate, calibrated ensemble probability was used for rule out and rule in assignment Increasing nominal MI retention reduced the mean radiologist review rate primarily by increasing the FNA recommendation rate External Domain Shift Evaluation The frozen five member deployment ensemble was applied unchanged to the independent TN3K cohort to assess discrimination, calibration, uncertainty behavior, and threshold transportability under dataset shift On TN3K, the model achieved an AUC ROC of 0.7870 and an AP of 0.7254 External calibration was poorer than in the TN5000 OOF analysis, with ECE of 0.1899 and Brier score of 0.2281 The fixed policy preserved malignancy capture primarily through extensive external deferral rather than by preserving the internal autonomous triage pattern Limitations and Future Directions The study did not evaluate prospective clinical utility or measure changes in radiologist workload or biopsy rates The present results do not support direct clinical deployment or claims of actual biopsy reduction Future work should evaluate the framework using multi institutional patient level datasets with complete linkage between images, nodules, and clinical outcomes The No-FNA assignments should not be interpreted as avoided biopsies because the model did not incorporate nodule size, TI RADS features, clinical history, laboratory findings, or radiologist assessment Key Findings on Transportability The external TN3K findings show that both uncertainty and probability thresholds require target population validation The absence of malignancies in the No-FNA group occurred alongside an 83.7% radiologist review rate These results do not demonstrate preservation of the TN5000 derived predictive value targets on TN3K The low No-FNA suggestion rate, high review burden, and reduced FNA recommendation PPV indicate limited transportability of the fixed TN5000 derived operating policy to TN3K Uncertainty Score Comparison In the internal comparison, MSP derived uncertainty achieved AURC and AUGRC values of 0.03020 and 0.02261, followed by predictive variance at 0.03234 and 0.02424 and MI at 0.03778 and Selective Prediction Evaluation The full resolution pooled OOF AURC was 0.037778 and the AUGRC was 0.15073 Lower selective risk indicates fewer classification errors among retained images relative to random selection The study evaluates whether calibrated ensemble disagreement can support a conservative selective prediction framework in which automated image based suggestions are issued only for low disagreement cases This study presents a calibrated deterministic deep ensemble framework for selective annotation assisted thyroid nodule ROI classification The strict MI guided policy demonstrated that conservative No-FNA suggestions can be limited to low MI low probability cases while higher MI and probability gray zone cases are referred for radiologist review These results support the feasibility of using ensemble disagreement as a hypothesis driven deferral signal within a retrospective image based triage framework The No-FNA assignments should not be interpreted as avoided biopsies because the model did not incorporate nodule size, TI RADS features, clinical history, laboratory findings, or radiologist assessment References

1: B.

Rosa: R.

Dev: Haugen et al., “2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer,” Thyroid, vol.

Rosa: 26, no.

Paper summary: Rosa: So, we're looking at this paper titled "Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation". The main idea here is that just having a deep learning model that can classify nodules well isn't enough for real clinical use.

Dev: Right, because the authors are focusing on probability calibration and uncertainty estimation when those automated triage suggestions might not be reliable, especially when the data changes between different sets. It's important for making these tools useful in a real clinic one <ref:2607.12075#pg1>.

Taro: What does that mean practically? It suggests we need more than just a classification score; we need to know how sure the model is about its guess and whether that certainty holds up when we move to new patient data.

Rosa: Exactly. The paper claims they developed a calibrated deterministic five-member deep ensemble framework for classifying thyroid nodule regions of interest and doing selective triage <ref:2607.12075#pg2>. They use TN5000 for building the models and internal validation, and then they test it on an independent set called TN3K to see how it does when the data shifts <ref:2607.12075#pg2>.

Dev: And what's the core mechanism they put into this framework? They combine a ConvNeXt-Tiny backbone with squeeze-and-excitation attention, five independently trained ensemble members, member-wise vector-scaling calibration, and mutual information as a score for when to defer <ref:2607.12075#pg2>.

Taro: Mutual information as an operational measure of ensemble disagreement sounds like a way to quantify the uncertainty among the different models, which is useful for deciding when to stop and ask a human expert <ref:2607.12075#pg2>.

Rosa: They set up a strict threetier retrospective policy: first, an image gets an image-based No-FNA suggestion based on that mutual information, then they have an FNA recommendation pathway for cases that pass that initial gate, and finally, if it doesn't fit those two categories, it goes to a radiologist <ref:2607.12075#pg2>.

Dev: That three-tier policy sounds like a way to manage the risk of misclassification by using the ensemble disagreement as a primary signal for review, which is interesting from a real-time control loop standpoint <ref:2607.12075#pg2>.

Taro: So if we think about how this system should handle bad input, what happens when the data shifts, like moving from TN5000 to TN3K? The paper explicitly tests that transportability <ref:2607.12075#pg2>.

Rosa: And it shows that while it did well internally on TN5000, when they moved to the external dataset of TN3K, the discrimination got reduced and the calibration degraded <ref:2607.12075#pg2>.

Paper summary: Dev: The external evaluation on TN3K showed an AUC ROC of zero point seven eight seven and an AP of zero point seven two five four, which is decent but shows a drop compared to what they saw internally on TN5000 <ref:2607.12075#pg2>.

Taro: And the external calibration was poorer than in the internal TN5000 out-of-fold analysis, with an ECE of zero point one eight nine nine and a Brier score of zero point two two eight one <ref:2607.12075#pg2>. That means their predictions are less reliable when the underlying data changes, which is a big deal for safety <ref:2607.12075#pg3>.

Rosa: The paper points out that the fixed policy didn't really preserve the internal predictive value targets from TN5000 on TN3K <ref:2607.12075#pg2>. The low No-FNA suggestion rate, combined with a high review burden, shows that the fixed policy isn't transporting well <ref:2607.12075#pg3>.

Dev: It confirms what we suspected: both uncertainty and probability thresholds need target population validation before you can trust them in deployment <ref:2607.12075#pg3>. This suggests that whatever works for one data set isn't guaranteed to work for another <ref:2607.12075#pg3>.

Taro: So, what does this mean for a system that needs to be robust? It means the uncertainty score and the probability thresholds are not universal; they are tied to the specific distribution of the data you're looking at <ref:2607.12075#pg3>.

Rosa: And it ties into their uncertainty comparison results too, where they saw that MSP derived uncertainty achieved an AUROC of zero point zero three zero two and an AUGRC of zero point one five zero seven when evaluating selective prediction <ref:2607.12075#pg4>. This lower selective risk indicates fewer classification errors among the retained images compared to random selection <ref:2607.12075#pg4>.

Dev: But we have to remember the strict policy they implemented, which assigns No-FNA suggestions only for cases with low mutual information and low probability gray zone cases <ref:2607.12075#pg4>. That selective prediction framework is what they are trying to support <ref:2607.12075#pg4>.

Taro: The limitation they flag, which I think is the most important for us, is that the No-FNA assignments shouldn't be interpreted as avoided biopsies because the model didn't incorporate nodule size or TI RADS features or clinical history <ref:2607.12075#pg3>. That context matters beyond just the image classification <ref:2607.12075#pg3>.

Rosa: So, in a nutshell, this paper presents a calibrated deterministic deep ensemble framework for selective annotation assistance based on thyroid nodule ROI classification <ref:2607.12075#pg1>. The strict mutual information guided policy suggests that conservative No-FNA suggestions should only happen in low mutual information and low probability cases <ref:2607.12075#pg4>.

Dev: It supports the feasibility of using ensemble disagreement as a hypothesis driven deferral signal within a retrospective triage framework <ref:2607.12075#pg1>. This is useful for setting up initial screening filters before handing things off to a radiologist <ref:2607.12075#pg4>.

Paper summary: Taro: It means that for any real-world application, you can't just deploy one set of thresholds and expect it to work everywhere because the uncertainty and calibration are highly dependent on the target population <ref:2607.12075#pg3>. This is a key piece of context for building robust autonomy <ref:2607.12075#pg3>.

Rosa: So, the paper shows that for image-based triage suggestions to be reliable, we need to calibrate those predictions and use ensemble disagreement in a conservative way when data shifts occur <ref:2607.12075#pg2>. We need to be careful about what we claim these suggestions actually mean clinically <ref:2607.12075#pg3>.

Dev: And the implication for deployment is that if you want to move forward, you have to validate those uncertainty and probability thresholds against the specific population you expect to serve, like TN3K versus TN5000 <ref:2607.12075#pg3>. That external validation step is crucial for any safety case <ref:2607.12075#pg3>.

Taro: And for me, it means when we test failure modes, we have to account for the drift in performance and calibration as the environment changes <ref:2607.12075#pg3>. Graceful recovery means knowing *when* to stop relying on a prediction and escalate it <ref:2607.12075#pg3>.

Rosa: So, we've seen how this framework uses calibration and disagreement to create a selective prediction system for thyroid nodules <ref:2607.12075#pg1>. It gives us a way to guide human review when the AI is uncertain or when the input data is different from what it was trained on <ref:2607.12075#pg3>.

Dev: The main thing here is that this work shows how ensemble disagreement can serve as a signal for deferral in an image-based triage framework, but it must be paired with careful consideration of calibration and uncertainty signals <ref:2607.12075#pg1>.

Taro: It's not a guarantee of performance in new domains, so the next step has to be rigorous testing across those shifts <ref:2607.12075#pg3>. We need to know how the system behaves when it encounters data it hasn't fully seen before <ref:2607.12075#pg3>.

Rosa: That's the path forward then, focusing on that external validation and understanding where those uncertainty thresholds break down <ref:2607.12075#pg3>. We keep digging into how these models handle real-world variability beyond the lab setting <ref:2607.12075#pg3>.

Dev: Right, so the focus shifts from just high accuracy to building a system that is robust and calibrated across different data distributions <ref:2607.12075#pg3>. That's where the engineering challenge lies <ref:2607.12075#pg3>.

Taro: Exactly. We need to design systems that can gracefully recover when they sense the environment has changed, rather than just failing silently or giving a confident wrong answer <ref:2607.12075#pg3>.

Rosa: So, this paper gives us a calibrated deterministic way to support image-based triage while acknowledging the limits of its prediction in new data environments <ref:2607.12075#pg3>. That's the gist of what they are presenting <ref:2607.12075#pg1>.

Conclusion: Rosa: So we’re wrapping up this look at this paper, "Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift."

Dev: The authors are focusing on how deep ensembles can give us more reliable probability estimates and uncertainty signals when we classify thyroid nodules.

Taro: It seems like they built a system that doesn't just give you a yes or no answer, but also tells you how confident the model is, which is important for safety.

Rosa: Exactly. They developed this framework to use ensemble disagreement as a signal for when to pause and get human review instead of an automated suggestion.

Dev: And they did this by combining several things—a ConvNeXt-Tiny backbone, five different ensemble members, and member-wise vector-scaling calibration.

Taro: The big thing is how they test it under dataset shift—moving from one set of data to a different one, like TN5000 to TN3K.

Rosa: And the results show that while it worked well internally on the first set, its performance drops when the data changes because the uncertainty and calibration degrade.

Dev: They found that for external evaluation, like on TN3K, their discrimination and calibration got worse compared to what they saw in their initial internal tests.

Taro: The key finding there is that you can't just use a fixed set of thresholds; both the probability thresholds and the uncertainty signals need to be validated for any specific population.

Rosa: That means the system isn't automatically robust across different clinical scenarios, and we have to be careful about applying those rules blindly.

Dev: The authors stress that these No-FNA suggestions shouldn't be seen as avoided biopsies because the model didn't account for things like nodule size or clinical history.

Taro: So, while this framework shows a way to use ensemble disagreement to guide triage, it highlights how critical it is to rigorously test those uncertainty signals when data shifts occur.

Rosa: That leads us right into where we need to think about real-world deployment and how we can build systems that handle that kind of environmental change gracefully.