Selective Prediction Forgetting for Adaptation of Weakly Supervised Localization of Histology Images

arXiv:2603.12468 · cs.CV · Submitted 2026-03-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Selective Prediction Forgetting for Adaptation of Weakly Supervised Localization of Histology Images".

Jane: The gist The proposed method, SFDA-DeP,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The title itself, "Selective Prediction Forgetting for Adaptation of Weakly Supervised Localization of Histology Images," tells you exactly what they are doing: selecting which predictions to forget during adaptation to fix the domain shift issues in WSOL models.

Jane: It's quite technical, but the core idea is that they aren't just trying to adapt the model generally; they are specifically targeting and correcting prediction bias under those cross-domain shifts, which is a big step since those shifts are often really complex in histopathology >

Lu: The authors list several researchers from different universities, which suggests this work is coming from a broad collaboration tackling the problem from multiple angles, especially regarding representation learning and how features shift between domains >

Meng: It’s interesting to see the team structure here; it shows they are coming from different research backgrounds, which usually means they're looking at the problem in many ways, not just one narrow angle >

Lalam: I read their introduction, and it sets up the idea that when you have strong shifts, the model starts over-predicting some classes and under-representing others, leading to this highly skewed distribution in the target domain >

Tom: Exactly. So what they are proposing is a mechanism to actively reduce that skewed distribution by selectively forgetting predictions from those overpredicted classes during adaptation >

The paper's summary: Jane: So, what is the actual method they propose? The summary says they formulate this as an iterative process of identifying and correcting prediction bias, inspired by machine unlearning approaches >

Lu: It suggests that instead of just retraining everything from scratch or relying on simple self-training which just reinforces the initial bias, they periodically identify target images from overpredicted classes >

Meng: And for those specific images, they selectively reduce the predictive confidence for uncertain, high-entropy predictions while keeping the confident ones intact >

Tom: That sounds like a targeted way to steer the model away from those dominant class biases without throwing away all the useful information learned during training >

Lalam: The paper introduces a joint optimization of several loss terms. They have Lretain, which keeps predictions consistent with what was already predicted, and Lforget, which pushes the model to refrain from predicting that same dominant class for samples in a specific forget set >

Jane: And then there's the localization part, Lloc, which is trained separately using those pseudo-labels to anchor the localization features against the domain shift >

The paper's improvements: Tom: The key improvement seems to be this dynamic resampling of Br, which prevents early incorrect forgetting decisions from becoming irreversible by acting like an iterative correction mechanism >

Lu: That dynamic resampling is crucial because it means the model isn't just making one big decision about forgetting; it’s constantly refining its decision boundaries by reassignment >

Meng: From a deployment standpoint, that sounds robust. If we have to adapt this model on new data, we don't want one bad batch of adaptation steps to lock us into a biased state >

Jane: They also use a jointly optimized pixel-level classifier that is tuned using those class activation maps, or CAMs, which restores the localization features even under the domain shift >

Tom: So they have two main levers: first, this selective forgetting based on prediction confidence and uncertainty, and second, this joint optimization of the pixel-level head to anchor localization features >

Conclusion: Jane: To wrap up, the paper argues that this approach addresses the limitation where existing SFDA methods reinforce class asymmetries instead of correcting them >

Lu: They show that by using selective prediction forgetting, they can debias predictions for WSOL in histopathology under significant cross-domain shifts and then use a joint pixel-level head to restore localization features >

Meng: It's an improvement over previous SFDA baselines because it actively corrects the bias rather than just letting the model drift into an even worse biased state >

Lalam: The authors confirm that this method consistently outperforms state-of-the-art SFDA in both classification and localization tasks on benchmarks like PixelCAM and DeepMIL >

Tom: So, in a nutshell, Selective Prediction Forgetting for Adaptation of Weakly Supervised Localization of Histology Images proposes an iterative correction mechanism using dynamic resampling to reduce prediction bias and a joint pixel-level head to restore localization accuracy when moving between histology datasets >

Alexis Guichemerre, Banafsheh Karimian, Soufiane Belharbi, Natacha Gillet, Nicolas Thome, Pourya Shamsolmoali, Mohammadhadi Shateri, Luke McCaffrey, & Eric Granger

ÉTS Montréal Institute of Technology Montreal Canada University of York McGill University Sorbonne Université

cs.CV

Submitted: 2026-03-12

Updated: 2026-10-14

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: The gist The proposed method, SFDA-DeP, introduces a Source-Free Domain Adaptation (SFDA) approach that debiases predictions for Weakly Supervised Object Localization (WSOL) in histopathology by

Key concepts

Weakly Supervised Object Localization (WSOL): This technique allows models to perform two tasks simultaneously: classifying the entire image based on a single class label, and precisely locating specific objects within that image. In histology, this means identifying what kind of tissue is present and where it is located in the slide.
Prediction Bias: When adapting models to new data (like different organs or staining methods), the model often becomes overly confident in a few classes—the 'dominant' ones—and ignores others. This creates a skewed distribution where most target samples are incorrectly labeled as these dominant classes, which degrades performance.
SFDA-DeP Method: This is an iterative source-free domain adaptation approach that fixes the bias. It uses two sets: a 'retain set' for confident predictions and a 'forget set' for uncertain ones from dominant classes. By adjusting the loss functions, the model learns to reduce its reliance on these biased predictions while preserving accurate localization.
Pixel-Level Classifier Optimization: Instead of just adapting the main image classifier, this method also optimizes a separate classifier at the pixel level. This helps restore fine-grained visual cues necessary for accurate object localization, ensuring that even after domain shift adaptation, the model can precisely pinpoint where objects are.

Terminology

Adaptation — 2× in this paper · explained in 1 episode
Domain Shift — 2× in this paper · explained in 6 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 9 episodes
Domain Adaptation — 1× in this paper · explained in 5 episodes
Machine Unlearning — 1× in this paper · explained in 3 episodes
Weakly Supervised Object Localization — 1× in this paper

Summary

The gist The proposed method, SFDA-DeP, introduces a Source-Free Domain Adaptation (SFDA) approach that debiases predictions for Weakly Supervised Object Localization (WSOL) in histopathology by iteratively identifying and correcting prediction bias under significant cross-domain shifts.

Problem Statement

Weakly Supervised Object Localization (WSOL) models enable joint classification and region-of-interest localization in histology images using only image-class supervision<ref:2603.12468#pg2> When deployed in a target domain, distributions shift remains a major cause of performance degradation, especially when applied on new organs or institutions with different staining protocols and scanner characteristics<ref:2603.12468#pg2> Under stronger crossdomain shifts, WSOL predictions can become biased toward dominant classes, producing highly skewed pseudo-label distributions in the target domain<ref:2603.12468#pg2> We identify this amplification of prediction bias as a primary obstacle to the SFDA of WSOL models in histopathology<ref:2603.12468#pg2>.

Limitations of Existing SFDA Methods

Source-Free (Unsupervised) Domain Adaptation (SFDA) methods are commonly employed to address domain shift<ref:2603.12468#pg2> However, because they rely on self-training, the initial bias is reinforced over training iterations, degrading both classification and localization tasks<ref:2603.12468#pg2>. Most existing SFDA approaches rely on pseudo-labeling or clustering mechanisms<ref:2603.12468#pg4> These strategies implicitly assume that the source classifier provides sufficiently reliable predictions on the target domain to reliably separate target samples by predicted class<ref:2603.12468#pg4>. Under severe shift such as cross-organ and distributional shifts in histopathology, this assumption often breaks down<ref:2603.12468#pg4> The source model may become bias by over-predicting a subset of classes while underrepresenting others, leading to highly skewed distribution as in Fig. 1<ref:2603.12468#pg4>. When applied to biased predictions, SFDA performance is limited, as these methods tend to reinforce existing class asymmetries rather than correct them<ref:2603.12468#pg4>.

SFDA-DeP Method

SFDA-DeP formulates SFDA as an iterative process of identifying and correcting prediction bias<ref:2603.12468#pg6> It periodically identifies target images from overpredicted classes and selectively reduces the predictive confidence for uncertain (high entropy) images, while preserving confident predictions<ref:2603.12468#pg6>. This process reduces the drift of decision boundaries and bias toward dominant classes<ref:2603.12468#pg6>. A jointly optimized pixel-level classifier further restores discriminative localization features under distribution shift<ref:2603.12468#pg6>.

The total adaptation loss is formulated as L = λretainLretain + λforgetLforget + λlocLloc<ref:2603.12468#pg6>. The first two terms are used to steer the model from class prediction bias while maintaining reliable predictions<ref:2603.12468#pg6>, and the last term aims at anchoring localization to avoid drifting during adaptation<ref:2603.12468#pg6>.

The loss terms are defined as follows:

Lretain = Exi∈Br h − log pi(ˆy)i, where yˆ(x) = arg maxc pϕtrg (y = c x), and its minimization allows the model to continue predicting the same sample pseudo-label yˆ<ref:2603.12468#pg6>.
Lforget = Exi∈Bf h − log 1 − pi(ˆy)i, where Bf = topρ (B, H(x)), (1), and its minimization pushes the model to refrain from predicting the same dominant class yˆ for samples in the forget set<ref:2603.12468#pg6>.
Lloc = −(1 − Yp) log(h(zp)0) − Yp log(h(zp)1), p ∈ omega′, which is trained using those pseudo-labels using standard binary cross-entropy loss<ref:2603.12468#pg6>.

Experimental Validation

Extensive experiments on cross-organ and-center histopathology benchmarks (GlaS, CAMELYON16, CAMELYON17) with several WSOL models show that SFDA-DeP consistently improves classification and localization over stateof-the-art SFDA baselines<ref:2603.12468#pg6>. For PixelCAM, SFDA-DeP reaches 44.1 PxAP/ 67.1 CL on average, yielding significant improvements over SFDA-DE (+16.1 PxAP and +12.5 CL)<ref:2603.12468#pg8>. For DeepMIL, SFDA-DeP improves both tasks, reaching 40.7 PxAP and 73.4 CL on average (+20.2 PxAP and +19.5 CL vs. SFDA-DE)<ref:2603.12468#pg8>.

Key Findings

Ablations further indicate that dynamic resampling of Br and the pixel-level localization loss are key components of SFDA-DeP<ref:2603.12468#pg9>. Frequent reassignment prevents early incorrect forgetting decisions from becoming irreversible, acting as an iterative correction mechanism<ref:2603.12468#pg9>. Qualitative CAMs (Fig. 4) support this observation: SFDA-DeP yields more coherent activations concentrated on tumor tissue, with reduced responses on background regions, whereas SFDA baselines often highlight spurious areas under strong shift<ref:2603.12468#pg9>. SFDA-DeP consistently outperforms stateof-the-art SFDA in both classification and localization tasks<ref:2603.12468#pg6>.

Conclusion

SFDA-DeP is a SFDA method that debiases predictions for WSOL in histopathology that explicitly mitigates dominant class prediction bias under significant cross-domain shifts<ref:2603.12468#pg6>. It considers adaptation as an iterative process inspired by machine unlearning: uncertain image samples from over-predicted classes are selectively penalized, while other samples are retained to refine the decision-boundary<ref:2603.12468#pg6>. A jointly pixel-level head optimized with CAMconsistency is also employed to restore discriminative localization under domain shift, improving foreground/background separation after adaptation<ref:2603.12468#pg6>. It consistently outperforms stateof-the-art SFDA in both classification and localization tasks<ref:2603.12468#pg6>.

Improvements for AI systems

Bold class boundary correction using SFDA-DeP: The system can move the classifier f class-boundaries around dominant classes, allowing minority classes to be predicted more often by selectively penalizing and forgetting predictions from dominant classes identified via normalized entropy.
Joint pixel-level localization restoration: The improved system restores discriminative features under domain shift by employing a lightweight pixel-level head is jointly optimized for foreground/background priors derived from class activation maps (CAMs), improving foreground–background separation after adaptation.
Iterative bias mitigation via dynamic resampling: The adaptation process becomes more robust by using dynamic resampling of Br to prevent early incorrect forgetting decisions from becoming irreversible, acting as an iterative correction mechanism, ensuring the model continuously refines its decision boundaries.

Abstract

Under strong cross-domain shifts, weakly supervised object localization (WSOL) models often lose their capacity to discriminate and develop a strong prediction bias toward a subset of classes. As a result, the majority of target samples are assigned to these dominant classes. Since source-free (unsupervised) domain adaptation (SFDA) methods typically rely on pseudo-labels, inaccurate pseudo-label estimation causes suboptimal adaptation and poor classification and localization performance. This issue is particularly challenging in computational histopathology, where shifts in organ-specific morphology, staining protocols, and acquisition conditions can significantly impact class-discriminative visual cues. This paper introduces SFDA with selective prediction forgetting (SFDA-SPF), a method designed to mitigate excessive concentration of predictions for dominant classes during target-domain adaptation. SFDA-SPF identifies prediction-dominant classes based on the model's target predictions and selectively revises uncertain samples assigned to them. Importantly, this procedure does not require knowledge of the true target class distribution. In contrast with previous SFDA methods, SFDA-SPF introduces a 'forget set' composed of uncertain samples from dominant classes near their decision boundaries, enabling the classification boundaries to shift away from biased predictions. A second 'retain set' is used to anchor certain and accurate samples. To forget the predictions of dominant classes, SFDA-SPF introduces a complementary optimization objective that reduces their probability while allowing alternative class assignments to emerge. This iterative optimization process reassesses target images for forget and retain sets at regular intervals. In addition to improving image-level pseudo-label estimation, the pixel-level classifier is jointly optimized, further enhancing localization.