Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
summary
The gist
Multiple Instance Learning is addressed through a denoising mutual knowledge distillation process that augments both bag- and instance-level learning with pseudo-label correction capabilities to…
In short
This research proposes a denoising mutual knowledge distillation process to improve Multiple Instance Learning (MIL) by combining bag-level and instance-level training. It uses a dual-level architecture with mutual distillation and self-confidence losses to correct noisy pseudo-labels, leading to superior performance on both bag and instance predictions.
Key concepts
- Dual-Level Architecture
- The system has two parallel learning branches: one for the bag level using bag labels, and another for the instance level that learns to distinguish positive and negative instances without explicit labels. They communicate through mutual distillation.
- Loss Functions (Lbag)
- The main loss function combines four components: cross-entropy from the bag label, supervision from filtered instance predictions (Linst), a self-confidence loss for feature enhancement (Lself), and an attention term (Lattn). This balances different sources of supervision.
- Self-Confidence Losses
- These losses are used to refine the model's learning. They correct pseudo-label errors when the model is not confident in its soft labels or when the bag label and instance branch signals conflict, helping to leverage weak supervision effectively.
- Mutual Knowledge Distillation
- This process involves each branch teaching and learning from the other. It aligns the soft labels of one branch with the predictions of its counterpart, creating a bi-directional framework that boosts overall performance.
Terminology used across episodes
This episode discusses
The paper
Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning · Read on arXiv
Chen Shu, Boyu Fu, Yiman Li, Ting Yin, Wenchuan Zhang, Jie Chen, Yuhao Yi
College of Computer Science, Sichuan University · Sichuan University-Pittsburgh Institute, Sichuan University
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning".
Jane: Here's the gist of the paper. Multiple Instance Learning is addressed through a denoising mutual knowledge distillation process that augments both bag- and instance-level learning with pseudo-label correction capabilities to improve performance on dual-level MIL algorithms.
Tom: The Core Problem of Existing MIL Approaches Existing Multiple Instance Learning methods face challenges related to the quality of supervision they receive, as bag-level methods can focus on easy instances or spurious correlations while instance-level methods are susceptible to noise in instance pseudo labels during training.
Jane: To mitigate these issues, previous work has proposed mechanisms like hard instance mining (HIM) and causal inference to enhance robustness and reduce sensitivity to data distribution.
Tom: The Proposed Dual-Level Architecture The proposed algorithm introduces a dual-level learning architecture that encompasses both bag- and instance-level training using a mutual distillation process.
Title and authors: Tom: So we're looking at this paper, "Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning." It’s tackling a problem where traditional multiple instance learning methods struggle because they have issues with noisy supervision at both the bag and individual instance levels.
Jane: That sounds complicated. To put it simply, the authors are proposing a way to train models that learn from both the whole slide label and individual patch labels simultaneously, but in a cleaner way.
Lu: The core idea is using mutual knowledge distillation to bridge the gap between those two learning levels by correcting pseudo-label errors across both branches.
Meng: So it’s not just one model trying to do everything, it’s like two models talking to each other, refining their predictions based on what the other one thinks is right.
Tom: Exactly. They introduce this dual-level architecture where you have a branch for bag labels and another for instance labels that learn from each other through this distillation process.
Jane: And they use some specific loss functions to guide this conversation, which is interesting because it’s not just one simple error term being minimized.
Lu: They define a loss function Lbag that combines the bag label cross-entropy with terms for instance supervision, self-confidence, and attention scores.
Meng: I see that Lbag has four parts: Llabel, Linst, Lself, and Latten. We need to figure out what each of those terms actually does in practice.
Tom: The first term, Llabel is a weighted sum of the cross-entropy between the bag label and two things: the bag prediction from its branch and an instance prediction based on the highest attention score within that bag.
Jane: So it’s not just comparing a bag to itself; it’s comparing it to a specific instance's prediction, which is a nice way to inject that instance-level signal into the bag learning process.
Lu: The second term, Linst, provides supervision by aggregating filtered patch-level predictions from the instance branch.
Meng: That means we’re filtering out some of the noise from individual instance predictions before feeding them back into the bag learning objective.
Tom: Then there's Lself, which is a self-confidence loss function that encourages the model to focus on features it already knows well and stop it from easily changing those features.
Jane: That sounds like a way to make sure the model stays confident in its strong predictions while correcting errors when things get murky.
Lu: The self-confidence losses are key because they determine how the bag classification head updates itself during each bag iteration, especially when the bag label and the instance branch signal disagree.
Title and authors: Meng: So if both branches are fighting over what’s right, Lself helps decide which signal to trust more for that specific update step.
Tom: It sounds like they're using this feedback loop to improve robustness against that inherent noise in instance pseudo-labels that we talked about earlier.
Jane: And the authors show experimental results on pathology datasets where this dual approach actually outperforms previous distillation-based boosting methods, especially when it comes to instance predictions.
Lu: Specifically, they found that models like CLAM-SB+Ours and CLAM-MB+Ours achieved the best performance on the instance level among all these weakly supervised models tested.
Meng: That’s a solid number to see, but I always wonder about the deployment cost. How much overhead does this distillation process add when we try to run this at scale in a real production environment?
Tom: Well, the implementation details mention training mini-batch sizes of one for the bag branch and one thousand twenty-four for the instance branch, which points to some computational considerations we'll need to look into later.
Jane: And they use a periodic scheduling strategy where you first train the bag branch for kappa iterations and then do one iteration on the instance branch in each cycle.
Lu: That periodic scheduling mimics a multi-task learning setup, ensuring both branches are optimized at a similar pace, which is crucial for this mutual distillation framework to work effectively.
Meng: It sounds like they’re trying to keep the two parts balanced during training so neither one gets totally overwhelmed by the other's signal too early.
Tom: They also mention using hard instance mining modules in the bag branch to mask easy instances during training, which filters out some of that low-hanging fruit noise.
Jane: That sounds like a smart way to force the model to pay attention to harder examples instead of just memorizing the simple ones early on.
Lu: And they also have an additional loss term, Lattn, which calibrates normalized attention scores using cross-entropy between the max of all instance scores and the true label for that instance.
Meng: That Lattn term seems like it’s trying to learn a direct correlation between a bag class and which specific instance gets the highest attention score.
Tom: So they're not just looking at which patches are attended, but linking that attention directly to what the bag is supposed to be.
Jane: And they also use Youden’s J statistic for optimizing the threshold when testing, which gives them a concrete way to evaluate performance beyond just accuracy.
Title and authors: Lu: They confirm in their ablation study that all three main loss terms—Llabel, Linst, and Lself—are indispensable; removing any of them caused a measurable drop in performance on both bag and instance levels.
Meng: That’s reassuring; it means the whole system isn't just one component working well by accident, but every part is contributing to that better result.
Tom: It really shows how critical that balance between the bag supervision and the instance supervision is for getting good results in this setting.
Jane: So, while they achieved strong performance on these public pathology datasets, it’s worth remembering the authors themselves noted a limitation: there are no theoretical guarantees or deep discussions about the fundamental capability limits of this mutual-improvement approach.
Lu: That’s a fair point; without those theoretical proofs or a deeper dive into the underlying principles of mutual improvement, we don't know how far this architecture can actually go.
Meng: From an engineering standpoint, that lack of guarantees means we have to be really careful when we try to apply this to something safety-critical where a single wrong prediction could cause real problems.
Tom: And I think the authors also hinted at some potential negative societal impacts, specifically mentioning mistrust in AI-generated predictions.
Jane: That’s a big concern when we move these kinds of diagnostic tools into real-world clinical settings, especially when people are relying on them for important health decisions.
Lu: The paper's conclusion is that this bi-directional distillation framework enhances dual-level multiple instance learning by aligning the soft labels of each branch with the predictions of its counterpart, using self-confidence losses to boost pseudo-label correction capability.
Meng: So to wrap up, it’s a sophisticated way to blend bag and instance learning by having them constantly check each other's work through these distillation steps.
Tom: It’s an interesting system because it explicitly addresses the weakness of bag methods focusing on easy instances and the weakness of instance methods being too sensitive to noise.
Jane: So, overall, this paper suggests that by making the two levels talk to each other intelligently with those specific loss functions, we can get better results on both bag classification and instance discrimination.
Lu: The work significantly enhances the performance of ABMIL and DSMIL on both datasets compared to the other two frameworks they tested.
Meng: For deployment, it looks promising if you can manage the computational overhead, but we still need more data on how this generalizes outside of these specific public pathology datasets.
Tom: So, that’s our look at "Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning." It’s a lot of moving parts there for an interesting approach to MIL.
The paper's summary: Tom: So, to wrap up what we just talked about on that paper, it’s basically saying they took two separate ways of learning multiple instances—the bag level and the individual instance level—and forced them to teach each other things through a process called mutual knowledge distillation.
Jane: That sounds like a lot of back and forth between the two learning methods. So, what does that actually mean for how we look at these complex images?
Lu: It means they create this dual-level architecture where the bag branch and the instance branch constantly refine their predictions by looking at each other’s soft labels. This is supposed to clean up those noisy pseudo-labels you get when you try to train instance classifiers without having perfect ground truth for every single patch.
Meng: So, it's like if one team makes a mistake on an individual piece, the other team corrects it because they see what the bag label expects that instance to be.
Tom: Exactly. They use these specific loss functions, like Lself and Linst, to make sure both branches are pulling in the right kind of information. One term helps the model focus on features it already knows well while making sure it doesn't get stuck on those bad initial guesses.
Jane: And they have this periodic scheduling thing, training the bag branch first for several steps and then one step for the instance branch. That’s designed to keep both parts moving at a similar speed so they actually learn together.
Lu: That structure is what allows them to leverage W2S generalization, which is this ability of models to generalize well even when their initial labels are weak or noisy. The distillation process essentially boosts that generalization capability by aligning the soft labels between the branches.
Meng: From my side, it’s interesting because it tries to make two separate training tasks—the bag task and the instance task—work in a more coordinated way than just running them separately. I'm wondering how much more stable this is when we try to deploy these things on real traffic or medical scans where the supervision isn't perfect.
Tom: That’s the million-dollar question, Meng. The paper shows strong performance on public pathology datasets, and they beat previous methods for instance predictions. They found CLAM-SB+Ours and CLAM-MB+Ours were top performers on the instance level among weakly supervised models tested.
Jane: So, if I’m a doctor or a diagnostician looking at an AI output for a biopsy, this paper suggests that the AI isn't just guessing based on one label; it’s cross-checking its bag prediction against what it thinks the individual pieces should look like.
Lu: Precisely. The core finding is that this bi-directional distillation framework significantly enhances performance on both ABMIL and DSMIL datasets compared to the other two frameworks they tested, showing a real uplift in accuracy for instance predictions specifically.
Tom: It’s not just about getting a higher AUC or accuracy number, though. The ablation study showed that every single part of that loss function—Llabel, Linst, and Lself—was absolutely necessary; if you take any one out, the performance drops measurably on both levels.
Jane: That’s good data for me because it proves the system isn't reliant on a lucky combination of techniques. It means they’ve actually built a mechanism where every piece contributes to that final result.
Meng: I still have my reservations about scaling this up. The training setup itself has mini-batch sizes of one and one thousand twenty-four for the instance branch, which suggests it might be computationally intensive when you try to run it on massive datasets in production. It’s a lot of overhead just to get those results on paper.
Tom: That computational cost is something we have to watch closely. But the mechanism itself—this way of aligning soft labels—is what makes this approach interesting for the future of multiple instance learning.
Jane: So, while they nailed the performance on these specific medical image tasks, a limitation they explicitly state is that there are no theoretical guarantees about how far this mutual improvement paradigm can actually go beyond these specific setups.
Lu: That’s true. They haven't provided theoretical proofs for the fundamental capability limits of this mutual-improvement approach yet, which means we don't have a complete picture of its long-term potential.
Meng: And I think that lack of formal guarantees is a big red flag when you are thinking about using this in safety-critical applications, like real-world medical diagnosis.
Tom: Yeah, and they also brought up some potential negative societal impacts, specifically the risk of mistrust in predictions generated by AI, which is something we need to be mindful of as these tools get more integrated into patient care.
Jane: So, the big picture here is that this paper provides a really solid engineering technique for making dual-level learning systems more robust against noisy supervision by having them actively correct each other’s errors.
Lu: It’s a clever way to leverage the weak supervision capability of W2S generalization by using these distillation signals to sharpen both the bag and instance predictions simultaneously.
Tom: So, we’ve seen that this method significantly enhances performance on ABMIL and DSMIL, putting it ahead of other frameworks they compared. It definitely shows how critical that feedback loop between the two learning levels is for getting good results in this domain.
The paper's improvements: Tom: So, we're talking about how this paper actually improves things beyond just saying they do distillation; they introduce these specific tweaks to the loss function Lbag.
Jane: Okay, so we already know it’s a combination of several terms, but what are these new additions doing to make the bag branch smarter?
Lu: They add a term for instance discrimination—let's call it Lattn—which calibrates those attention scores using cross-entropy between the max of all instance scores and the true label for that instance.
Meng: So, they’re trying to learn a direct link between which specific patch gets the most attention and what kind of bag class that should be. That sounds like it tightens up how we interpret those weak supervision signals.
Tom: Right, because it moves beyond just seeing *which* instance is looked at and starts linking that attention score directly to the expected bag class. It makes the soft labels from the instance branch much more reliable for training the bag branch.
Jane: That means if an instance is highly attended but its prediction disagrees with what the bag expects, that signal gets corrected more precisely because of this Lattn term.
Lu: And they also mentioned using hard positive mining modules in the bag branch, specifically setting normalized attention scores to zero if the instance classifier prediction exceeds a certain threshold.
Meng: That sounds like a filter for noise. If the instance model is overly confident or wrong on a patch, they essentially tell the bag branch to ignore that specific instance's input during training.
Tom: Exactly, it filters out those noisy instances and forces the bag branch to focus its learning on the most informative ones. It’s a form of intelligent data pruning during the training process itself.
Jane: So, they’re not just relying on the average signal from all patches; they are actively using instance predictions to clean up and prioritize which signals are actually useful for each level of learning.
Lu: And remember, those self-confidence losses, Lself, aren't just there to keep things stable; they are crucial because they correct pseudo label errors when the bag label and the instance branch supervision signal disagree.
Meng: That’s the part that makes it robust—it tells the model when its own confidence is low or when it’s getting conflicting instructions from different sources, so it can adjust its classification head accordingly.
Tom: It's a feedback loop for uncertainty management. The self-confidence loss ensures the model enhances features it already knows well, but also helps prevent those strong predictions from being easily altered by noisy instance inputs.
Jane: It makes sense that they focus on these corrections because they found that removing any of those components—Llabel, Linst, or Lself—caused a measurable drop in performance on both the bag and instance levels.
Lu: That’s a strong signal. It confirms that this specific balance of supervision is what drives the overall gain in dual-level learning capability for ABMIL and DSMIL.
Meng: I still think we have to watch how much computational overhead this adds when you try to run these refined loss functions at scale on huge medical image archives. That’s where the engineering reality kicks in.
Tom: True, the paper clearly states that their implementation uses mini-batch sizes of one for the bag branch and one thousand twenty-four for the instance branch, which points toward a significant computational footprint during training.
Jane: So, while this technique shows how to get better predictions by making two learning levels talk to each other more intelligently, we still have that caveat about needing more theoretical proof on its fundamental limits.
Conclusion: Tom: So, to wrap up on "Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning," the main point is that this bi-directional distillation framework creates a cleaner way to train bag and instance learning models by making them constantly check each other's work.
Jane: It’s like they are using mutual knowledge distillation to align the soft labels of each branch with what the other branch predicts, which helps correct those noisy pseudo-labels we talked about earlier.
Lu: The core idea is that by introducing self-confidence losses, Lself specifically encourages the model to enhance well-learned features while preventing them from being easily altered, leveraging the generalization capability of the bag branch through its own confident predictions.
Meng: It’s clever how they balance those two branches using a periodic scheduling strategy where you train the bag branch for kappa iterations and then one iteration for the instance branch in each cycle, keeping them optimized at a similar pace.
Tom: I agree, that structure is designed to keep both parts learning together rather than one dominating the other too early. The ablation study confirmed that every single term—Llabel, Linst, and Lself—was indispensable; removing any of them caused a measurable drop in performance on both levels.
Jane: That’s a strong result because it shows the whole system isn't just one component working by accident; every part is contributing to that better performance. But we have to remember they didn't provide theoretical guarantees about the fundamental capability limits of this mutual-improvement paradigm.
Lu: They haven't provided those theoretical proofs yet, which means we don’t know exactly how far this architecture can actually go beyond these specific setups, which is a limitation we need to keep in mind.
Meng: From an engineering standpoint, that lack of formal guarantees means we have to be really careful when we try to apply this to something safety-critical where a single wrong prediction could cause real problems.
Tom: And they also flagged some potential negative societal impacts, specifically mentioning the risk of mistrust in predictions generated by AI, which is something we need to be mindful of as these tools get more integrated into patient care.
Jane: So, while they achieved strong performance on those public pathology datasets, this paper shows how to get better results by making the two levels talk to each other more intelligently with those specific loss functions.
Lu: This work significantly enhances the performance of ABMIL and DSMIL on both datasets compared to the other two frameworks they tested, showing a real uplift in accuracy for instance predictions specifically.
Meng: For deployment, it looks promising if you can manage the computational overhead, but we still need more data on how this generalizes outside of these specific public pathology datasets before we can really trust it at scale.
Tom: So, "Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning" shows that by aligning soft labels with predictions from the counterpart using self-confidence losses, we get better dual-level multiple instance learning.
Jane: It’s a sophisticated way to blend bag and instance learning where they constantly refine each other's work through these distillation steps.
Lu: It’s an interesting system because it explicitly addresses the weakness of bag methods focusing on easy instances and the weakness of instance methods being too sensitive to noise.
Meng: I think we should be watching how this kind of cross-level supervision evolves as we move toward more complex, real-world scenarios where supervision is naturally sparse.
Tom: We’ll see what the next paper brings to the table, but this one definitely proves that thoughtful loss function design can really improve performance in these complex learning tasks.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation