Unifying Adversarially Robust Model Experts in Vision-Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Unifying Adversarially Robust Model Experts in Vision-Language Models".
Jane: Here's the gist of the paper. Vision-language models are vulnerable to adversarial attacks, and this paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies knowledge from multiple specialized model experts to improve robustness <ref:2607.27897#pg2>.
Tom: This approach is significant because it demonstrates that combining different fine-tuned strategies can yield a single unified model with complementary robustness properties <ref:2607.27897#pg4>.
Jane: The Proposed Framework CARE CARE maintains multiple experts, each derived from a different adversarial finetuning strategy, and unifies their learned knowledge into a single unified robust model <ref:2607.27897#pg5>.
Tom: The framework involves four main steps: initializing a universal perturbation, having each expert generate its own adversarial sample, optimizing the parameters using joint losses for specialization and harmonization, and finally combining the parameters using Exponential Moving Average (EMA) <ref:2607.27897#pg4> A Universal perturbation is constructed by aligning perturbed logits with those of Alignment expert and image embeddings with those of Invariance expert <ref:2607.27897#pg5>.
Paper summary: Tom: So we're diving deeper into "Unifying Adversarially Robust Model Experts in Vision-Language Models." The paper's main point is that adversarial fine-tuning creates different robust model experts, and these experts don't combine their individual robustness well >
Jane: The authors propose CARE, which is a collaborative adversarial fine-tuning framework. It claims this framework maintains multiple experts, enables knowledge exchange through embedding-space harmonization, and consolidates that learned knowledge into a single unified robust model >
Lu: What matters here is that the approach demonstrates that combining different fine-tuned strategies can actually yield a single unified model with complementary robustness properties >
Meng: So it's not just about stacking models; it's about how they learn from each other, which is interesting for building more versatile systems. But what makes them special compared to just training one big model directly?
Lalam: The framework sets up a universal perturbation initialization first, then each expert generates its own adversarial samples, and then the experts optimize their parameters using joint losses for both specialization and harmonization >
Tom: It’s this joint loss optimization that drives the core idea. It forces them to specialize in their own way while also learning from the other expert's knowledge through that harmonization loss >
Jane: So it's not just training one model harder; it's training a team where each member specializes and then learns from the others to make the whole team stronger >
Lu: Exactly. And finally, they combine all those learned parameters using Exponential Moving Average or EMA to create that final unified robust model >
Tom: The significance is showing that these different fine-tuning paths, like image-text alignment versus image invariance, aren't mutually exclusive; they can be used together to build something better >
Jane: It means if you need robustness for both recognizing the class name and just keeping the visual details stable under attack, this unified model might give you that in one package >
Meng: If we think about deployment, having a single unified model is much cleaner than trying to manage several separate specialized models for different tasks or datasets. That consolidation is definitely something I care about >
Lalam: The paper also points out that the harmonization mechanism involves aligning clean logits between the Invariance expert and the Alignment expert, and aligning clean image embeddings between those two experts >
Tom: So it’s a very specific way they force them to communicate in the embedding space so they can transfer useful information without just mixing their raw parameters >
Jane: That sounds like a smart way to manage the knowledge transfer, focusing on the representation itself rather than just superficial parameter mixing >
Conclusion: Tom: The paper "Unifying Adversarially Robust Model Experts in Vision-Language Models" by Nguyen Duc Thai et al is showing us how to combine different ways of making models robust into one powerful system >
Jane: It’s about taking the specialized robustness you get from different fine-tuning strategies, like image-text alignment versus image invariance, and unifying them into one model that has both kinds of protection >
Lu: The authors prove that this approach doesn't sacrifice the individual strengths of those experts; they preserve the specialization while adding a layer of shared knowledge >
Meng: The implication for me is that if we are building production AI, having a single unified model instead of three separate specialized ones to cover different failure modes is much better for maintenance and speed >
Tom: It suggests that collaborative learning across different model experts is actually a viable path to improving adversarial robustness in vision language models >
Jane: So the main implication is that we don't have to choose one path for defense; we can leverage the strengths of different specialized training angles together >
Nanyang Technological University · Agency for Science, Technology and Research (A*STAR)
cs.CV
Submitted: 2026-07-30
Updated: 2026-10-31
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 80/100
The gist: Vision-language models are vulnerable to adversarial attacks, and this paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies knowledge from multiple specialized model…
Key concepts
- CARE Framework
- A framework that maintains several independent 'experts,' each trained using a different adversarial fine-tuning method. CARE then unifies the knowledge learned by these experts into one single, robust model through joint optimization and parameter combination.
- Adversarial Fine-tuning Strategies
- These are different methods used to train models against attacks. The paper uses two main ones: one based on cross-entropy loss for aligning images with text, and another using L2-distance loss to enforce image invariance against perturbations.
- Knowledge Harmonization
- A mechanism that enables the different experts to share information effectively. This is achieved by adding an embedding alignment loss, forcing the experts' learned embeddings into a consistent space so they can exchange knowledge about robustness.
- Complementary Strengths
- The idea that combining different specialized models creates a better overall system than any single model. For example, one expert might be strong at text alignment while another excels at image invariance, and CARE leverages this combination for enhanced performance.
Terminology
Summary
Vision-language models are vulnerable to adversarial attacks, and this paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies knowledge from multiple specialized model experts to improve robustness <ref:2607. p. 0 We refer to these specialized finetuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative
Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables
knowledge exchange through embedding-space harmonization,
and consolidates the learned knowledge into a single unified
robust model. Experts benefit from one another while preserving
their individual specializations, enabling the final model to inherit
complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies
with complementary robustness behaviors.Where the paper says it
Where the paper says it
The Proposed Framework CARE
CARE maintains multiple experts, each derived from a different adversarial finetuning strategy, and unifies their learned knowledge into a single unified robust model <ref:2607.27897#pg5>. The framework involves four main steps: initializing a universal perturbation, having each expert generate its own adversarial sample, optimizing the parameters using joint losses for specialization and harmonization, and finally combining the parameters using Exponential Moving Average (EMA) <ref:2607. p. 2 8, AUGUST 2021 3
Step 1: Universal perturbation initialization Step 2: Independent adversarial generation
Step 3: Expert harmonization and update
Step 4: Knowledge generalization through EMA
Image-Text
alignment
expert
ImageInvariance
expert
More robust
class logits
More robust
image embedding
Class logits
Image embedding
Noise Universal
perturbation
Universal
perturbation
Adversarial
using CE loss
Adversarial
using L2 loss
Align adv
Invariance adv
Fine-tune
Fine-tune
Align
Image
embedding
Align
Class
logits
Invariance expert loss:
Alignment expert loss:
CARE
Lion
Dog
Cat
Panda
Image-Text
alignment
expert
ImageInvariance
expert
Image-Text
alignment
expert
ImageInvariance
expert
Image-Text
alignment
expert
ImageInvariance
expert
Align adv
Invariance adv
Fig. 2: Our method CARE consists of 4 steps: Initialize a universal perturbation with inputs from both experts; Each expert
generates its own adversarial samples; Experts optimize their parameters using joint losses: adversarial fine-tuning loss for
specialization and harmonization loss to learn from each other; Finally, the parameters are combined using the Exponential
Moving Average (EMA) method. We align the perturbed
logits with those of θAlign and its image embeddings with
those of θInv using negative cosine similarity. The loss for
generating the universal perturbation is:
h(˜x) = 1 − cos z(˜x, θAlign), z(x, θAlign)
, (18)
g(˜x) = 1 − cos fθInv (˜x), fθInv (x)
, (19)
LGAIP (˜x, x) = h(˜x) + g(˜x).Where the paper says it
Where the paper says it
How the training signal is built
The framework utilizes two base adversarial fine-tuning strategies: the Image-Text alignment expert, which relies on cross-entropy loss for adversarial training, and the Image-Invariance expert, which relies on L2-distance loss <ref:2607.27897#pg4>. The harmonization mechanism is introduced by adding an Embedding alignment loss to each expert to enable knowledge exchange through embedding space harmonization <ref:2607. p. 1 CARE maintains multiple
experts, each derived from a different adversarial finetuning strategy, and unifies their learned knowledge into
a single unified robust model. • We introduce an embedding-space harmonization mechanism that enables knowledge exchange among experts
during training, thus improving stability without sacrificing exploration. Meanwhile, θInv optimizes
the L2 loss by operating on the image embeddings. We add
an Embedding alignment loss to each expert: θInv will align
its clean logits (which it does not optimize) to those of θAlign
(which optimizes logits directly).Where the paper says it
Where the paper says it
Where the gains come from
The gains stem from leveraging complementary strengths between different robust model experts, such as TeCoA and FARE <ref:2607.27897#pg4>. The results show that CARE outperforms individually trained experts in both image classification and downstream vision–language tasks <ref:2607. p. 1 • We introduce an embedding-space harmonization mechanism that enables knowledge exchange among experts
during training, thus improving stability without sacrificing exploration. • Extensive experiments demonstrate that CARE outperforms individually trained experts in both image classification and downstream vision–language tasks. JOURNAL OF LATEX CLASS FILES, VOL. 8, AUGUST 2021 8
TABLE IV: Average zero-shot image classification accuracies
(%) of TeCoA4
, FARE4
, CARE4
and the two experts at the
end of training. Furthermore, since the objective of the Invariance
expert (as well as FARE) is to minimize differences between
clean and adversarial images, it has not been fine-tuned on the
image classification task. The information exchange with the
Alignment expert helps the model maintain task knowledge,
which is ignored entirely during training FARE.Where the paper says it
Where the paper says it
Where the paper says it
Comparison to other collaborative frameworks
CARE's harmonization method is compared against parameter redistribution, which involves overwriting experts' parameters periodically <ref:2607.27897#pg6>. The results show that the Expert harmonization model outperforms its redistribution counterpart completely, up to nearly 2% more robust accuracy on epsilon = 4/255 <ref:2607. p. 9 results can be found in Table VIII. It shows that the Expert
harmonization model outperforms its redistribution counterpart
completely, up to nearly 2% more robust accuracy on ϵ =
4
255 . Extensive experiments
on classic image classification and downstream vision-language
tasks display the effectiveness of our approach, with CARE
being able to outperform individually learned model experts. The
results suggest that collaborative learning across model experts
is a promising direction for improving adversarial robustness.Where the paper says it
Where the paper says it
Conclusion
In this paper, CARE, a novel framework for adversarial training that can generalize across robust model experts, is presented <ref:2607.27897#pg2>. CARE successfully combines the complementary strengths of TeCoA and FARE and gains the specialization of both <ref:2607. p. 7 However, CARE is still the
best for robust accuracy. This suggests that CARE is able to
capitalize on FARE’s strong specialization, while the imagetext alignment knowledge is transferable across text encoders,
since we use different text encoders for downstream tasks. Extensive experiments
on classic image classification and downstream vision-language
tasks display the effectiveness of our approach, with CARE
being able to outperform individually learned model experts. The
results suggest that collaborative learning across model experts
is a promising direction for improving adversarial robustness.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Collaborative Adversarial Robustness fine-tuning using Embedding alignment (CARE) allows for training multiple model experts simultaneously, enabling knowledge exchange through embedding space harmonization to consolidate learned knowledge into a single unified robust model. This unified model inherits complementary robustness properties from different specialized strategies, such as image-text alignment and image-invariance.
-
The Expert harmonization mechanism aligns each expert to implicitly learn the adversarial fine-tuning process of the other by adding an Embedding alignment loss to each expert's objective. Specifically, this involves aligning clean logits with those of the other expert and aligning clean image embeddings between them, which prevents drift and improves stability without sacrificing exploration.
-
The Universal perturbation initialization process (GAIP) constructs a shared universal perturbation using inputs from both experts by minimizing the sum of alignment losses for logits and image embeddings. This shared initialization benefits both adversarial specializations, leading to more coherent adversarial samples that are used by each expert during their respective optimization steps.
-
The CARE framework outperforms individually trained experts across various evaluation settings, including in-distribution accuracy and zero-shot accuracy on diverse datasets like ImageNet, Caltech101, and STL-10. This demonstrates the ability of CARE to generalize across different adversarial robust model experts, outperforming TeCoA and FARE.
-
Downstream vision-language tasks like image captioning (CIDEr score) and visual question answering (VQA accuracy) show that CARE is the best performer in all settings, suggesting it capitalizes on the specialization of both alignment and invariance experts to maintain consistent answers and captions under perturbations.
Abstract
Vision-language models (VLMs), such as CLIP, are vulnerable to adversarial attacks, posing a serious problem for real-life applications and deployment. Adversarial fine-tuning emerges as a prominent defense method; however, different fine-tuning strategies often produce specialized models with distinct robustness characteristics. Each fine-tuned model in turn thrives in some evaluation settings but falters on others, limiting their defensive capabilities. We refer to these specialized fine-tuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model. Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies with complementary robustness behaviors. Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.
Sources
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
- OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
- Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast
- Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
- Editing Models with Task Arithmetic
- Fine-Grained Visual Classification of Aircraft
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs