Unifying Adversarially Robust Model Experts in Vision-Language Models

arXiv:2607.27897 · cs.CV · Submitted 2026-07-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Unifying Adversarially Robust Model Experts in Vision-Language Models".

Jane: Here's the gist of the paper. Vision-language models are vulnerable to adversarial attacks, and this paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies knowledge from multiple specialized model experts to improve robustness <ref:2607.27897#pg2>.

Tom: This approach is significant because it demonstrates that combining different fine-tuned strategies can yield a single unified model with complementary robustness properties <ref:2607.27897#pg4>.

Jane: The Proposed Framework CARE CARE maintains multiple experts, each derived from a different adversarial finetuning strategy, and unifies their learned knowledge into a single unified robust model <ref:2607.27897#pg5>.

Tom: The framework involves four main steps: initializing a universal perturbation, having each expert generate its own adversarial sample, optimizing the parameters using joint losses for specialization and harmonization, and finally combining the parameters using Exponential Moving Average (EMA) <ref:2607.27897#pg4> A Universal perturbation is constructed by aligning perturbed logits with those of Alignment expert and image embeddings with those of Invariance expert <ref:2607.27897#pg5>.

Paper summary: Tom: So we're diving deeper into "Unifying Adversarially Robust Model Experts in Vision-Language Models." The paper's main point is that adversarial fine-tuning creates different robust model experts, and these experts don't combine their individual robustness well >

Jane: The authors propose CARE, which is a collaborative adversarial fine-tuning framework. It claims this framework maintains multiple experts, enables knowledge exchange through embedding-space harmonization, and consolidates that learned knowledge into a single unified robust model >

Lu: What matters here is that the approach demonstrates that combining different fine-tuned strategies can actually yield a single unified model with complementary robustness properties >

Meng: So it's not just about stacking models; it's about how they learn from each other, which is interesting for building more versatile systems. But what makes them special compared to just training one big model directly?

Lalam: The framework sets up a universal perturbation initialization first, then each expert generates its own adversarial samples, and then the experts optimize their parameters using joint losses for both specialization and harmonization >

Tom: It’s this joint loss optimization that drives the core idea. It forces them to specialize in their own way while also learning from the other expert's knowledge through that harmonization loss >

Jane: So it's not just training one model harder; it's training a team where each member specializes and then learns from the others to make the whole team stronger >

Lu: Exactly. And finally, they combine all those learned parameters using Exponential Moving Average or EMA to create that final unified robust model >

Tom: The significance is showing that these different fine-tuning paths, like image-text alignment versus image invariance, aren't mutually exclusive; they can be used together to build something better >

Jane: It means if you need robustness for both recognizing the class name and just keeping the visual details stable under attack, this unified model might give you that in one package >

Meng: If we think about deployment, having a single unified model is much cleaner than trying to manage several separate specialized models for different tasks or datasets. That consolidation is definitely something I care about >

Lalam: The paper also points out that the harmonization mechanism involves aligning clean logits between the Invariance expert and the Alignment expert, and aligning clean image embeddings between those two experts >

Tom: So it’s a very specific way they force them to communicate in the embedding space so they can transfer useful information without just mixing their raw parameters >

Jane: That sounds like a smart way to manage the knowledge transfer, focusing on the representation itself rather than just superficial parameter mixing >

Conclusion: Tom: The paper "Unifying Adversarially Robust Model Experts in Vision-Language Models" by Nguyen Duc Thai et al is showing us how to combine different ways of making models robust into one powerful system >

Jane: It’s about taking the specialized robustness you get from different fine-tuning strategies, like image-text alignment versus image invariance, and unifying them into one model that has both kinds of protection >

Lu: The authors prove that this approach doesn't sacrifice the individual strengths of those experts; they preserve the specialization while adding a layer of shared knowledge >

Meng: The implication for me is that if we are building production AI, having a single unified model instead of three separate specialized ones to cover different failure modes is much better for maintenance and speed >

Tom: It suggests that collaborative learning across different model experts is actually a viable path to improving adversarial robustness in vision language models >

Jane: So the main implication is that we don't have to choose one path for defense; we can leverage the strengths of different specialized training angles together >

Nanyang Technological University · Agency for Science, Technology and Research (A*STAR)

cs.CV

Submitted: 2026-07-30

Updated: 2026-10-31

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: Vision-language models are vulnerable to adversarial attacks, and this paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies knowledge from multiple specialized model…

Key concepts

CARE Framework
A framework that maintains several independent 'experts,' each trained using a different adversarial fine-tuning method. CARE then unifies the knowledge learned by these experts into one single, robust model through joint optimization and parameter combination.
Adversarial Fine-tuning Strategies
These are different methods used to train models against attacks. The paper uses two main ones: one based on cross-entropy loss for aligning images with text, and another using L2-distance loss to enforce image invariance against perturbations.
Knowledge Harmonization
A mechanism that enables the different experts to share information effectively. This is achieved by adding an embedding alignment loss, forcing the experts' learned embeddings into a consistent space so they can exchange knowledge about robustness.
Complementary Strengths
The idea that combining different specialized models creates a better overall system than any single model. For example, one expert might be strong at text alignment while another excels at image invariance, and CARE leverages this combination for enhanced performance.

Terminology

Summary

Vision-language models are vulnerable to adversarial attacks, and this paper proposes CARE, a collaborative adversarial fine-tuning framework that unifies knowledge from multiple specialized model experts to improve robustness <ref:2607. p. 0

Where the paper says it

We refer to these specialized finetuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model.

27897#pg2>. This approach is significant because it demonstrates that combining different fine-tuned strategies can yield a single unified model with complementary robustness properties <ref:2607. p. 0
Where the paper says it

Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies with complementary robustness behaviors.

27897#pg4>.

The Proposed Framework CARE

CARE maintains multiple experts, each derived from a different adversarial finetuning strategy, and unifies their learned knowledge into a single unified robust model <ref:2607.27897#pg5>. The framework involves four main steps: initializing a universal perturbation, having each expert generate its own adversarial sample, optimizing the parameters using joint losses for specialization and harmonization, and finally combining the parameters using Exponential Moving Average (EMA) <ref:2607. p. 2

Where the paper says it

8, AUGUST 2021 3 Step 1: Universal perturbation initialization Step 2: Independent adversarial generation Step 3: Expert harmonization and update Step 4: Knowledge generalization through EMA Image-Text alignment expert ImageInvariance expert More robust class logits More robust image embedding Class logits Image embedding Noise Universal perturbation Universal perturbation Adversarial using CE loss Adversarial using L2 loss Align adv Invariance adv Fine-tune Fine-tune Align Image embedding Align Class logits Invariance expert loss: Alignment expert loss: CARE Lion Dog Cat Panda Image-Text alignment expert ImageInvariance expert Image-Text alignment expert ImageInvariance expert Image-Text alignment expert ImageInvariance expert Align adv Invariance adv Fig. 2: Our method CARE consists of 4 steps: Initialize a universal perturbation with inputs from both experts; Each expert generates its own adversarial samples; Experts optimize their parameters using joint losses: adversarial fine-tuning loss for specialization and harmonization loss to learn from each other; Finally, the parameters are combined using the Exponential Moving Average (EMA) method.

27897#pg4> A Universal perturbation is constructed by aligning perturbed logits with those of Alignment expert and image embeddings with those of Invariance expert <ref:2607. p. 4
Where the paper says it

We align the perturbed logits with those of θAlign and its image embeddings with those of θInv using negative cosine similarity. The loss for generating the universal perturbation is: h(˜x) = 1 − cos z(˜x, θAlign), z(x, θAlign) , (18) g(˜x) = 1 − cos fθInv (˜x), fθInv (x) , (19) LGAIP (˜x, x) = h(˜x) + g(˜x).

27897#pg5>.

How the training signal is built

The framework utilizes two base adversarial fine-tuning strategies: the Image-Text alignment expert, which relies on cross-entropy loss for adversarial training, and the Image-Invariance expert, which relies on L2-distance loss <ref:2607.27897#pg4>. The harmonization mechanism is introduced by adding an Embedding alignment loss to each expert to enable knowledge exchange through embedding space harmonization <ref:2607. p. 1

Where the paper says it

CARE maintains multiple experts, each derived from a different adversarial finetuning strategy, and unifies their learned knowledge into a single unified robust model. • We introduce an embedding-space harmonization mechanism that enables knowledge exchange among experts during training, thus improving stability without sacrificing exploration.

27897#pg5>. Specifically, the Invariance expert aligns its clean logits with those of the Alignment expert, and the Alignment expert aligns its clean image embeddings with those of the Invariance expert <ref:2607. p. 3
Where the paper says it

Meanwhile, θInv optimizes the L2 loss by operating on the image embeddings. We add an Embedding alignment loss to each expert: θInv will align its clean logits (which it does not optimize) to those of θAlign (which optimizes logits directly).

27897#pg5>.

Where the gains come from

The gains stem from leveraging complementary strengths between different robust model experts, such as TeCoA and FARE <ref:2607.27897#pg4>. The results show that CARE outperforms individually trained experts in both image classification and downstream vision–language tasks <ref:2607. p. 1

Where the paper says it

• We introduce an embedding-space harmonization mechanism that enables knowledge exchange among experts during training, thus improving stability without sacrificing exploration. • Extensive experiments demonstrate that CARE outperforms individually trained experts in both image classification and downstream vision–language tasks.

27897#pg5>. For instance, CARE4 outperforms both TeCoA4 and FARE4 on all datasets <ref:2607. p. 7
Where the paper says it

JOURNAL OF LATEX CLASS FILES, VOL. 8, AUGUST 2021 8 TABLE IV: Average zero-shot image classification accuracies (%) of TeCoA4 , FARE4 , CARE4 and the two experts at the end of training.

27897#pg9> The analysis suggests that the Invariance expert benefits from knowledge exchange with the Alignment expert, allowing it to maintain task knowledge ignored during FARE training <ref:2607. p. 7
Where the paper says it

Furthermore, since the objective of the Invariance expert (as well as FARE) is to minimize differences between clean and adversarial images, it has not been fine-tuned on the image classification task. The information exchange with the Alignment expert helps the model maintain task knowledge, which is ignored entirely during training FARE.

27897#pg4>.

Comparison to other collaborative frameworks

CARE's harmonization method is compared against parameter redistribution, which involves overwriting experts' parameters periodically <ref:2607.27897#pg6>. The results show that the Expert harmonization model outperforms its redistribution counterpart completely, up to nearly 2% more robust accuracy on epsilon = 4/255 <ref:2607. p. 9

Where the paper says it

results can be found in Table VIII. It shows that the Expert harmonization model outperforms its redistribution counterpart completely, up to nearly 2% more robust accuracy on ϵ = 4 255 .

27897#pg9> This indicates that the collaborative learning across model experts is a promising direction for improving adversarial robustness <ref:2607. p. 0
Where the paper says it

Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.

27897#pg2>.

Conclusion

In this paper, CARE, a novel framework for adversarial training that can generalize across robust model experts, is presented <ref:2607.27897#pg2>. CARE successfully combines the complementary strengths of TeCoA and FARE and gains the specialization of both <ref:2607. p. 7

Where the paper says it

However, CARE is still the best for robust accuracy. This suggests that CARE is able to capitalize on FARE’s strong specialization, while the imagetext alignment knowledge is transferable across text encoders, since we use different text encoders for downstream tasks.

27897#pg2>. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness <ref:2607. p. 0
Where the paper says it

Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.

27897#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Collaborative Adversarial Robustness fine-tuning using Embedding alignment (CARE) allows for training multiple model experts simultaneously, enabling knowledge exchange through embedding space harmonization to consolidate learned knowledge into a single unified robust model. This unified model inherits complementary robustness properties from different specialized strategies, such as image-text alignment and image-invariance.

  2. The Expert harmonization mechanism aligns each expert to implicitly learn the adversarial fine-tuning process of the other by adding an Embedding alignment loss to each expert's objective. Specifically, this involves aligning clean logits with those of the other expert and aligning clean image embeddings between them, which prevents drift and improves stability without sacrificing exploration.

  3. The Universal perturbation initialization process (GAIP) constructs a shared universal perturbation using inputs from both experts by minimizing the sum of alignment losses for logits and image embeddings. This shared initialization benefits both adversarial specializations, leading to more coherent adversarial samples that are used by each expert during their respective optimization steps.

  4. The CARE framework outperforms individually trained experts across various evaluation settings, including in-distribution accuracy and zero-shot accuracy on diverse datasets like ImageNet, Caltech101, and STL-10. This demonstrates the ability of CARE to generalize across different adversarial robust model experts, outperforming TeCoA and FARE.

  5. Downstream vision-language tasks like image captioning (CIDEr score) and visual question answering (VQA accuracy) show that CARE is the best performer in all settings, suggesting it capitalizes on the specialization of both alignment and invariance experts to maintain consistent answers and captions under perturbations.

Abstract

Vision-language models (VLMs), such as CLIP, are vulnerable to adversarial attacks, posing a serious problem for real-life applications and deployment. Adversarial fine-tuning emerges as a prominent defense method; however, different fine-tuning strategies often produce specialized models with distinct robustness characteristics. Each fine-tuned model in turn thrives in some evaluation settings but falters on others, limiting their defensive capabilities. We refer to these specialized fine-tuned models as robust model experts and propose a collaborative adversarial fine-tuning framework: CARE - Collaborative Adversarial Robustness fine-tuning using Embedding alignment. CARE maintains multiple experts during training, enables knowledge exchange through embedding-space harmonization, and consolidates the learned knowledge into a single unified robust model. Experts benefit from one another while preserving their individual specializations, enabling the final model to inherit complementary robustness properties. In this paper, we demonstrate CARE on two different adversarial fine-tuning strategies with complementary robustness behaviors. Extensive experiments on classic image classification and downstream vision-language tasks display the effectiveness of our approach, with CARE being able to outperform individually learned model experts. The results suggest that collaborative learning across model experts is a promising direction for improving adversarial robustness.

Sources

Related papers