Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks
The research addresses vulnerabilities in multimodal web agents that use both screenshots and accessibility trees for interaction. It found that attackers can corrupt both observation channels simultaneously, with visual attacks significantly outperforming text-only ones. The proposed Dual-Modality Multi-Stage Adversarial Safety Training (DMAST) framework hardens these agents through a three-stage co-evolutionary process, resulting in agents that are more robust against cross-modal deception.