Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders
The study tested whether a similarity score from an attacked vision-language model predicts if an independent model can recover the original target image. While robust encoders showed better performance than vanilla models, no single score reliably predicted recovery across all cases. The paper proposes a new evidence taxonomy to better frame claims about adversarial attacks.