When Masking Helps or Hurts Robustness in Compressed CLIP: A Pre-Deployment Diagnostic
The study investigates whether masking tokens helps or hurts robustness before deployment using a pre-deployment diagnostic called the Spurious Inversion Metric (SIM). SIM predicts this effect based on background/foreground similarity in CLIP's embedding space. A gating policy uses SIM to decide whether to apply a masking method (MARS) or fall back to standard pruning, ensuring efficiency while maintaining robustness.