MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models
MissClick exploits how GUI grounding models convert visual data into screen coordinates using their place-value system. It proposes two attacks: one for untargeted disruption by pushing clicks far outside the target area, and one for targeted hijacking by optimizing specific digit probabilities. Results show these methods significantly outperform existing defenses.