Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
This research benchmarks methods for quantifying uncertainty (UQ) in computer-use agents that translate vision-language model predictions into GUI clicks. The study found that UQ performance is selective: certain methods are stable across datasets for a fixed model, but performance degrades when switching between different types of models or interfaces. It concludes that UQ quality depends heavily on the specific context where the score is observed.