VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
cs.CL, cs.AI, cs.SE
Submitted: 2026-04-23
Updated: 2026-09-24
Comments: The first two authors contribute equally
Code: https://github.com/UCSC-VLAA/VLAA-GUI
Project page: https://ucsc-vlaa.github.io/VLAA-GUI
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same
Terminology
Abstract
Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. We present VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search. First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step -- with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies show that all three proposed components consistently improve a strong backbone, while a weaker backbone benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models.
Sources
- Agent S: An Open Agentic Framework that Uses Computers Like a Human
- Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
- Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification
- ScreenAI: A Vision-Language Model for UI and Infographics Understanding
- Qwen3-VL Technical Report
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
- Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
- Why Do Multi-Agent LLM Systems Fail?
- A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation
- Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- The BrowserGym Ecosystem for Web Agent Research
- Training Verifiers to Solve Math Word Problems
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
- WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
- Mano Technical Report
- Scaling Agents for Computer Use
- A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
- OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering