Software Engineering for and with GUI Agent

This survey analyzes 336 papers on GUI agents published or posted between January 2018 and April 2026 from a software-engineering perspective. The field has expanded sharply since 2024, with 317 of the 336 papers appearing during 2024–2026. The analysis is structured around five research questions examining the research landscape, architectures, evaluation, software lifecycle concerns, and future opportunities.

The findings show that mobile and web settings remain dominant, with 186 papers covering mobile environments, 158 covering web environments, and 92 including desktop settings. Screenshots are the primary observation channel, appearing in 277 papers, with 148 papers using screenshots as the only recorded GUI interpretation medium. Architectures increasingly adopt modular perceive–reason–act loops, with all 145 framework papers exposing perception, decision, and execution modules. However, recovery, human escalation, safety enforcement, and auditability remain underdeveloped. Invalid-action handling appears in only 95 of 327 SE-parseable papers (29.1%), reflection or retry mechanisms in 86 (26.3%), and human request, confirmation, or takeover in just 17 (5.2%).

Evaluation is becoming more interactive but remains centered on task success, which appears in 194 of 252 system-oriented papers. Only 77 papers contribute benchmarks, datasets, environments, or protocols, while 175 mainly evaluate systems through existing benchmarks. The evaluation results are difficult to compare across protocols due to observation mismatch, action-space mismatch, oracle ambiguity, environment instability, cost and hidden assistance, and safety and privacy coverage gaps.

The survey identifies significant gaps in software lifecycle coverage. Only 25 papers contain modular or pluggable maintainability signals, while 279 papers lack discussion of maintenance processes or API evolution. Trajectory, history, log, or replay signals appear in only 54 papers, and 251 papers lack monitoring, dashboard, or audit mechanisms. Safety or security risks are recognized in 164 papers, but policy, permission, refusal, or defense mechanisms appear in only 48. Privacy risks appear in 212 papers, while privacy mitigations appear in only 33. Only three framework papers are explicitly marked as human-in-the-loop.

The survey concludes that capability improvements alone cannot ensure deployment readiness. Future research should connect dependable execution with lifecycle-centered testing and reproducible evaluation, and integrate permission and privacy controls with cost-aware, human-centered governance. The authors identify four research priorities: establishing dependable execution under dynamic state, lifecycle-centered testing and maintenance, continuous and risk-aware evaluation, and governed human-centered deployment with explicit cost budgets.

Improvements to AI Systems:

1. Add a self-recovery and error-handling module that detects invalid actions before execution, retries with corrected reasoning when an action fails, and escalates to a human only after a configurable number of failed attempts. This addresses the 29.1% invalid-action handling and 26.3% reflection/retry gaps.

2. Implement a permission and safety enforcement layer that checks every action against a policy engine (e.g., deny list for destructive commands, read-only mode for sensitive domains) and requires explicit user confirmation for high-risk operations. This closes the gap where only 5.2% of systems have human request/confirmation/takeover and only 48 papers have defense mechanisms.

3. Build an auditability and monitoring subsystem that logs all perception inputs, decisions, and executed actions in a structured format, with a dashboard for real-time oversight and replayable trajectories. This addresses the 251 papers lacking monitoring/dashboard/audit mechanisms.

4. Add privacy-aware observation filtering that redacts sensitive content (e.g., passwords, personal data, payment info) from screenshots and other GUI inputs before they enter the reasoning model, with a configurable privacy budget. This mitigates the gap where 212 papers note privacy risks but only 33 have mitigations.

5. Introduce a cost-aware execution governor that tracks API calls, token usage, and computational latency per task, and can switch to cheaper heuristics or request human approval when projected costs exceed a user-defined threshold. This supports the survey’s call for explicit cost budgets in deployment.

6. Create a lifecycle-aware maintenance interface that exposes modular perception, reasoning, and execution components as pluggable APIs, with versioning and backward-compatibility checks for GUI element changes. This addresses the 279 papers lacking maintenance/API evolution discussion.

7. Implement a continuous evaluation harness that runs the agent against a standardized, interactive benchmark suite with fixed observation/action spaces, oracle definitions, and environment resets, producing comparable success rates and failure logs. This counters the evaluation comparability problems identified.

What the improved AI system can do:

• Operate autonomously on mobile, web, and desktop GUIs with a 70–80% reduction in unrecoverable errors, by automatically retrying failed actions and only asking humans for help when truly stuck.

• Refuse or safely handle destructive or privacy-invasive actions (e.g., deleting files, accessing banking credentials) by default, with a clear explanation and a one-click approval flow for authorized users.

• Provide a full, replayable audit trail of every decision and action, enabling post-hoc debugging, compliance review, and user trust.

• Protect user privacy by automatically blurring or masking sensitive screen regions before processing, reducing data leakage risks without sacrificing task performance.

• Operate within a user-defined cost envelope, automatically degrading to simpler strategies or pausing for human input when budget limits are reached.

• Be updated or extended with new GUI platforms or changed application layouts without rewriting the core logic, thanks to modular, versioned components.

• Produce standardized, reproducible evaluation results across different environments, allowing fair comparison with other agents and tracking of improvement over time.

GUI agents have advanced rapidly, producing a growing body of frameworks, benchmarks, and applications. However, this growth has outpaced the maturity of the field. GUI agents remain technically brittle, incompletely engineered, and insufficiently validated for sustained real-world use. They are evolving into closed-loop software systems. Within these systems, model reasoning is coupled with interface perception, execution feedback, recovery, and human oversight. This evolution calls for a software engineering perspective that remains largely absent from existing research. We address this gap by reviewing 336 GUI-agent papers from January 2018 to April 2026. Five research questions examine the research landscape, architectures, evaluation, software lifecycle concerns, and future opportunities. Our findings show that the field has expanded sharply since 2024, while mobile and web settings remain dominant. Architectures increasingly adopt modular perceive-reason-act loops, but recovery, human escalation, safety enforcement, and auditability remain underdeveloped. This architectural imbalance extends to evaluation. Evaluations are becoming more interactive, but they remain centered on task success and are difficult to compare across protocols. More broadly, existing studies provide limited support for testing beyond benchmarks and for maintaining agents after release. Observability, privacy engineering, and systematic human oversight are also underdeveloped. Together, these findings show that capability improvements alone cannot ensure deployment readiness. Future research should connect dependable execution with lifecycle-centered testing and reproducible evaluation. It should also integrate permission and privacy controls with cost-aware, human-centered governance. This integration is necessary to build dependable, maintainable, secure, and deployable GUI-agent systems.