Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
cs.HC, cs.AI
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: Accepted to the EMNLP 2026 Main Conference
Code: https://github.com/Satwikram/OLLA
License: http://creativecommons.org/licenses/by/4.0/
The gist: Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs.
Terminology
Abstract
Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.
Sources
- Qwen3-VL Technical Report
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
- The BrowserGym Ecosystem for Web Agent Research
- OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents
- WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
- ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
- ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
- OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness
- OpenAI GPT-5 System Card
- An Illusion of Progress? Assessing the Current State of Web Agents
- Capability Self-Assessment in Large Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- Fine-Tuning Language Models from Human Preferences
Related papers
- Calibrating LLM Explanations to Human Decisions with Reinforcement Learning
- Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization
- EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
- EvoDesign: Agentic Editable Diagram Creation via Design Expertise Evolution
- HAGI++: Head-Assisted Gaze Imputation and Generation
- Explanation User Interfaces: A Systematic Literature Review