ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
Vincent Siu, Manasi Sharma, Dawn Song, Daniel Yue Zhang, Chenguang Wang, Ying Liu
cs.AI, cs.CL
Submitted: 2026-08-19
Updated: 2026-08-20
License: http://creativecommons.org/licenses/by/4.0/
The gist: Computer use agents are evaluated almost exclusively on atomic desktop tasks, but realistic desktop work requires sustaining state across multiple objectives.
Terminology
Abstract
Computer use agents are evaluated almost exclusively on atomic desktop tasks, but realistic desktop work requires sustaining state across multiple objectives. We study this gap with ChainWorld, which composes atomic OSWorld tasks into long horizon desktop workloads through directional compatibility search while preserving the source evaluators. The resulting workload contains 347 chains of length two to four and compares two renderings of the same task sequence. In single turn evaluation, all tasks are presented together in one prompt. In multi turn evaluation, tasks are revealed one at a time. Across four current computer use agents, maximum chain completion is 31%. Multi turn evaluation improves completion for three models, but both protocols remain challenging. The two protocols also expose different failure profiles. Single turn failures concentrate on artifact precision, while multi turn failures more often reflect session management problems such as fragmented progress and later turn disengagement.
Sources
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
- Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
- GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
- LLMs Get Lost In Multi-Turn Conversation
- ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
- Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
- Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents
- Autonomous Evaluation and Refinement of Digital Agents
- OpenAI GPT-5 System Card
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
- Kimi K2.5: Visual Agentic Intelligence
- UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
- OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows
- AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
- AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
- Agent-as-a-Judge: Evaluate Agents with Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems