Computer-Use Agents Gain Little Benefit from a World Model When Conditioned on Tutorials
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/realjoenguyen/RAGPJ-official
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent work has improved computer-use agents (CUAs) by combining retrieved task-relevant tutorials and look-ahead planning using world models with MLLM judges.
Terminology
Abstract
Recent work has improved computer-use agents (CUAs) by combining retrieved task-relevant tutorials and look-ahead planning using world models with MLLM judges. However, the contribution of each component -- tutorials, the judge, and the world model -- to overall improvement has not been evaluated in isolation. To fill this gap, we first construct OSWorld-Tutorial and ScienceBoard-Tutorial: two datasets for CUAs with curated multimodal tutorials, covering everyday and scientific domains. We then propose RAG-PolicyJudge (RAG-PJ), a policy-judge framework for CUAs conditioned on task-specific tutorials: the policy proposes candidate actions, and the judge selects among them. To isolate the effect of world models given tutorials, we compare RAG-PJ with and without common world models: using either zero-shot text-based descriptions of next states or an oracle that returns ground-truth visual observations. We find that RAG-PJ gains little benefit from these rollouts when tutorials are present, even with perfect visual futures. This suggests prior world-model gains in tutorial-conditioned methods come mainly from tutorial understanding and the judge's ability to select the proposed actions, rather than from predicted rollouts. Error analysis reveals that the bottleneck is not rollout fidelity, but the judge's ability to reason over future states and recognize task completion. This highlights the need to improve not just next-state prediction in world models, but also reasoning over their predicted rollouts. Datasets and code are available at https://github.com/realjoenguyen/RAGPJ-official.
Sources
- Qwen3-VL Technical Report
- MobileDreamer: Generative Sketch World Model for GUI Agent
- WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning
- TimeWarp: Evaluating Web Agents by Revisiting the Past
- Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding
- MobileWorldBench: Towards Semantic World Modeling For Mobile Agents
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- Beyond Next-Screen Prediction: The Decision Value of GUI World Models
- UI-Venus-1.5 Technical Report
- How Mobile World Model Guides GUI Agents?
- EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience
- Step-GUI Technical Report
- AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
- Mobile-Agent-v3: Fundamental Agents for GUI Automation
- Code2World: A GUI World Model via Renderable Code Generation
- Qwen-AgentWorld: Language World Models for General Agents
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning