HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

arXiv:2609.38008 · cs.CV · Submitted 2026-09-29 · Read on arXiv

cs.CV

Submitted: 2026-09-29

Updated: 2026-09-29

Comments: Project Page: https://zjureal.com/HybridCUA/ Code: https://github.com/ZJU-REAL/HybridCUA

Code: https://github.com/anthropics/claude-code

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 18× in this paper · explained in 1 episode
• Benchmark — 13× in this paper · explained in 2 episodes
• Action Space — 12× in this paper · explained in 1 episode
• Reinforcement Learning — 10× in this paper · explained in 4 episodes
• Fine-Tuning — 8× in this paper · explained in 5 episodes
• Verifier — 8× in this paper · explained in 2 episodes
• Verification — 7× in this paper · explained in 2 episodes
• GRPO — 6× in this paper · explained in 6 episodes
• Generalization — 4× in this paper · explained in 13 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Benchmarking — 3× in this paper · explained in 3 episodes
• Cooperation — 3× in this paper · explained in 2 episodes
• Online Reinforcement Learning — 3× in this paper · explained in 2 episodes
• Two-stage training — 3× in this paper · explained in 2 episodes
• Verifiable Rewards — 3× in this paper · explained in 2 episodes
• Credit assignment — 2× in this paper · explained in 3 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Visual Grounding — 2× in this paper · explained in 2 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• On-Policy Distillation — 1× in this paper · explained in 3 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Task Planning — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• Qwen3-VL Technical Report
• RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
• ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
• ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
• WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
• UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
• SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
• OpenCUA: Open Foundations for Computer-Use Agents
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience
• MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models