Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
cs.CV, cs.AI
Submitted: 2026-10-05
Updated: 2026-10-05
Code: https://github.com/swaydy-n/Infinite-Dreamer
License: http://creativecommons.org/licenses/by/4.0/
The gist: Graphical User Interface (GUI) agents have emerged as a promising paradigm for automating complex digital workflows across diverse applications.
Terminology
Abstract
Graphical User Interface (GUI) agents have emerged as a promising paradigm for automating complex digital workflows across diverse applications. However, training highly capable and generalizable agents fundamentally relies on massive, high-fidelity visual-action trajectories, which are notoriously difficult to acquire. While human demonstrations are unscalable, existing GUI world models rely on text descriptions or HTML rendering, discarding crucial pixel-level visual details like icons and layout styles. To address this issue, we introduce Infinite-Dreamer, a simulation-free data synthesis method powered by a pixel-level Image Editing World Model. By conceptualizing GUI transitions as image editing tasks, we leverage Vision-Language Models (VLMs) to describe action-induced UI changes as structured delta-text. We then fine-tune an image editing backbone to controllably synthesize realistic screenshot transitions. We utilize this model to generate both single-frame visual robustness data and multi-step imaginary trajectories. To validate the effectiveness of our approach, we fine-tune the Qwen3-VL baseline solely on the synthesized data to obtain Infinite-Actor, and evaluate it on AndroidWorld, MobileWorld, and AndroidControl-Curated benchmarks. Infinite-Actor consistently outperforms the Qwen3-VL baselines across scales: Infinite-Actor-8B improves AndroidWorld Pass@1 by +4.45 and nearly doubles the MobileWorld Pass@3 success rate, while Infinite-Actor-2B improves Pass@1 by +9.05. Code is available at https://github.com/swaydy-n/Infinite-Dreamer.
Sources
- InstructPix2Pix: Learning to Follow Image Editing Instructions
- MobileDreamer: Generative Sketch World Model for GUI Agent
- GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
- GameGen-X: Interactive Open-world Game Video Generation
- AdaWorld: Learning Adaptable World Models with Latent Actions
- WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis
- GUIrilla: A Scalable Framework for Automated Desktop UI Exploration
- Mastering Atari with Discrete World Models
- Mastering Diverse Domains through World Models
- Efficient Agent Training for Computer Use
- MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning
- ViMo: A Generative Visual GUI World Model for App Agents
- AVID: Adapting Video Diffusion Models to World Models
- NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
- UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
- WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens
- UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models