GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments
cs.AI, cs.CV
Submitted: 2026-02-09
Updated: 2026-09-26
Comments: 24 pages, 11 figures. Project page: https://github.com/stepfun-ai/GEBench
Code: https://github.com/stepfun-ai/GEBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: An image generation model used as a graphical user interface (GUI) environment must produce not only a plausible screen but also the correct response to an action.
Terminology
Abstract
An image generation model used as a graphical user interface (GUI) environment must produce not only a plausible screen but also the correct response to an action. Existing visual-generation benchmarks do not directly assess this combination of functional correctness and temporal consistency in GUIs. We introduce GUI-GenBench, a benchmark of 700 curated samples across five task suites: single-step transitions, multi-step planning, fictional-app generation, real-app trajectories, and coordinate-based grounding. We also introduce GUI-Score, a VLM-based metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Evaluating 12 image generation models on Chinese and English subsets, we find that strong single-step performance does not consistently extend to coherent multi-step trajectories or accurate grounding. The highest aggregate GUI-Scores are 69.62 on the Chinese subset and 63.16 on the English subset. Qualitative analysis further identifies failures in text rendering, icon interpretation, and coordinate-conditioned transitions. Together, these results distinguish visual plausibility from functional reliability and identify limitations that must be addressed before generated GUIs can serve as dependable interaction environments.
Sources
- UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
- Qwen3-VL Technical Report
- Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Emerging Properties in Unified Multimodal Pretraining
- Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens
- ControlGUI: Guiding Generative GUI Exploration through Perceptual Visual Flow
- Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- GPT-4o System Card
- Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation
- MobileWorldBench: Towards Semantic World Modeling For Mobile Agents
- Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback
- Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
- Step1X-Edit: A Practical Framework for General Image Editing
- UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
- ViMo: A Generative Visual GUI World Model for App Agents
- WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Seedream 4.0: Toward Next-generation Multimodal Image Generation
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems