Interface Quilt: Modifying Applications with Computer Use Agents and Generative UI
cs.HC
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: Preprint, under review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Modifying an existing application is difficult when it does not provide programmatic access for that change.
Terminology
Abstract
Modifying an existing application is difficult when it does not provide programmatic access for that change. Recent advances in computer-use agents (CUAs) enable richer understanding and operation of existing interfaces, while generative UI (GenUI) makes it practical to generate new ones. We examine how these advances expand software modification through a design space of what can be changed and a framework for how modifications can be implemented. We then introduce Interface Quilt, a system that combines CUAs and GenUI to generate interface patches, which are new interfaces displayed over an existing application to modify its presentation, interaction, and capabilities. Through demonstrations across a variety of applications, we show that interface patches support a range of modifications without source code, APIs, or accessibility data.
Sources
- Revibing Code from Papers: Reimplementing HCI Artifacts
- Agent S: An Open Agentic Framework that Uses Computers Like a Human
- TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces
- HIDAgent: A Toolkit Enabling "Personal Agents" on HID-Compatible Devices
- Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents
- Generative Interfaces for Language Models
- The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and Beyond
- BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks
- Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation
- Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpages
- Generative UI: LLMs are Effective UI Generators
- ALLOY: Generating Reusable Agent Workflows from User Demonstration
- TaskLens: Generating Task-Conditioned Scaffolded Interfaces for Learning Professional Creative Software
- OmniParser for Pure Vision Based GUI Agent
- Making Abstraction Concrete: A Design Space and Interaction Model of Abstraction in Interactive Systems
- Magentic-UI: Towards Human-in-the-loop Agentic Systems
- Morae: Proactively Pausing UI Agents for User Choices
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- CI4A: Semantic Component Interfaces for Agents Empowering Web Automation
- From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems