Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

This paper proposes a decoupled intent-driven grounding framework for GUI (Graphical User Interface) grounding that addresses the core dilemma of translating abstract user instructions into precise element coordinates. The authors identify a persistent dual obstacle in existing approaches: conventional grounding models lack semantic richness to interpret abstract instructions, while end-to-end multimodal large language models (MLLMs) suffer from coordinate hallucinations caused by deficient fine-grained perception.

The paper states: "GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception."

The authors note that existing approaches each encounter fundamental bottlenecks:

• Methods parsing interface metadata (HTML DOM trees, Android view hierarchies) are often inapplicable in sandboxed or secure environments

• End-to-end MLLMs like CogAgent and SeeClick directly predict coordinates but their visual encoders lack inductive biases for GUI layout regularities, leading to persistent coordinate hallucinations

• Traditional grounding models such as Grounding DINO suffer from weak text understanding and no spatial layout perception

The proposed framework is regression-free, where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression.

The design must simultaneously:

1. predict precise pixel coordinates without ever learning a coordinate regression mapping

2. internalize the basic layout primitives of a GUI—text blocks and icons—without requiring instance-level semantic labels

3. handle spatially modified queries ('upper-right,' 'left side') without per-query spatial annotations, while remaining robust when spatial information is irrelevant

Stage 1: Instruction Elaboration — A frozen-parameter MLLM (e.g., Qwen-VL, GPT-4V) serves as an intent parser. The paper explains: the MLLM receives the current screenshot I and the user instruction U, and is guided via in-context prompt engineering to produce a structured visual description Q. For example, click the login button is concretized as a blue rectangular button labeled 'Login' located in the center of the page.

Stage 2: Layout-Prior Candidate Region Generation — A GUI-domain-adapted DINO detector is fine-tuned using only a small set of screenshots annotated with the two most basic layout types—'Text' and 'Icon'. The detector learns to distinguish text blocks from icon blocks without involving specific semantic categories.

Stage 3: Hallucination-Resistant Cross-Modal Feature Matching — A frozen CLIP (ViT-B/32) is used as the feature extractor. For each candidate region, the image is cropped and visual features are extracted, while the visual query is encoded as text features. Cosine similarity is computed, and the candidate region with the highest similarity is selected as the grounding result.

Lightweight Geometric Feature Fusion — For spatially sensitive queries, a learnable single-layer linear projection maps geometric vectors to spatial offsets: hi = Wg gi + bg where Wg ∈ R dc×6 and bg ∈ R dc are the only learnable parameters. The offset is additively fused into the original visual feature: f̃i = fi + hi.

• Training Data: Approximately 200k screenshots sampled from widely used GUI datasets; automated parsing tools extract textual and iconic regions; Qwen3-VL generates visual descriptions for each region

• Stage 1 (DINO detector): Swin-Tiny backbone initialized from COCO 12-epoch weights, trained on 8 RTX 3090 GPUs with batch size 8 for 50 epochs, using AdamW with learning rate 1×10−4

• Stage 2 (Matching): CLIP ViT-B/32 frozen, only geometric projection matrix Wg and bias bg trained, using Adam with learning rate 1×10−3 and batch size 256 for 20 epochs

• Loss Functions: Detection loss Ldet = λreg Lreg + λgiou Lgiou + λcls Lcls with λreg=5.0, λgiou=2.0, λcls=2.0; contrastive loss (InfoNCE) with temperature τ=0.07

The method achieves 87.1% average accuracy across Mobile, Desktop, and Web platforms, outperforming all compared methods including UGround (73.3%), SeeClick (53.4%), Qwen2.5VL-72B (22.2%), GPT-4o (16.6%), CogAgent (5.7%), and Fuyu (5.7%).

The method achieves 41.3% average score across six specialized domains (Dev, Creative, CAD, Scientific, Office, OS), compared to OS-Atlas-7B (18.9%), UGround-7B (16.5%), ShowUI (7.7%), CogAgent (7.7%), MiniCPM-V (3.0%), SeeClick (1.1%), and GPT-4o (0.6%).

The framework achieves an overall score of 67.6%, improving 7.9% over the best baseline (SeeClick at 59.3%) and 24.2% over Qwen2.5VL-72B alone (43.4%).

The method achieves substantial gains in Element Accuracy and Step Success Rate across all task configurations (Cross-Task, Cross-Website, Cross-Domain), with Element Accuracy of 44.9%, 43.8%, and 40.1% respectively, compared to Qwen2.5VL-72B at 27.7%, 25.6%, and 24.8%.

The paper lists three main contributions:

1. Layout-Aware GUI Grounding Model: By replacing coordinate regression with frozen cross-modal matching, this model fundamentally prevents coordinate hallucinations while delivering precise pixel-level localization.

2. Layout-prior injection and lightweight geometric fusion mechanism: Using only a small set of Text/Icon-level annotations, endows the grounding model with both text semantic understanding and spatial layout perception.

3. Decoupled intent-driven grounding framework: A frozen MLLM handles instruction parsing and our model performs precise localization... achieving leading performance on multiple benchmarks while avoiding the high costs of full fine-tuning.

The paper concludes: "Our approach resolves the core dilemma in GUI grounding by decoupling instruction elaboration from precise localization... On ScreenSpot-Pro and Mind2Web, our method achieves over 20% and 15% improvement respectively over end-to-end state-of-the-art systems, with especially large gains on spatially sensitive queries—demonstrating a practical path toward hallucination-free GUI grounding."

Improvements to AI Systems:

1. Hallucination-Free GUI Grounding via Decoupled Architecture: Replace end-to-end coordinate regression in MLLMs with a three-stage pipeline: (a) frozen MLLM parses abstract instructions into structured visual descriptions, (b) a GUI-adapted DINO detector generates candidate regions using only Text/Icon layout priors, (c) frozen CLIP performs cross-modal feature matching to select the best region. This eliminates coordinate hallucinations entirely while retaining semantic richness.

2. Layout-Prior Injection Without Semantic Labels: Train a detector on only two basic layout categories (Text, Icon) using 200k screenshots. This internalizes GUI layout regularities (e.g., button shapes, text block boundaries) without requiring instance-level annotations, making it applicable to unseen domains and sandboxed environments where metadata is unavailable.

3. Lightweight Geometric Feature Fusion for Spatial Queries: Add a single learnable linear projection (Wg, bg) that maps 6D geometric vectors (e.g., position, size, aspect ratio) to spatial offsets, fused additively into CLIP visual features. This enables handling of spatially modified queries (upper-right, left side) without per-query spatial annotations, while remaining robust when spatial info is irrelevant.

4. Regression-Free Precision: Achieve pixel-level localization by selecting from candidate regions via cosine similarity in CLIP embedding space, rather than predicting coordinates. This removes the need for coordinate regression training, reducing overfitting and improving generalization across platforms (Mobile, Desktop, Web).

5. Cost-Efficient Adaptation: Keep MLLM (e.g., Qwen-VL) and CLIP frozen; only train the DINO detector (Swin-Tiny) and the geometric projection matrix. This reduces fine-tuning costs by orders of magnitude compared to full MLLM fine-tuning, while outperforming 72B-parameter end-to-end models.

What the Improved AI System Can Do:

• Operate on raw screenshots in any environment (including secure/sandboxed) without parsing HTML or view hierarchies, achieving 87.1% accuracy on ScreenSpot (vs. 22.2% for Qwen2.5VL-72B).

• Handle abstract instructions like click the login button by first elaborating them into concrete visual descriptions (e.g., blue rectangular button labeled 'Login' in center), then grounding via layout-aware detection and CLIP matching.

• Resolve spatial queries (the icon to the upper-right of the search bar) with 41.3% accuracy on ScreenSpot-Pro (vs. 18.9% for OS-Atlas-7B), a 20%+ improvement over end-to-end systems.

• Execute multi-step tasks on Android (AITW) with 67.6% overall score (7.9% better than SeeClick) and on web navigation (Mind2Web) with Element Accuracy up to 44.9% (vs. 27.7% for Qwen2.5VL-72B).

• Deploy at scale with minimal computational overhead: only ~200k screenshots for detector training and a single linear layer for geometric fusion, making it feasible for edge devices and real-time GUI agents.

• Generalize across domains (Dev, Creative, CAD, Scientific, Office, OS) without retraining, thanks to layout-prior injection and frozen cross-modal matching that avoids domain-specific coordinate regression.

GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.