Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
cs.LG, cs.CV
Submitted: 2026-10-05
Updated: 2026-10-05
Code: https://github.com/aadcha/action-conditioned-gui-agent
Terminology
Sources
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- MolmoPoint: Better Pointing for VLMs with Grounding Tokens
- What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs
- AutoGLM: Autonomous Foundation Agents for GUIs
- OmniParser for Pure Vision Based GUI Agent
- TinyClick: Single-Turn Agent for Empowering GUI Automation
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
- Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning
- Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control
- Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
- GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
- Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
- GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks