Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions

The gist The AIDE framework enables robots to explore and act in unfamiliar environments under ambiguous human instructions by interactively identifying task-relevant objects, which addresses challenges in efficient reasoning and environmental interaction for existing vision-language model (VLM)-based methods.

How it works

AIDE is a dual-stream framework that integrates interactive exploration with vision-language reasoning, where Multi-Stage Inference (MSI) serves as the decision-making stream and Accelerated Decision-Making (ADM) as the execution stream, enabling zero-shot affordance analysis and interpretation of ambiguous instructions <ref:2602.05273#pg2> The MSI stream is invoked once for key decision-making when the task is novel or ADM fails to identify any valid tool-related object in the updated scene (by applying a threshold based on the detection confidence produced and similarity matching), producing the task planning result stored for subsequent ADM retrieval <ref:2602.05273#pg4> The ADM stream enables closed-loop, real-time task execution by continuously performing task planning and updating the scene through robot interaction <ref:2602.05273#pg5>

Key Components and Processes

The framework consists of four key parts:

1. Multimodal Chain-of-Thought (CoT) Module: This module jointly leverages GPT-5’s multimodal reasoning capability and the grounding submodule’s localization ability to produce the task grounding result, including labels, images and locations for the required tool and its critical parts <ref:2602.05273#pg4>

2. Instruction-Tool Relationship Space: This space establishes task-tool associations to guide analysis of critical object affordances through content construction based on the MMCoT and relationship construction derived from affordance analysis <ref:2602.05273#pg5>

3. Efficient Retrieval Scheme (ERS): The ERS performs efficient matching between the input and retrieved candidate task planning results obtained via affordance-based retrieval from the Instruction-Tool Relationship Space, outputting either a complete task grounding result or retrieved candidate task planning results <ref:2602.05273#pg5>

4. Interaction Exploration Policy: This policy localizes a rectangular region likely containing the tool to support subsequent approach and execution when the required tool is distant, occluded, or barely visible in complex scenes <ref:2602.05273#pg5>

Affordance Analysis and Decision Making

The Instruction-Tool Relationship Space is built through two key processes: content construction based on the MMCoT and the Exploration Policy, and relationship construction derived from affordance analysis <ref:2602.05273#pg5> For relationship construction, GPT-5 performs affordance scoring over X dimensions for each instruction and a randomly sampled tool image to yield X-dimensional affordance vectors, which are then clustered to induce many-to-many instruction-tool, instruction-instruction, and tool-tool relationships <ref:2602.05273#pg5> The 19 dimensions used for scoring capture key appearance and functional traits of tools <ref:2602.05273#pg5>

Performance and Validation

AIDE achieves a task-level decision frequency of 10 Hz and over 80% accuracy across 400 test samples, surpassing state-of-the-art task planning methods <ref:2602.05273#pg5> The framework demonstrates superior exploration capability, exceeding 60% success rates on tool-absent scenes compared to other baselines <ref:2602.05273#pg5> Furthermore, AIDE achieves nearly 10 Hz closed-loop planning by combining affordance analysis with the interactive exploration policy <ref:2602.05273#pg5> Real-world experiments show over 95% success on valid frames across six novel tasks, highlighting the framework’s closed-loop robustness in real-world settings <ref:2602.05273#pg5>

Error Mitigation Strategies

AIDE employs dedicated mitigation strategies for each type of error; for planning-level errors arising from GPT-5 hallucinations, AIDE incorporates human-in-the-loop interactions, including tool reselection and further exploration <ref:2602.05273#pg5> For perception-level errors caused by visual degradation, the system relies on the fact that such errors account for only 0.3-3.9% of valid frames and do not impact overall task completion <ref:2602.05273#pg5> The cross-verification between GPT5 and the segmentation model SAM2 significantly improves SAM2’s segmentation performance in the experimental setting <ref:2602.05273#pg5>

Conclusion

AIDE represents a substantial advancement in robot task planning, facilitating future research on efficiently handling ambiguous instructions in complex and dynamic environments <ref:2602.05273#pg5> The framework successfully combines MSI and ADM to ensure efficient and reliable tool selection <ref:2602.05273#pg5>.

--- Page 1 ---

Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions

Hengxuan Xu1,2,∗ Fengbo Lan1,2,∗ Zhixin Zhao1 Shengjie Wang3 Mengqiao Liu1 Jieqian Sun1 Yu Cheng4† Tao Zhang1†

--- Page 2 ---

I want to drink water. tool kind critical parts instruction Unseen Region Detection Generated Real Augmentation Scene Dataset I want to chop bones. Instruction Dataset Relationship Space with Affordance unseen region mug body(functional region) handle(operational region) ●: mug's handle ●: mug's body Task Planning: How to complete the instruction in the scene? Task Grounding Result Efficient Retrieval Scheme (ERS) Relationship Space I want to drink water. I want to enjoy hot beverages. Invisible Exploration No Exploration Exploration Policy: Multi-Stage Inference (MSI): Accelerated Decision-Making (ADM)

--- Page 3 ---

or optimization-based methods [Sutton and Barto, 2018; Toussaint, 2015] the advent of machine learning [Ichter et al., 2020; Xie et al., 2023] has reshaping this domain. The introduction of Visual Language Action (VLA) methods [Kim et al., 2024; Wen et al., 2025] has further shifted the focus toward end-toend task execution

--- Page 4 ---

Task Planning: How to complete the instruction in the scene? Task Grounding Result Efficient Retrieval Scheme (ERS) Relationship Space I want to drink water. I want to enjoy hot beverages. invisible exploration No Exploration Exploration Policy: Multi-Stage Inference (MSI): Accelerated Decision-Making (ADM)

--- Page 5 ---

slow, limiting the applicability of VLMs to real-world task planning and execution To address these challenges, we propose AIDE, a novel vision-language task planning framework featuring a crosscoupled dual-stream mechanism for reasoning and decisionmaking (Figure 2) AIDE provides an efficient closed-loop planning approach based on interactive exploration <ref:2602.05273#pg5>

--- Page 6 ---

Method Qwen3-Next-80B Llama3-70B GPT-5 Magma-8B Robopoint wentao yuan/robopoint v1 vicuna v1.5-13b 13B SAM2 facebook/sam2.1-hiera large 856M MobileCLIP apple/MobileCLIP-B-LT 599M yolov8x-worldv2 yolov8l-worldv2

--- Page 7 ---

AIDE (w/o ADM) and AIDE (w/o MSI) represent AIDE variants that rely solely on the designed MSI and ADM, respectively, while AIDE (Full) represents the complete framework comprising both MSI and ADM Table 1: Zero-shot performance comparison on G-Dataset R-Dataset

--- Page 8 ---

Table 1: Zero-shot performance comparison on G-Dataset R-Dataset AIDE (w/o ADM) 90.5 6.5 1.0 0.5 0.0 0.05

--- Page 9 ---

Table 2: Ablation study on different components of AIDE AIDE (w/o MSI) - - - -

--- Page11 ---

Figure 8: Definition of the affordance scoring dimensions, encompassing both the appearance and functional traits of tools.

1. Bold header: Enhanced Ambiguous Instruction Understanding via Affordance-Aware Reasoning. This improved system can perform zero-shot affordance analysis and interpretation of ambiguous instructions by projecting results into the Instruction-Tool Relationship Space, enabling a sufficient cross-modal understanding of the input instruction.

2. Bold header: Robust Real-Time Closed-Loop Execution via Dual Stream Architecture. The system can achieve a closed loop, 10Hz Real-Time Task Execution and Input Update by alternating between the Multi-Stage Inference (MSI) stream for key decision-making and the Accelerated Decision-Making (ADM) stream for execution.

3. Bold header: Adaptive Interaction Strategy in Open Environments. The system can handle scenarios where tools are not directly visible by employing interactive exploration, specifically utilizing visible exploration when objects are detectable but distant, and invisible exploration to find unseen regions when the tool is occluded or undetected by YOLO-World.

4. Bold header: Mitigation of LLM Hallucinations through Principled Filtering. The system can reduce errors arising from GPT-5 reasoning by employing a principled pipeline to filter task planning results, mitigating hallucinations through human interaction for tool re-identification and further exploration.

5. Bold header: High Efficiency in Tool Selection via Affordance Retrieval. The system can ensure reliable tool selection by performing an affordance-based depth-first search (DFS) over clusters in the relationship space until selecting an instruction whose affordance vector lies within a Euclidean distance c of the input.