Evidence-Gated Task and Motion Planning with Vision-Language Models

arXiv:2608.20084 · cs.RO, cs.AI · Submitted 2026-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Evidence-Gated Task and Motion Planning with Vision-Language Models".

Dev: Here's the gist of the paper. Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility <ref:2608.20084#pg2> under partial observability, the availability of goalrelevant objects may be uncertain <ref:2608.20084#pg2>.

Rosa: We propose Evidence Acquisition and Feasibility Gating (EAFG), a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution <ref:2608.20084#pg4>.

Dev: EAFG then applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt <ref:2608.20084#pg4>.

Rosa: Our experiments show that, in cooking tasks with ambiguous object use, EAFG improves recipe completion by discovering task-relevant objects before planning <ref:2608.20084#pg4>.

Paper summary: Rosa: So we're talking about this paper called Evidence-Gated Task and Motion Planning with Vision-Language Models. The main idea here is that robots doing long manipulation tasks from natural language instructions need to consider both the meaning of the instruction and whether they can actually physically do it, especially when things aren't fully observable or when they aren't sure if an object exists.

Dev: Right, so what this paper claims is that existing methods that combine Vision Language Models and Task and Motion Planning often generate subgoals relying on the VLM’s prior knowledge without any actual visual support. This leads to execution failures or things going wrong unexpectedly because the robot assumes something is there when it isn't one <ref:2608.20084#pg1>.

Taro: So basically, they are proposing a framework called Evidence Acquisition and Feasibility Gating, or EAFG, which actively gathers visual evidence through subgoals generated by the VLM and then uses Task and Motion Planning to execute those subgoals two <ref:2608.20084#pg1,Evidence Acquisition and Feasibility Gating>.

Rosa: And then after that execution, the VLM checks if what it sees is enough information to actually plan toward the final goal. If it is, they generate a task plan. If not, they decide whether to keep exploring for more evidence or just stop trying two <ref:2608.20084#pg1>.

Dev: The paper points out that existing approaches often assume a fully observable state of objects three four, and plans made under partial observability might ignore unobserved objects or try to use things that aren't actually there <ref:2608.20084#pg2>. This makes the plans inefficient and can lead to failures against the instruction two <ref:2608.20084#pg1>.

Taro: The paper says they propose a mechanism for actively verifying unobserved objects before planning two <ref:2608.20084#pg1>. They do this by generating subgoals to resolve missing information, like opening doors of closed storage spaces two <ref:2608.20084#pg1>.

Rosa: Each of these subgoals is solved by TAMP, executed by the robot, and those observations are then used to update the evidence state. The VLM then looks at that updated evidence to decide what's next two <ref:2608.20084#pg1>.

Dev: The VLM outputs a feasibility status. It can be ready to plan, need more evidence, or halt two <ref:2608.20084#pg1>. If it thinks the acquired evidence is enough for the goal, it's ready to plan. If not, but there are still unexplored areas relevant to verifying objects or conditions, it needs more evidence.

Taro: And if it decides that the acquired evidence isn't enough and further exploration won't help verify what they need, then the status is halt two <ref:2608.20084#pg1>. This is when Evidence Acquisition cannot verify the objects or conditions needed for the goal and more exploratory subgoals are unlikely to be useful.

Paper summary: Rosa: The experimental results show that in cooking tasks with ambiguous object use, EAFG actually improves recipe completion by discovering task-relevant objects before they even start planning four <ref:2608.20084#pg2>. It also helps when an instruction requires an absent object.

Dev: That's interesting, because the paper shows that for instructions requiring an absent object, EAFG promotes appropriate halt decisions and reduces repeated attempts to manipulate that object four <ref:2608.20084#pg2>.

Taro: I mean, it moves away from just blindly trying to execute a plan when something is missing or hidden. It makes the robot think about gathering information first two <ref:2608.20084#pg1>.

Rosa: So what this means for us in the real world is that instead of just handing a long instruction to a robot and hoping for the best, we build in this loop where it actively checks if it knows what it needs before committing to a full plan.

Dev: It shifts the planning from being purely symbolic and geometric to being adaptive based on real-time visual confirmation two <ref:2608.20084#pg1>. The paper suggests that this active information gathering is key when dealing with partial observability.

Taro: It changes how we think about failure modes because instead of failing silently or looping endlessly trying to grab something that isn't there, the system knows when it should stop and ask for more input.

Rosa: I think the authors are showing us a way to bridge that gap between high-level language understanding and low-level physical execution under uncertainty.

Dev: They are doing this by using VLM-generated exploratory subgoals coupled with TAMP execution, which is a hierarchical integration approach two <ref:2608.20084#pg1>. It's not just one system doing everything at once; it's a structured way to gather data first.

Taro: It seems like the core idea is that you use the VLM to propose what information you need, and then you use TAMP to physically get that information, and only then do you plan the final move two <ref:2608.20084#pg1>.

Rosa: So it's about making the robot more cautious when things are unclear in its environment. That caution is built into how it acquires evidence before committing to a sequence of actions.

Dev: The paper is about Evidence-Gated Task and Motion Planning with Vision-Language Models, and it highlights how integrating active visual evidence acquisition with TAMP execution can make long-horizon manipulation more robust under uncertainty two <ref:2608.20084#pg1,Evidence-Gated Task and Motion Planning with Vision-Language Models>.

Paper summary: Taro: It suggests that for complex tasks, the system needs to decide between planning, gathering more evidence, or halting at each step based on what it sees.

Rosa: That's a pretty concrete way to handle the ambiguity of natural language instructions when the world doesn't look exactly like we expect it to be.

Dev: It shows that by incorporating this gating logic, you can actively discover hidden objects before committing to a plan, which improves completion rates in tasks with ambiguous object use four <ref:2608.20084#pg2>.

Taro: So for anyone listening, the implication is that future manipulation systems won't just rely on what they were told; they will actively probe the environment to confirm feasibility two <ref:2608.20084#pg1>.

Rosa: It makes sense. We're moving toward systems that are less reliant on perfect initial world models and more reliant on continuous, directed exploration when things get tricky.

Dev: The authors are showing how this framework handles the scenario where an instruction requires an object that is absent, leading to better halt decisions instead of just trying repeatedly four <ref:2608.20084#pg2>.

Taro: So the system learns when it's hitting a dead end and stops trying to manipulate something that isn't there, which is a smart way to handle those failures.

Rosa: It seems like the main point is that this EAFG framework gives robots a way to reason about both what they need to do and whether they have the right information before they start moving two <ref:2608.20084#pg1>.

Dev: Exactly. The mechanism of generating subgoals for resolving missing information, solving them with TAMP, and then gating based on the acquired evidence is how they manage that uncertainty in real-time two <ref:2608.20084#pg1>.

Taro: It’s about making the system autonomous enough to decide when it needs to pause its forward progress and seek more data before continuing with a task plan two <ref:2608.20084#pg1>.

Rosa: So for listeners interested in robotics, this paper points toward a future where these systems are much better at handling the messy reality of everyday environments.

Dev: The title, Evidence-Gated Task and Motion Planning with Vision-Language Models, really captures the essence of what they're proposing: using evidence acquisition and feasibility gating to guide task planning with VLMs two <ref:2608.20084#pg1,Evidence-Gated Task and Motion Planning with Vision-Language Models>.

Taro: It’s not just about making the robot faster; it’s about making it more reliable when the environment doesn't cooperate perfectly with the initial plan two <ref:2608.20084#pg1>.

Rosa: That’s what I think. It moves us closer to having robots that can truly follow natural language instructions in complex, messy settings.

Conclusion: Rosa: So, this paper is about Evidence-Gated Task and Motion Planning with Vision-Language Models. It boils down to robots actively gathering visual proof before they start planning any action at all, which we call feasibility gating one.

Dev: Right, so instead of the robot just guessing what it needs based on the language instruction, it makes subgoals to get visual evidence first two.

Taro: It means when a robot tries to do something complex from a natural language command, it doesn't just plan and hope; it probes the environment for missing objects or conditions two.

Rosa: The authors are showing how this process improves task completion rates in messy situations, like cooking where objects might be ambiguous four.

Dev: They also show that when an instruction requires something that isn't there, this gating logic helps the robot stop trying to manipulate it and just halt four.

Taro: The core idea is using the VLM to generate exploratory subgoals, solving them with TAMP, and then using a feasibility gate to decide if they should plan further or stop two.

Rosa: It moves us toward systems that are much more cautious about what they know versus what they need to know one.

Dev: This isn't just about making the robot faster; it's about making it more reliable when the environment doesn't look exactly like we expect two.

Taro: So, instead of failing silently or looping endlessly trying to grab something that isn't there, the system knows when to pause and seek more data four.

Rosa: It seems like this framework gives robots a structured way to reason about both what they need to do and whether they have the right information before they start moving two.

Dev: If you look at the authors, this is work by Kumar, Shen, Ramos, Fox, Lozano-Perez, and Kaelbling from two thousand nineteen and two thousand twenty four.

Taro: And those earlier works have a lot of stuff on combined task and motion planning under partial observability six.

Rosa: It suggests that for complex tasks, the system needs to decide between planning, gathering more evidence, or halting at each step based on what it sees two.

Dev: This active information gathering is key when dealing with partial observability in real-world manipulation two.

Taro: So this paper is about making the robot more robust under uncertainty by incorporating this gating logic into the planning process one.

Waseda University · Flinders University

cs.RO, cs.AI

Submitted: 2026-08-20

Updated: 2026-10-29

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility <ref:2608.20084#pg2> under partial

Key concepts

Evidence Acquisition
This mechanism involves the robot executing exploratory subgoals based on VLM suggestions. These actions collect visual observations from the environment. The VLM then analyzes these images to update its understanding of what is present or missing, iteratively gathering necessary environmental information for task planning.
Feasibility Gating Logic
This is a decision-making process where the VLM assesses the collected evidence against the natural language goal. It outputs one of three statuses: 'ready to plan', 'need more evidence', or 'halt'. This gate controls whether the robot should proceed with planning, continue exploring for more information, or stop attempting actions.
TAMP-based Execution
Task and Motion Planning (TAMP) is used to solve subgoals generated by the VLM. TAMP combines symbolic reasoning about the task structure (what needs to be done) with geometric planning (how to move the robot). This ensures that each subgoal is translated into a sequence of low-level actions that are physically possible and collision-free.
Vision-Language Models (VLMs)
VLMs integrate visual data and language understanding. They interpret natural language instructions to generate high-level subgoals, such as 'open the door.' They also evaluate collected images to determine if the current evidence is sufficient for planning, allowing the system to reason about semantic meaning in a visual context.

Terminology

Summary

Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility <ref:2608.20084#pg2> under partial observability, the availability of goalrelevant objects may be uncertain <ref:2608.20084#pg2>. We propose Evidence Acquisition and Feasibility Gating (EAFG), a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution <ref:2608.20084#pg4>. EAFG then applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt <ref:2608.20084#pg4>. Our experiments show that, in cooking tasks with ambiguous object use, EAFG improves recipe completion by discovering task-relevant objects before planning <ref:2608.20084#pg4>. For instructions requiring an absent object, EAFG promotes appropriate halt decisions and reduces repeated attempts to manipulate that object <ref:2608.20084#pg4>.

Problem Formulation

Robots operating in everyday environments must be able to execute complex, long-horizon manipulation tasks from natural-language instructions <ref:2608.20084#pg2>. Such tasks require the robot to interpret the semantic meaning of the language instruction while satisfying geometric and kinematic constraints, such as grasp feasibility, placement stability, and collision-free reachability <ref:2608.20084#pg2>. Task and Motion Planning (TAMP) has therefore been widely used as a core technique for long-horizon manipulation planning, as it can generate action sequences that account for the geometric and kinematic aspects of the problem <ref:2608.20084#pg2>. Vision-Language Models (VLMs) can integrate visual and linguistic information to infer commonsense procedures and semantically meaningful intermediate goals from natural-language objectives <ref:2608.20084#pg2>. A hierarchical integration in which a VLM generates high-level subgoals and TAMP refines each subgoal into a geometrically feasible action sequence is a promising approach to natural-language long-horizon robot manipulation <ref:2608.20084#pg2>.

Evidence Acquisition Mechanism

EAFG first generates subgoals for resolving missing or uncertain information, such as opening the doors of closed storage spaces <ref:2608.20084#pg4>. Each subgoal is solved by TAMP, executed by the robot, and used to collect observation images <ref:2608.20084#pg4>. The VLM then evaluates whether the acquired observations provide sufficient information for planning toward the goal <ref:2608.20084#pg4>. If sufficient evidence has been collected, the system generates a task plan conditioned on this information <ref:2608.20084#pg4>.

Evidence Acquisition is designed to collect the environmental information required to determine whether the natural-language goal can be achieved <ref:2608.20084#pg4>. It proceeds as an iterative loop, indexed by t = 0, 1,..., in which the robot executes exploratory subgoals and the VLM updates its understanding of the environment from the resulting observations <ref:2608.20084#pg4>. At iteration t, the VLM generates evidence-acquisition subgoals G EA t from the language goal, current observations, and accumulated evidence state to inspect potentially relevant regions before any task-directed subgoals are produced <ref:2608.20084#pg4>. These subgoals are passed to the TAMP module, which attempts to instantiate each subgoal as a geometrically and kinematically feasible low-level action sequence <ref:2608.20084#pg4>. After each subgoal execution, the resulting observation image is recorded <ref:2608.20084#pg4>. The VLM updates the evidence state from Et to Et+1 based on the observation collage It, allowing outdated or contradicted descriptions to be removed rather than blindly appended <ref:2608.20084#pg4>.

Feasibility Gating Logic

Feasibility Gating determines whether the evidence acquired through Evidence Acquisition is sufficient for planning toward the natural-language goal <ref:2608.20084#pg4>. At each iteration t, the VLM receives the goal G eng, the current evidence state Et, and the observation collage It as input, and outputs a feasibility status st: st ∈ ready to plan, need more evidence, or halt <ref:2608.20084#pg4>. If the VLM determines that the acquired evidence is sufficient to plan for achieving G eng, it outputs st = ready to plan <ref:2608.20084#pg4>. If the VLM determines that the current evidence is still insufficient but that unexplored regions relevant to verifying the required objects or conditions remain, it outputs st = need more evidence <ref:2608.20084#pg4>. The status st = halt is selected when Evidence Acquisition cannot verify the objects or conditions needed for G eng, and further exploratory subgoals are unlikely to help <ref:2608.20084#pg4>.

Experimental Results

The experiments use a kitchen environment for making chicken soup <ref:2608.20084#pg5>. The evaluation is conducted across three settings: (a) Initial kitchen state, (b) Object layout in the kitchen <ref:2608.20084#pg6>. Setting S1 evaluates RQ1, where EAFG increases Recipe CR from 0.20 to 0.45 for GPT-5.5 and improves the step completion rates for pepper and salt to 1.00 <ref:2608.20084#pg8>. Setting S2 evaluates RQ2, showing that with EAFG, Recipe CR increases to 1.00 for GPT-5.5 from 0.35 without EAFG. Setting S3 evaluates RQ3, where EAFG increases Halt SR from 0.45 to 0.90 for GPT-5.5 and reduces Missing-object Attempt Count from 4.00 to 0.55.

Conclusion

We proposed EAFG, a framework that actively acquires visual evidence before task planning and uses a feasibility gate to decide whether to plan, explore further, or halt. Experiments show that EAFG improves grounded task completion under partial observability by discovering task-relevant hidden objects before planning. EAFG also promotes appropriate halting when required objects cannot be confirmed, reducing unnecessary execution attempts toward absent objects.

References

[1] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel Combined task and motion planning through an extensible planner-independent interface layer In 2014 IEEE International Conference on Robotics and Automation ICRA pages 639–646 2014

[2] C. R. Garrett, T. Lozano-Perez, and L. P. Kaelbling Pddlstream Integrating symbolic planners and blackbox samplers via optimistic adaptive planning In Proceedings of the international conference on automated planning and scheduling volume 30 pages 440–448 2020

[3] Z. Yang, C. Garrett, D. Fox, T. Lozano-Perez, and L. P. Kaelbling Guiding long-horizon task and motion planning with vision language models In 2025 IEEE International Conference on Robotics and Automation ICRA pages 16847–16853 IEEE 2025

[4] N. Kumar, W. Shen, F. Ramos, D. Fox, T. Lozano-Perez, L. P. Kaelbling Combined task and motion planning under partial observability An optimization-based approach In 2019 International Conference on Robotics and Automation ICRA pages 9000–9006 2019

[5] C. R. Garrett, C. Paxton, T. Lozano-Perez, L. P. Kaelbling, and D. Fox Online replanning in belief space for partially observable task and motion problems In 2020 IEEE International Conference on Robotics and Automation ICRA pages 5678–5684 2020

[6] C. Phiquepal and M. Toussaint Combined task and motion planning under partial observability An optimization-based approach In 2019 International Conference on Robotics and Automation ICRA pages 9000–9006 2019

[7] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Perez Integrated task and motion planning Annual review of control robotics and autonomous systems 4(1)265–293 2021

[8] L. P. Kaelbling and T.

Improvements for AI systems

  1. Evidence Acquisition and Feasibility Gating (EAFG) can actively gather visual evidence through VLM-generated exploratory subgoals while TAMP refines each subgoal into a geometrically feasible action sequence. This allows the system to discover task-relevant objects before planning, improving recipe completion by grounding plans in verified observations rather than relying solely on prior knowledge.

  2. A feasibility gate mechanism can be implemented to decide whether to proceed with task planning, acquire further evidence, or halt based on acquired evidence. If the VLM determines that the evidence required for goal achievement is insufficient, the system halts execution, which prevents repeated attempts to manipulate that object and avoids generating plans that attempt to use objects that are not present.

  3. Evidence-Conditioned Planning can be designed so that task-subgoal generation is explicitly grounded in verified information. The planner receives the accumulated evidence state as explicit textual context, allowing it to generate subgoals based on verified observations collected during Evidence Acquisition. This ensures planning is grounded in what has been confirmed rather than just assumptions.

Abstract

Robots executing long-horizon manipulation tasks from natural-language instructions must reason about both semantic task structure and geometric feasibility. Hierarchical approaches combining Vision-Language Models (VLMs) with Task and Motion Planning (TAMP) offer a promising solution, but VLM-generated subgoals may rely on prior knowledge without sufficient observational support. This can lead to the omission of task-relevant subgoals or attempts to execute instructions whose requirements cannot be met in the environment. We propose Evidence Acquisition and Feasibility Gating (EAFG), a framework for evidence-gated task and motion planning that acquires visual evidence to specify appropriate task plans and assess whether given instructions can be fulfilled. EAFG first acquires evidence by having a VLM generate subgoals and a robot execute action sequences planned by TAMP for these subgoals. For feasibility gating, the acquired evidence is used to decide whether to proceed with task planning, acquire further evidence, or halt. When proceeding, the VLM generates task subgoals conditioned on the acquired evidence, and TAMP computes low-level actions. We evaluate EAFG in three cooking scenarios involving explicitly specified required objects, underspecified instructions, and instructions requiring an absent object. EAFG improves task completion over a VLM-TAMP baseline in the explicit and underspecified scenarios and enables appropriate halting when a required object is absent.

Sources

Related papers