Instance-anchored interaction evidence: Grounding robot plans in human pointing and handling

The gist The authors propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances to ground robot plans in human pointing and handling; this method significantly improves plan success on implicit-intent tasks compared to existing vision-language models.

Instance-Anchored Interaction Evidence (IAE)

IAE is a representation that registers every object of the final scene to its public identifier by fitting the public layout to the detections, propagates instance masks backwards through the video so that each identity is kept while objects are handled and moved, and describes each frame by the geometric relation between the detected hands, forearms and these instances In this framework, a robot must act on what a person has shown rather than said The system outputs a set of operations A = (o, d) that move object o into region d

Evidence Network and Learning Scheme

The evidence network is trained only from task outcomes scores the instances For pointing tasks, a grammar-constrained dynamic program trained with a structured loss decodes object–destination programs The training scheme uses multiple-instance labels where objects and destinations that appear in its goal were indicated at some unknown time, and all other candidates were never indicated This is formalized by the class-balanced binary cross-entropy loss called MIL The program-level structured loss adds a structured hinge loss computed with the same dynamic program to train the evidence to make the decoded program correct rather than only to rank candidates

Grammar-Constrained Program Decoding

The decoding process involves a grammar-constrained dynamic program trained with a structured loss decodes object–destination programs The deictic grammar is therefore (O+D)+, where all objects preceding a destination cue are assigned to it The recursion for finding the best program is exact over the candidate events; times outside the retained peaks are not considered This process determines at once which candidates were indicated and how objects pair with destinations

Performance and Comparison

On 1,255 WatchAct benchmark requests, IAE reaches 64.2% plan success on implicit-intent tasks against 27.5% for a 32B vision-language model This represents a gain of 36.7 percentage points above the 32B VLM For restoration, reversal and imitation it reaches 46.4% without task-specific training, where the 32B VLM reaches 27.0% The same evidence lifts a 32B VLM from 27.5% to 57.4% plan success

Error Analysis and Limitations

The main remaining steps are suppressing spurious object events, which cause most strict failures on pointing tasks, and resolving destinations that are ambiguous in depth The results show that the decoding cost trades the two metrics against each other, and a cost selected by nested cross-validation does not raise strict success on new activities while it lowers plan success Small errors in registration rely on the public layout of the execution scene, and with errors of one grid cell a third of the videos change their identity assignment The plans are evaluated by symbolic execution in the benchmark’s simulator, showing which objects a robot should move and where, not whether it can grasp and place them

Conclusion

Instance-anchored interaction evidence binds human interactions in a demonstration video to the object instances that a robot will manipulate Registering the execution scene to its public layout and tracking every instance backwards makes identity explicit; hand and forearm geometry relative to these instances forms the evidence Reading pointing gestures is the harder part, benefiting from learned evidence and from decoding whole programs, remaining the main source of errors The authors conclude that suppressing spurious object events, which cause most strict failures on pointing tasks, and resolving destinations that are ambiguous in depth are the main remaining steps

--- Page 1 ---

The gist The authors propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances to ground robot plans in human pointing and handling.

Instance-Anchored Interaction Evidence (IAE)

IAE is a representation that registers every object of the final scene to its public identifier by fitting the public layout to the detections, propagates instance masks backwards through the video so that each identity is kept while objects are handled and moved, and describes each frame by the geometric relation between the detected hands, forearms and these instances

Evidence Network and Learning Scheme

The evidence network is trained only from task outcomes scores the instances For pointing tasks, a grammar-constrained dynamic program trained with a structured loss decodes object–destination programs The training scheme uses multiple-instance labels where objects and destinations that appear in its goal were indicated at some unknown time, and all other candidates were never indicated This is formalized by the class-balanced binary cross-entropy loss called MIL

Performance and Comparison

On 1,255 WatchAct benchmark requests, IAE reaches 64.2% plan success on implicit-intent tasks against 27.5% for a 32B vision-language model This represents a gain of 36.7 percentage points above the 32B VLM For restoration, reversal and imitation it reaches 46.4% without task-specific training, where the 32B VLM reaches 27.0%

Error Analysis and Limitations

The main remaining steps are suppressing spurious object events, which cause most strict failures on pointing tasks, and resolving destinations that are ambiguous in depth The results show that the decoding cost trades the two metrics against each other, and a cost selected by nested cross-validation does not raise strict success on new activities while it lowers plan success

--- Page 2 ---

The gist The authors propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances to ground robot plans in human pointing and handling.

1. Bold header: Instance-Anchored Interaction Evidence (IAE) Registration and Tracking

Improvement: Implement a system that "registers every object of the final scene to its public identifier by fitting the public layout to the detections, propagates instance masks backwards through the video so that each identity is kept while objects are handled and moved. This explicitly solves the problem where the plan depends on which instance a person interacted with" by maintaining object identities across time.

2. Bold header: Weak Supervision for Deictic Evidence Learning

Improvement: Develop a weakly supervised learning scheme for deictic evidence that uses only task outcomes, combining a multiple-instance loss, grammar-constrained program decoding and a program-level structured loss. This allows the system to learn from task outcomes without frame labels, which is crucial for generalization.

3. Bold header: Grammar-Constrained Program Decoding

Improvement: Utilize a grammar-constrained dynamic program trained with a structured loss decodes object–destination programs, where the grammar enforces (O+D)+ and the decoder computes the plan by maximizing F(w) = ∑ e∈w (ze - λ) subject to constraints like uses distinct objects and at most Nmax = 3 objects.

4. Bold header: Reference Disambiguation via Learned Scores

Improvement: Use a symbolic program [to handle reference disambiguation] trained with the same decoder to interpret evidence scores, allowing the system to select targets based on learned criteria, such as choosing between the selected or the other objects.

5. Bold header: Episodic Task Execution via Instance Tracking

Improvement: For restoration and reversal tasks, utilize backward mask tracking combined with instance placement rules to determine the final location of every object (The program moves every instance whose place changed) by reading the state at anchor frames, ensuring every successful plan, which moves only instances whose place changed, also left all other objects in place.

6. Bold header: Robustness Against Identity Permutation

Improvement: Enhance registration robustness by using Count-constrained detection and Layout registration to ensure that even with identity permutations (Noise sigma = 1 cell), the system can maintain a valid map, which is vital because their success rests on identity bookkeeping.