STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration
cs.RO, cs.AI
Submitted: 2026-08-27
Updated: 2026-08-27
Comments: Published in IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2026, 8 pages, 4 figuers, 5 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload.
Terminology
Abstract
Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- Forecasting Future Action Sequences with Neural Memory Networks
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- Reasoning with Language Model is Planning with World Model
- Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency
- LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Model
- Making Large Language Models into World Models with Precondition and Effect Knowledge
- WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agents
- Hierarchical Deep Learning for Intention Estimation of Teleoperation Manipulation in Assembly Tasks
- How Can We Know What Language Models Know?
- Generate & Rank: A Multi-task Framework for Math Word Problems
- Pretrained Transformers as Universal Computation Engines
- EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies