Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning
cs.RO
Submitted: 2026-03-06
Updated: 2026-09-16
Code: https://github.com/AIRLab-POLIMI/multimodal-BT
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Large language models have been widely used for robotic task planning, often taking advantage of representations such as Behavior Trees (BTs).
Terminology
Abstract
Large language models have been widely used for robotic task planning, often taking advantage of representations such as Behavior Trees (BTs). Vision-Language Models (VLMs) have extended these works by grounding the generated plans in the observed scene. However, existing methods are either text-only or rely on large proprietary VLMs, while no dataset pairs visual observations and task instructions with executable and ROS2-compatible BTs. We address this gap with a multi-stage teacher pipeline that converts 1,622 Open X-Embodiment episodes into an augmented multimodal BT dataset containing 2,433 examples. We use this dataset to fine-tune compact and open-source VLMs, ranging from 500M to 4B parameters, using parameter-efficient fine-tuning (PEFT). We then evaluate the generated BTs offline in terms of syntactic correctness and by executing them on 15 household tasks in BEHAVIOR-1K. Our best model, Gemma-3 4B, achieves perfect BT validity and an 87% success rate, outperforming Claude Opus 4.8 and approaching GPT-5, while running locally. Finally, our ablation studies show that adding visual observations increases task success from 40% to 87%, while data augmentation increases BT validity from 65% to 100%.
Sources
- VLM-driven Behavior Tree for Context-aware Task Planning
- Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly
- BTGenBot-2: Efficient Behavior Tree Generation with Small Language Models
- Integrating Intent Understanding and Optimal Behavior Planning for Behavior Tree Generation from Human Instructions
- Active Learning for Convolutional Neural Networks: A Core-Set Approach
- SmolVLM: Redefining small and efficient multimodal models
- Qwen2.5-VL Technical Report
- Gemma 3 Technical Report
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies