What Actually Matters? A Paired Evaluation of Inference Settings for pi 0.5 on BEHAVIOR-1K

arXiv:2610.37917 · cs.RO, cs.AI, cs.CV, cs.LG · Submitted 2026-10-27 · Read on arXiv

cs.RO, cs.AI, cs.CV, cs.LG

Submitted: 2026-10-27

Updated: 2026-10-27

License: http://creativecommons.org/licenses/by-sa/4.0/

The gist: Vision-Language-Action (VLA) models offer a promising paradigm for general-purpose household robotics.

Abstract

Vision-Language-Action (VLA) models offer a promising paradigm for general-purpose household robotics. However, inference and data-collection strategies that have been deployed in smaller-scale settings do not necessarily generalize to large-scale, diverse task suites. To address this gap, we conduct a systematic, paired evaluation of previously proposed inference and data-collection strategies across all 100 tasks. Our evaluation uses a fine-tuned π 0.5 model based on the 2025 BEHAVIOR Challenge winner, which achieves a Q-score of 0.39 on the public test set. These results show that proposed strategies do not transfer uniformly to new environments and tasks. Project page, videos, and detailed analysis: https://rubenhammele.github.io/behavior2026

Related papers