Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies
The study tested how vision-language-action policies handle multiple simultaneous disturbances using a six-axis benchmark called LIBERO-CTRL. It found that simply averaging success rates from individual tests doesn't capture true robustness because opposing failures and successes can cancel each other out. Therefore, matched, paired evaluations are necessary to accurately measure compound robustness.