When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
Vision-language-action (VLA) models often fail when a task requires a different action than what is familiar, even when given an instruction. This failure, called instruction-action binding, occurs because the model selects a familiar trajectory family instead of the one dictated by the instruction. The paper introduces Equivariant Counterfactual Training (ECT) to fix this by training models on pairs of scenes requiring different actions under the same instruction.