Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models

The gist: Adaptive Failure-Informed Learning (AFIL) is an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion and flow-based Vision-Language-Action (VLA) policies to improve robustness and task success rates.

Introduction and Motivation

Vision-language-action (VLA) models offer a promising paradigm for scalable robotic manipulation, but their reliance on success-only behavioral cloning leaves them brittle because they lack corrective training signals, causing minor execution errors to compound into unrecoverable failures <ref:2605.08434#pg2>. Existing VLA models often struggle to generalize during deployment because training data typically consists primarily of teleoperated, success-only trajectories collected in confined environments <ref:2605.08434#pg2>. The paper highlights that Just as a smooth sea never made a skilled sailor, a smooth demonstration never made a skilled robot <ref:2605.08434#pg2>. Recent work has shown that unsuccessful trajectories can improve robustness and task success rates <ref:2605.08434#pg2>. However, existing approaches often rely on handcrafted or predefined failure modes, which require domain expertise and may fail to capture the diversity of realistic deployment-time failures <ref:2605.08434#pg2>.

Proposed Architecture and Training

AFIL proposes a Dual Action Generator (DAG)-VLA architecture that jointly learns from success and failure data while sharing a common vision-language backbone <ref:2605.08434#pg2>. This architecture comprises two action generators: a Success action generator Gsucc trained only on successful trajectories, and a Failure action generator Gfail trained only on failure trajectories <ref:2605.08434#pg2>. Both generators share the same vision-language backbone ΦVLM <ref:2605.08434#pg2>. The training involves two datasets, D =

D =

D =

D =

D =

D =

Ds and Df, where Ds and Df contain success and failure trajectories respectively <ref:2605.08434#pg2>. For diffusion-based policies, each action generator predicts noise at diffusion step η and minimizes a loss function that includes both success and failure data <ref:2605.08434#pg2>. Alternatively, for flow-based policies, the action head predicts a velocity field to match the ground-truth velocity between noisy actions and target actions <ref:2605.08434#pg2>.

Adaptive Failure-Informed Sampling Mechanism

During inference/sampling time, AFIL uses online-generated failure trajectories as adaptive negative guidance to steer action generation away from failure-prone regions <ref:2605.08434#pg2>. This is achieved by iteratively querying both success and failure action generators and using the predicted failure actions as adaptive negative guidance <ref:2605.08434#pg2>. The proposed Failure-Informed (FI) score combines the success generator's prediction with a weighted negative prompt derived from the failure generator's prediction <ref:2605.08434#pg2>. The guidance scale is adaptively modulated using a cosine similarity-based strategy, defined by Equation 8, which is proportional to the cosine distance between success and failure distributions <ref:2605.08434#pg2>. This mechanism captures how distinguishable successful and failed behaviors are locally, increasing guidance strength when failures become more apparent <ref:2605.08434#pg2>.

Experimental Results

Experiments across in-domain and out-of-domain robotic manipulation tasks show that AFIL consistently improves task success rates and robustness over existing VLA baselines <ref:2605.08434#pg2>. In the Maniskill environment, DAG-VLA with Adaptive Failure-Informed Learning (Adaptive-FIL) consistently achieves the strongest performance across both short- and long-horizon manipulation tasks <ref:2605.08434#pg2>. For instance, on the long-horizon settings, Adaptive-FIL shows gains of +7.4% and +12.0% over static-FIL <ref:2605.08434#pg2>. On the LIBERO benchmark, DAG-VLA with AFIL consistently achieves the strongest performance among all methods, improving average success rates from 96.9% to 98.4% on the most challenging LIBERO-10 long-horizon setting <ref:2605.08434#pg2>. Furthermore, in out-of-domain evaluation, incorporating corrected failure data improves performance across all tasks, demonstrating that adaptive failure-informed guidance improves robustness to visual, geometric, and task-level distribution shifts <ref:2605.08434#pg2>.

Conclusion

AFIL introduces a framework for improving diffusion and flow-based VLA policies by using failure trajectories as adaptive negative guidance <ref:2605.08434#pg2>. AFIL augments conventional success-only VLA learning with a Dual Action Generator architecture that jointly models successful and failed behaviors while sharing a common vision-language backbone <ref:2605.08434#pg2>. This approach improves robustness without requiring handcrafted failure modes, task-specific recovery heuristics, or separate large-scale VLA models <ref:2605.08434#pg2>. These results show that explicitly modeling failure distributions can provide an efficient and scalable mechanism for robust action generation <ref:2605.08434#pg2>. Failure-informed guidance offers a promising direction for developing VLA policies that better generalize to realistic deployment settings where errors, distribution shifts, and recovery demands are unavoidable >.

References

Yahav Avigal, Lars Berscheid, Tamim Asfour, Torsten Kroger, and Ken Goldberg. Speedfolding: Learning efficient bimanual folding of garments. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022.

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164.

Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi et al. π0.5: a visionlanguage-action model with open-world generalization. In Annual Conference on Robot Learning, 2025.

Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau et al. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2025.

Hyungjin Chung, Jeongsol Kim, Geon Yeong Park et al. CFG++: Manifold-constrained classifier free guidance for diffusion models. In International Conference on Learning Representations, 2025.

Bo Dai, Zhen Liu, Hanjun Dai et al. Exponential family estimation via adversarial dynamics embedding. In Neural Information Processing Systems, 2019.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In International Conference on Neural Information Processing Systems, 2021.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi et al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, 2023.

Jiafei Duan, Wilbert Pumacay, Nishanth Kumar et al. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation. arXiv preprint arXiv:2410.00371.

Pete Florence, Corey Lynch, Andy Zeng et al. Implicit behavioral cloning. In Annual Conference on Robot Learning, 2021.

Peter R. Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 2019.

Xiaomeng Fu and Jia Li. Tcfg: Truncated classifier-free guidance for efficient and scalable text-toimage acceleration. In IEEE/CVF International Conference on Computer Vision, 2025.

Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman and David Bau. Erasing concepts from diffusion models. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition, 2023.

Will Grathwohl, Kuan-Chieh Wang et al. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning, 2020.

Jiayuan Gu, Fanbo Xiang, Xuanlin Li et al. Maniskill2: A unified benchmark for generalizable manipulation skills.

1. Robustness to compounding errors in long-horizon tasks is significantly enhanced by using failure trajectories as adaptive negative guidance during sampling, which steers action generation away from failure-prone regions and toward more reliable success modes. This allows the policy to recover from suboptimal or unseen states during inference time.

2. Efficient policy learning with limited parameter overhead is achieved through the Dual Action Generator (DAG) architecture, which jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone. This avoids the need for handcrafted failure-mode design or human-in-the-loop recovery by learning from mixed success and failure data.

3. Dynamic guidance strength is modulated using a cosine similarity strategy, where the guidance scale is set to alpha (c.f. Eq. 8) = alpha (1 - cos(epsilon succ, epsilon fail)). This mechanism ensures that negative guidance is only applied when the success and failure predictions are semantically aligned and amplified when they represent distinct behavioral choices, preventing suppression of valid actions in ambiguous states.

4. The system achieves superior generalization across distribution shifts by leveraging online-inferred failure trajectories, which captures a richer, more realistic failure distribution compared to handcrafted taxonomies. This enables the policy to improve robustness to visual, geometric, and task-level distribution shifts in out-of-domain evaluations.

Vision-language-action (VLA) models provide a promising paradigm for scalable robotic manipulation, yet their reliance on success-only behavioral cloning leaves them brittle; lacking corrective training signals, minor execution errors rapidly compound into unrecoverable, out-of-distribution failures. To address this limitation, we propose Adaptive Failure-Informed Learning (AFIL), an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies. AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design or human-in-the-loop recovery. It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone, enabling efficient failure-aware policy learning with limited parameter overhead. During sampling, the failure generator adaptively steers action generation away from failure-prone regions and toward more reliable success modes, with guidance strength determined by the per-diffusion-step distance between success and failure distributions. Experiments across in-domain and out-of-domain robotic manipulation tasks, covering both short- and long-horizon settings, show that AFIL consistently improves task success rates and robustness over existing VLA baselines, demonstrating its effectiveness, efficiency, and generality.