VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models

arXiv:2606.21386 · cs.LG, cs.CV · Submitted 2026-06-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models".

Jane: The gist: VLA-FAIL proposes a lightweight and broadly applicable failure detection framework for Vision-Language-Action Models (VLAs) that combines two novel detectors,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let’s talk a bit about who wrote this, Seligmann and his colleagues. They come from the Autonomous Learning Robots Lab at Karlsruhe Institute of Technology and FZI Forschungszentrum Informatik. They are clearly deep in the weeds on these robotic learning problems.

Jane: It’s important to know where the research is coming from because it gives you a sense of what kind of problems they are actually tackling with these models, which is manipulation tasks in general.

Lu: Their background suggests they have a solid foundation in both the vision side and the action planning side, which is crucial since VLA models merge those two domains together. They aren't just looking at one piece of the puzzle.

Meng: From an engineering standpoint, seeing that it’s coming from an institute focused on autonomous learning tells me they understand how these systems are supposed to behave under uncertainty in a physical setting.

Lalam: It sounds like this work is positioned right where we need it—bridging the gap between high-level model performance and the messy reality of deploying something in a real environment.

The paper's summary: Tom: So, what’s the core mechanism? The paper introduces two distinct failure detectors. First is Last-Layer Mahalanobis Distance, which they use to spot out-of-distribution states by looking at token-wise deviations in the last layer features compared to what the model was trained on.

Jane: That means if a specific part of the input data causes a weird signal in those final features, that’s a flag that something is off, even if the actions look perfectly smooth on paper.

Lu: And they do this by sampling a fixed prior noise vector to figure out the statistics for training data and then evaluating rollouts on a reduced feature space defined by f*(o) equals f(o, t = zero a*zero), where a*zero is just some random but fixed normal distribution.

Meng: That sounds like it’s trying to isolate the impact of the observation itself by normalizing things against some expected noise baseline. It’s an attempt to separate real task failure from just natural variability in the input features.

Lalam: Then they have Action Chunk Consistency, which targets erratic behavior by measuring how inconsistent overlapping action chunks are, using a velocity-normalized mean absolute error across action dimensions.

The paper's improvements: Tom: They show this combination of LLMD and ACC is what makes VLA-FAIL lightweight and broadly applicable. It monitors two complementary indicators: internal feature shift from LLMD, and action plan inconsistencies from ACC.

Jane: That’s the improvement—it doesn't rely on one fragile signal; it checks both the internal state of the model *and* its external behavior in action planning simultaneously.

Lu: The authors point out that using a single global feature mean and covariance matrix across all token positions captures significantly less useful information for out-of-distribution detection than using per-token statistics.

Meng: That explains why LLMD is necessary, because you need to know what’s happening at each individual step of the vision input, not just some average. It’s about capturing fine detail in the feature space.

Lalam: And they show that ACC gets better when you use a strong smoothing factor of alpha equals zero point nine, which helps filter out the noise in those action chunks to get a clearer signal on inconsistency.

Conclusion: Tom: So, to wrap up, the main point is that VLA-FAIL offers a lightweight framework for detecting task failures in finetuned VLAs without needing any failure data or external models. It monitors internal feature shift and action plan inconsistencies using LLMD and ACC.

Jane: And they introduce the AUCPDT metric to evaluate this system, which is threshold-independent, capturing precision, recall, and detection time all at once in a single number.

Lu: The results show that FAIL achieves failure detection performance comparable to or even better than significantly more expensive baseline methods while keeping the computational overhead minimal. LLMD specifically does a good job of catching failures when the policy loops or resorts to default actions that don't match the current environment state.

Meng: It’s interesting how they found that token-wise LLMD is necessary because using just one global feature mean and covariance matrix captures much less useful information for out-of-distribution detection. That’s a practical insight for deployment, focusing on local deviations rather than broad averages.

Lalam: And I think the improvement in ACC when you use velocity normalization, like on the simulated Libero-Plus benchmark where AUCPDT goes from zero point three eight down to zero point two eight for X-VLA on the Spatial suite, shows how much that temporal consistency check helps reduce detection time issues.

Tom: It really does. So, VLA-FAIL is a practical step toward human intervention pipelines and the safe real-world deployment of VLAs because it’s so efficient and robust across diverse tasks. That’s what we have today with this paper on VLA-FAIL.

Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik

cs.LG, cs.CV

Submitted: 2026-06-19

Updated: 2026-10-09

Comments: CoRL 2026

Project page: https://anonymous-vla-fail.github.io/vla-fail-2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist: VLA-FAIL proposes a lightweight and broadly applicable failure detection framework for Vision-Language-Action Models (VLAs) that combines two novel detectors, Last-Layer Mahalanobis

Key concepts

Last-Layer Mahalanobis Distance (LLMD): This detector measures how much the features of the model's last layer deviate from what it learned during training. It checks token-wise deviations in these features against a fixed noise vector to identify out-of-distribution states, even when the actions look smooth.
Action Chunk Consistency (ACC): ACC detects failures by analyzing overlapping segments of actions generated by a receding-horizon control system. It measures the velocity-normalized disagreement between these chunks, flagging inconsistencies that suggest erratic or unpredictable behavior in the action plan.
AUCPDT Metric: This is a new metric used to evaluate failure detectors because it doesn't rely on a fixed threshold. It combines precision, recall, and detection time into one score. A lower AUCPDT value indicates that the detector is more precise and detects failures earlier.
Receding-Horizon Control: This is a control strategy where the system plans actions over a short future horizon and then replans based on new information at each step. ACC exploits the temporal overlap created by this method to check if consecutive action chunks remain consistent.

Terminology

Accuracy — 1× in this paper · explained in 1 episode
Consistency — 1× in this paper · explained in 3 episodes
Failure Modes — 1× in this paper · explained in 3 episodes

Summary

The gist: VLA-FAIL proposes a lightweight and broadly applicable failure detection framework for Vision-Language-Action Models (VLAs) that combines two novel detectors, Last-Layer Mahalanobis Distance (LLMD) and Action Chunk Consistency (ACC), to reliably detect out-of-distribution states without requiring any failure data.

VLA-FAIL Framework

VLA-FAIL is a lightweight failure detection framework for finetuned VLAs that requires no failure data or external models and incurs minimal computational overhead, combining LLMD, which detects internal feature shift, and ACC, which targets action plan inconsistencies pg2. The framework triggers whenever either LLMD or ACC exceeds its calibrated threshold to enable accurate, early detection of task failures in VLAs pg7.

Last-Layer Mahalanobis Distance (LLMD)

The first detector, last-layer Mahalanobis distance (LLMD), measures token-wise deviations from the training data’s feature distribution, allowing it to capture failures even when actions appear smooth pg2. To isolate the influence of the observation, a fixed prior noise vector is sampled to compute training data statistics and evaluate rollouts on a reduced feature space defined by f∗(o):= f(o, t = 0, a∗0), where a∗0 ∼ N (0, I) random but fixed pg4. The LLMD score is the maximum of the squared token-wise Mahalanobis distances: s tLLMD(ot) = max s∈[1,…,H] h(f∗(ot)s − µs)⊤Σλ s−1(f∗(ot)s − µs) i pg4.

Action Chunk Consistency (ACC)

The second detector, action chunk consistency (ACC), targets erratic behavior by measuring the velocity-normalized disagreement between overlapping action chunks in receding-horizon control pg5. The unsmoothed ACC score at timestep t is then the velocity-normalized mean absolute error across D action dimensions: s tACC = 1/D sum d 1/(v td H) sum i a t-1R+i,d − a ti,d pg5. Strong smoothing (α = 0.9) is required for good detection rates pg5.

AUCPDT Metric

To evaluate the trade-off between detection accuracy and detection latency, AUCPDT is introduced as a threshold-independent metric that jointly captures precision, recall, and detection time pg7. The area under the penalized detection time curve (AUCPDT) yields a single, bounded, threshold-independent metric where lower values indicate more precise and earlier detectors pg13.

Experimental Results

Across real-world and simulated manipulation tasks, FAIL achieves failure detection performance comparable to, and often better than, significantly more expensive baseline methods while incurring minimal computational overhead pg8. The combination of LLMD and ACC enables robust failure detection across diverse tasks, frequently outperforming significantly more expensive baselines pg7. Notably, LLMD tends to achieve higher detection rates on tasks where the policy repeatedly retries an action in an infinite loop or resorts to default actions that do not correspond to the current environment state pg8.

Key Observations

Token-wise LLMD is necessary because using a single global feature mean and covariance matrix across all token positions captures significantly less useful information for OOD detection than per-token statistics pg8. Velocity normalization improves ACC, as shown on the simulated Libero-Plus benchmark, reducing AUCPDT from 0.38 to 0.28 for X-VLA on the Spatial suite pg15. Furthermore, ACC outperforms STAC on nearly all real-world tasks pg7.

Limitations

LLMD requires access to the finetuning data for a preprocessing pass, which may be impossible when the data is unavailable pg19. ACC requires receding-horizon control with sufficient action overlap and does not directly apply to fully open-loop chunk execution pg7. Finally, VLA-FAIL can miss failures that are consistent in features and actions, such as confidently stopping or ignoring a language instruction pg7.

The paper presents VLA-FAIL, a lightweight runtime failure detection framework for finetuned VLAs that requires no failure data pg8. FAIL combines LLMD, which measures token-wise deviations in last-layer features relative to the training data, with ACC, which measures velocity-normalized disagreement between overlapping action chunks pg7. Across real-world and simulated manipulation tasks, FAIL detects complementary failure modes with only marginal computational overhead, making VLA-FAIL a practical step toward human intervention pipelines and the safe real-world deployment of VLAs pg8.

References

[1] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B.

Improvements for AI systems

Bold header: VLA-FAIL framework implementation for real-time deployment. This system can detect task failures without requiring failure data or external models, making it suitable for safe real-world deployment of VLAs by monitoring internal feature shift and action plan inconsistencies.
Bold header: Early detection of out-of-distribution states via LLMD. The system can detect OOD states by measuring token-wise deviations from the training data’s feature distribution, specifically using the score defined in Equation (3): s tLLMD(o t) = max s∈ 1,…,H h(f∗(o t)s − µs)⊤Σλλ−1(f∗(o t)s − µs)i where higher values indicate more likely failure.
Bold header: Detection of erratic behavior via ACC. The system can exploit temporal overlap to detect failures when consecutive action chunks become inconsistent, measuring the velocity-normalized mean absolute error across action dimensions, specifically using Equation (5): s tACC = 1/D1H−R X Dd=1 1/vt d H X−R i=1 a t-1 R+i,d − a t i,d and utilizing exponential moving average for smoothing.
Bold header: Threshold-independent performance evaluation via AUCPDT. This metric can capture the trade-off between detection accuracy and detection latency by evaluating precision, recall, and detection time jointly across different score thresholds using Equation (9), yielding a single, bounded, threshold-independent metric.
Bold header: Robust cross-task failure detection capability. The combined FAIL detector enables reliable performance across diverse tasks because it combines LLMD and ACC into a single failure detector, FAIL, which is shown to be robust across tasks, frequently outperforming significantly more expensive baselines.

Abstract

Vision-language-action models (VLAs) achieve state-of-the-art performance on many robotic manipulation tasks, yet they can still behave unpredictably in out-of-distribution scenarios. Runtime failure detection is therefore essential for the safe real-world deployment of VLAs.However, existing task failure detectors require computationally expensive action sampling, are based on architectural assumptions that limit their applicability to VLAs, or need access to failure rollouts. We propose VLA-FAIL, a lightweight and broadly applicable failure detection framework for VLAs that combines two novel failure detectors with minimal overhead, without requiring failure data. The first, last-layer Mahalanobis distance (LLMD), detects out-of-distribution states by measuring token-wise deviations in last-layer features relative to the training data. The second, action chunk consistency (ACC), exploits the temporal overlap induced by receding-horizon control and detects failures when consecutive action chunks become inconsistent. To capture the trade-off between detection accuracy and detection latency, we introduce AUCPDT, a threshold-independent metric that jointly evaluates precision, recall, and detection time. Through extensive real-world and simulation experiments, we demonstrate that LLMD and ACC capture complementary failure modes whose combination enables reliable and early failure detection across diverse tasks, frequently outperforming significantly more expensive baseline methods. Project page: https://vla-fail.github.io/vla-fail-2026