Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

arXiv:2604.13788 · cs.RO, cs.CV · Submitted 2026-04-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering".

Dev: The gist The introduction proposes FIDeL,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: We're wrapping up on "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering". The authors, Quentin Rolland, Fabrice Mayran de Chamisso, and Jean-Baptiste Mouret, are proposing FIDeL as a way to make robot monitoring more reliable.

Dev: Basically, this paper is about taking those imitation learning robots that work fine in the lab and figuring out how to actually spot when they fail out there in real life.

Taro: The main idea is using vision-based anomaly detection, but with a smart filter on top to make sure you're not flagging normal bumps as real failures.

Rosa: They call their system FIDeL, and it builds a memory of what normal robot actions look like from the training demonstrations.

Dev: It uses optimal transport matching to compare what’s happening now against that stored memory to get an anomaly score.

Taro: And then they use conformal prediction for thresholds, which means the system knows when a deviation is big enough to be worrying.

Rosa: But here's the kicker, they add a Vision-Language Model, or VLM, to semantically check if that weird state is actually a failure.

Dev: They found that this approach beats existing methods on their own dataset, BotFails, by showing better failure detection accuracy and anomaly detection scores.

Taro: The implication is that you don't just need a good policy; you also need a system robust enough to tell the difference between a little slip and something actually broken.

Rosa: It’s about building monitoring systems for robots that can actually interpret what they see when things go wrong in the messy real world.

Dev: But we have to keep an eye on the limitations, like how much variation in training data it needs and how much processing power that VLM filter uses.

Taro: Exactly, because if you don't have diverse demos, the system might just flag every tiny change as a disaster.

Rosa: So while FIDeL shows real promise for making robot monitoring more reliable, we still need to figure out how to make those filters run fast enough on the actual hardware.

Conclusion: Rosa: So we're wrapping up on "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering". This paper is really about taking those imitation learning robots that work fine in the lab and figuring out how to actually spot when they fail out there in real life.

Dev: Exactly. They built a system, FIDeL, that uses vision-based anomaly detection combined with semantic filtering to separate real failures from normal bumps in the road when things go wrong during robot deployment.

Taro: The authors are Quentin Rolland and Fabrice Mayran de Chamisso and Jean-Baptiste Mouret. It’s interesting because they tackle that false positive problem where you get flagged for something that isn't actually a disaster.

Rosa: They do this by building a memory of what nominal robot actions look like from the training demonstrations, then using optimal transport matching to compare current observations against that memory to get anomaly scores.

Dev: And they use conformal prediction to set those thresholds, which means the system knows when a deviation is big enough to worry about, accounting for both how fast things are moving and where on the robot it's happening.

Taro: The key addition is this Vision-Language Model that does semantic filtering—it looks at the anomaly heatmaps and decides if what it sees is actually a failure or just something benign that happens all the time.

Rosa: It seems like this approach really balances robustness to natural task variations with sensitivity to those critical failure-related deviations we're looking for in these imitation learning systems.

Dev: But we have to keep an eye on the limitations, like how much variation in training data it needs and how much processing power that VLM filter uses for real-time operation.

Taro: That makes sense because if you only train it on one specific way to do a task, anything slightly different will look like an error.

Rosa: So while FIDeL shows real promise for making robot monitoring more reliable, we still need to figure out how to make those filters run fast enough on the actual hardware.

Dev: That makes sense. So this whole process of encoding demonstrations, aligning them with new observations using optimal transport, thresholding with conformal prediction, and finally filtering with a VLM is what they are proposing in "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering".

Taro: It points toward a future where robot monitoring isn't just about how good the policy is on paper but also about recognizing and interpreting when things go wrong in practice.

Universite Paris-Saclay · CEA List FactoryAI

cs.RO, cs.CV

Submitted: 2026-04-15

Updated: 2026-04-15

Comments: 8 pages, Appendix coming soon, accepted at ICRA 2026

Code: https://github.com/huggingface/lerobot

Project page: https://cea-list.github.io/FIDeL

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: The gist The introduction proposes FIDeL, a policy-independent failure detection module that uses vision-based anomaly detection combined with semantic filtering to distinguish between benign

Key concepts

Optimal Transport Matching: This technique is used to align incoming observations with the compact representation of nominal demonstrations. It calculates the minimum cost required to transform a query feature into a key from the learned memory, providing a robust way to measure how different an observation is from what was expected during training.
Conformal Prediction (CP) Extension: This framework is extended to create dynamic thresholds for the anomaly detector. It helps set bounds that account for both temporal structure and variations in visual patches, ensuring that the system only flags states that are statistically significant deviations from normal behavior.
Vision-Language Model (VLM) Filtering: A VLM acts as a semantic filter to distinguish between benign anomalies and true failures. It is prompted with task context and visual evidence to classify an anomaly, helping the system avoid unnecessary interventions while reliably identifying genuine execution errors.

Terminology

AUROC — 1× in this paper · explained in 3 episodes
Accuracy — 1× in this paper · explained in 1 episode
Conformal Prediction — 1× in this paper · explained in 9 episodes
Imitation Learning (IL) — 1× in this paper · explained in 2 episodes
Multimodal Dataset — 1× in this paper · explained in 2 episodes
Optimal Transport — 1× in this paper · explained in 5 episodes
Vision-Language Model (VLM) — 1× in this paper · explained in 3 episodes
Vision-based Anomaly Detection — 1× in this paper

Summary

The gist The introduction proposes FIDeL, a policy-independent failure detection module that uses vision-based anomaly detection combined with semantic filtering to distinguish between benign deviations and genuine failures in imitation learning policies.

How it works

FIDeL leverages recent AD methods to build a compact representation of nominal demonstrations and aligns incoming observations via optimal transport matching to produce anomaly scores and heatmaps.
Spatio-temporal thresholds are derived with an extension of conformal prediction, and a Vision–Language Model (VLM) performs semantic filtering to discriminate benign anomalies from genuine failures.
The process involves a three-stage mechanism: Anomaly Detection assigns a score reflecting the likelihood of deviation from nominal behavior, Thresholding compares this score against calibrated bounds, and Semantic Filtering uses a VLM to determine if the anomaly is failure-related.

Memory Representation

The memory module M is built offline from nominal demonstrations by encoding trajectories using a vision encoder, resulting in feature tensors ϕ(YN) where features are aggregated across episodes by computing Gaussian statistics. This results in a memory encoding the expected distribution of nominal features: M = Σµt,p,f, σt,p,f t ∈ J1, TK, p ∈ J1, PK, f ∈ J1Page 3>.

Anomaly Score Computation

At runtime during the online phase at each timestep t∗, the anomaly detector DA computes a score DA(ϕ(YN), ϕt∗) by aligning query features ϕt∗ with keys from YN using optimal transport. This score is compared against conformal thresholds to decide if Ot∗ is anomalous. The final anomaly score is obtained as the minimum transport cost across nominal timesteps: DA(ϕ(YN), ϕt∗) = min t∈J1,TK Wt, tmin = arg min t∈J1,TK WtPage 3>.

Thresholding and Semantic Filtering

The Conformal Prediction (CP) framework is extended to define a dynamic threshold for the anomaly detector DA, accounting for both temporal structure and patch-level variations. The VLM filter then performs semantic filtering by being prompted with contextual task information and visual evidence to classify the anomaly as either False positive or Failure.

Evaluation and Results

FIDeL consistently outperforms state-of-the-art baselines, yielding +5.30% AUROC in anomaly detection and +17.38% failuredetection accuracy on BotFails compared to existing methods. The results show that semantic filtering is crucial for reliable failure monitoring—avoiding unnecessary interventions while still capturing genuine failures. FIDeL balances robustness to natural task variability with sensitivity to failure related deviations.

Limitations

A limitation of FIDeL is that its performance depends heavily on the diversity of demonstrations, which may cause benign variations to be flagged as anomalies. Furthermore, the use of the VLM filter is computationally costly. The method’s spatial and temporal invariance can reduce sensitivity to fine-grained or order-dependent deviations. Finally, balancing invariance and sensitivity remains taskdependent.

FIDeL provides a practical interface for monitoring robot behavior in deployment where reliable monitoring hinges not only on policy quality but also on the ability to recognize and interpret failures.

Improvements for AI systems

Bold representation-based anomaly detection tailored for autonomous robotics: This improves upon existing methods by aligning incoming observations with stored nominal demonstrations via optimal transport matching to produce anomaly scores and heatmaps, which allows the system to build a compact representation of nominal demonstrations that is more robust to variable execution speeds than methods relying solely on reconstruction error.
Spatio-temporal thresholding for dynamic decision-making: Extending Conformal Prediction with memory-based temporal alignment that ensures robustness to variable execution speeds allows the system to set adaptive bounds, resulting in a mechanism where an input is flagged as anomalous when the fraction of patches exceeding their respective bounds surpasses a user-defined percentage.
Semantic filtering for failure discrimination: The integration of a Vision–Language Model (VLM) allows for explicit semantic interpretation, enabling the system to perform the crucial task of distinguishing genuine failures from benign deviations by prompting the VLM to Classify the anomaly as either: False positive (DF = 0): the anomaly is benign, the robot may continue. Failure (DF = 1): the anomaly is a failure.

Abstract

Imitation learning (IL) policies in robotics deliver strong performance in controlled settings but remain brittle in real-world deployments: rare events such as hardware faults, defective parts, unexpected human actions, or any state that lies outside the training distribution can lead to failed executions. Vision-based Anomaly Detection (AD) methods emerged as an appropriate solution to detect these anomalous failure states but do not distinguish failures from benign deviations. We introduce FIDeL (Failure Identification in Demonstration Learning), a policy-independent failure detection module. Leveraging recent AD methods, FIDeL builds a compact representation of nominal demonstrations and aligns incoming observations via optimal transport matching to produce anomaly scores and heatmaps. Spatio-temporal thresholds are derived with an extension of conformal prediction, and a Vision-Language Model (VLM) performs semantic filtering to discriminate benign anomalies from genuine failures. We also introduce BotFails, a multimodal dataset of real-world tasks for failure detection in robotics. FIDeL consistently outperforms state-of-the-art baselines, yielding +5.30% percent AUROC in anomaly detection and +17.38% percent failure-detection accuracy on BotFails compared to existing methods.