InFeR: Informed Failure Resilience in Learned Visual Navigation Control
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "InFeR: Informed Failure Resilience in Learned Visual Navigation Control".
Paper summary: Dev: They claim this method allows the policy to detect when it sees something out of distribution and also localizes the specific image region causing that issue, which then informs a recovery policy.
Rosa: The authors show that they use Grad-CAM for this localization step, which essentially maps gradients back to the image pixels to show where the failure is originating.
Taro: The mechanism relies on maximizing a mutual information objective called LVIB, which balances retaining information about the action with keeping the latent representation close to a simple prior distribution.
Dev: So at inference time, when an out of distribution input comes in, the resulting latent feature diverges significantly from that simple prior distribution because it has captured something unexpected.
Rosa: The results show this works in practice on real robots, like Boston Dynamics Spot, where InFeR enables informed recovery across different robot control policies.
Taro: The paper demonstrates that this approach is effective against four types of critical navigation failures observed in the experiments.
Conclusion: Dev: The authors are Zishuo Wang, Joel Loo and David Hsu. The main implication here is moving beyond just training systems for success to building systems that can handle failure gracefully in complex real-world settings without needing new data for every potential problem.
Rosa: It means we can deploy these navigation policies with a much higher degree of trust because they are designed to recognize and react intelligently when the world misbehaves unexpectedly.
Taro: The paper shows this works across different architectures, which is important because it’s not tied to one specific way you've built your learning model.
Dev: It suggests that we can achieve safer, more robust navigation in the real world by focusing on how the learned system understands its own state and potential errors.
Zishuo Wang, Joel Loo, David Hsu
School of Computing & Smart Systems Institute, National University of Singapore
cs.RO
Submitted: 2025-10-28
Updated: 2026-11-17
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: While imitation learning (IL) has enabled successful visual navigation in many common environments, IL policies are prone to unpredictable failures under out-of-distribution (OOD) scenarios InFeR…
Key concepts
- Variational Information Bottleneck (VIB) Loss
- This loss function is used during retraining to structure the policy's latent space. It balances maximizing information about actions with minimizing information about the input, specifically designed to distinguish between normal inputs and out-of-distribution (OOD) inputs. This makes the policy better at recognizing when it encounters something it hasn't seen before.
- Grad-CAM
- Grad-CAM is a visual explainability technique used to locate where in an input image a specific decision was made. By calculating gradients with respect to the feature maps, it highlights the exact image region that most influenced the policy's output, which helps identify what caused a failure.
- OOD Recognition
- This process uses conformal prediction and KL-based scores to establish thresholds for detecting inputs that are significantly different from the training data. This detection mechanism is crucial because it signals when an input is likely out-of-distribution, triggering the recovery process.
- Informed Failure Resilience
- This goes beyond just detecting a failure; it means recognizing *why* it happened by pinpointing the source (using Grad-CAM) and then using that knowledge to autonomously recover. This allows the policy to handle diverse real-world failures without needing new demonstrations.
Terminology
Summary
While imitation learning (IL) has enabled successful visual navigation in many common environments, IL policies are prone to unpredictable failures under out-of-distribution (OOD) scenarios InFeR proposes a general framework for building IL policies with informed failure resilience without failure or recovery demonstrations This framework retrains an IL policy with a Variational Information Bottleneck (VIB) loss to structure its latent space for OOD failure detection It applies a visual explainability technique, Grad-CAM, to localise an image region as the source of failure and inform a heuristic policy for recovery Real-world experiments show that InFeR enables informed failure recovery across two different policy architectures, yielding robust longrange navigation in complex environments. p. 0 InFeR retrains an IL policy with a Variational Information
Bottleneck (VIB) loss to structure its latent space for OOD
failure detection. It applies a visual explainability technique,
Grad-CAM, to localise an image region as the source of
failure and inform a heuristic policy for recovery.Where the paper says it
Framework Overview
InFeR operates within the standard offline IL setting, learning policies from a single, static dataset of successful demonstrations where failures are naturally approximated as scenarios outside the dataset’s distribution. The framework introduces three desiderata for failure-resilient policies: (i) the ability to autonomously recover from failures; (ii) the ability to recognise failure sources to inform recovery; and (iii) detection and recognition grounded in the policy’s internal representations. p. 0 These
limitations motivate three desiderata for failure-resilient policies: (i) the ability to autonomously recover from failures;
(ii) the ability to recognise failure sources to inform recovery;
and (iii) detection and recognition grounded in the policy’s
internal representations to capture its true failure modes. We propose InFeR, a general framework for building
informed, failure-resilient visual navigation policies without any failure or recovery demonstrations. regularised training objective to equip the policy to detect and
recognise out-of-distribution (OOD) elements in its inputs
without additional data or changes to its architecture. To
recover from OOD scenarios, InFeR pairs the IL policy with
a recovery policy guided by OOD recognition.Where the paper says it
Where the paper says it
How the training signal is built
InFeR retrains an IL policy on the original data, into an OOD-aware policy that detects OOD image inputs and localises the image regions responsible. This is achieved with a Variational Information Bottleneck (VIB) loss which structures the latent space between vision encoder and action decoder to distinguish in- and out-of-distribution inputs. p. 0 Specifically, InFeR retrains an IL policy on the original
data, into an OOD-aware policy that detects OOD image
inputs and localises the image regions responsible. This is
done with a Variational Information Bottleneck (VIB) loss,
which structures the latent space between vision encoder
and action decoder to distinguish in- and out-of-distribution
inputs. The first term simplifies to a behaviour cloning
loss, while the second term regularises the latent representation Z to stay close to prior p(z), which we define as
a unit Gaussian N (0,I). This regularisation means that at
inference, an OOD input oˆ is likely to produce latent feature
zˆ that diverges significantly from the prior.Where the paper says it
Where the paper says it
Where the gains come from
OOD detection involves applying conformal prediction (CP) to obtain time-varying thresholds on KL scores OOD recognition is achieved with Grad-CAM, which computes gradients of the prediction with respect to feature map activations in preceding convolutional layers. The recognition result is then computed as MGrad-CAM = ReLU X k αkA k! p. 3 Specifically, we compute the weight for
the k-th feature map of the layer as:
αk =
1
N
X
i
X
j
∂KL[qϕ(zt|ot)∥N (0,I)]
∂Ak
ij
(4)
where Ak
is the k-th feature map and N normalises across
all Ak
. The recognition result is:
MGrad-CAM = ReLU X
k
αkA
k
! Both InFeR methods achieve
high AUC scores across both detection and recognition
compared with density-based approaches (NF), one-class
discriminators (RND), and reconstruction-based approaches
(AE, VAE-R). InFeR-DEC achieves the best performance
by a clear margin, with high TPR and relatively few false
positives.Where the paper says it
Where the paper says it
Failure Resilience in Practice
In real-world experiments on a Boston Dynamics Spot robot, InFeR enables two policies—DECISION [2] and GNM [3]—to detect and recover from critical failures in the real world. The framework successfully handles four types of critical Navigation Failures (NF1–NF4). p. 3 We train our policies and baselines on the training dataset
from [2], comprising 30 hours of teleoperated trajectories
demonstrating the task of path-following navigation in indoor and outdoor scenes. We identify four types of critical
Navigation Failures (NF) to detect and recover from, which
are OOD for this dataset:
NF1 Sensor failure. Specifically, we test whether InFeR policies
can detect and recover from the key failure types identified
(NF1–NF4). To illustrate the importance of informed failure
resilience, we show that InFeR allows the DECISION policy
to autonomously complete a 300m route with varied terrain
(cluttered indoor plaza, outdoor park with forested paths)
despite critical failures. We compare InFeR-DEC with an uninformed variant
(InFeR-DEC-Blind) that ignores the recognition heatmap
Mt, and randomly selects corrective actions. InFeR-DEC
consistently outperforms InFeR-DEC-Blind, especially in
recovering from close-range dynamic obstacles (NF3), where
timely, informed actions are critical.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
We argue that informed failure resilience in IL policies extends beyond detection to encompass recognising failure sources and autonomously recovering from them. InFeR requires no additional data and no changes to the IL policy’s architecture, generalizing across two distinct policy architectures and a range of diverse failure modes. p. 7 It requires no additional
data, and no changes to the IL policy’s architecture. InFeR
generalises across two distinct policy architectures, and a
range of diverse failure modes across different scenes, enabling IL policies to succeed at long-range navigation in
complex environments without intervention. InFeR
generalises across two distinct policy architectures, and a
range of diverse failure modes across different scenes, enabling IL policies to succeed at long-range navigation in
complex environments without intervention. This shows that
InFeR is a clear step toward safe, robust navigation with
IL. Overall, we draw
three specific conclusions:
InFeR effectively recovers from diverse real-world failures. Table II shows that InFeR enables high detection and
recovery success across varied failure types, supporting H2. This yields the
per-bin recognition score only when the input is flagged as
OOD. The results in Figure 3 support H1: InFeR produces
OOD-aware policies effective at task-relevant OOD detection/recognition.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Variational Information Bottleneck (VIB) loss for latent space structure allows InFeR to distinguish in-distribution from out-of-distribution inputs, enabling OOD detection without additional training data or architectural changes.
-
Grad-CAM is applied to the KL-based OOD scores predicted by the model to localize visual regions responsible for the detection, providing a spatial source of failure information for recovery.
-
The system can autonomously recover from critical failures in real-world scenarios, such as sensor failures (NF1), blocked pathways (NF2), and close-range dynamic obstacles (NF3), without requiring human intervention.
-
InFeR enables informed recovery by pairing the IL policy with a heuristic recovery policy that selects corrective actions based on the OOD recognition heatmap to
maximally avoid OOD features in the observation ot.
-
The system can generalize across different visual navigation policies, specifically demonstrating effectiveness for both DECISION and GNM architectures, supporting H3.
Abstract
While imitation learning (IL) has enabled successful visual navigation in many common environments, IL policies are prone to unpredictable failures under out-of-distribution (OOD) scenarios. This necessitates failure-resilient policies, which not only detect failures, but also recognise their sources and recover from them autonomously. We propose InFeR, a general framework for building IL policies with informed failure resilience without failure or recovery demonstrations. Given an IL policy's architecture and training dataset, InFeR retrains a new OOD-aware policy with a latent space structured for OOD failure detection, with a modified objective based on the Variational Information Bottleneck. It applies a visual explainability technique, Grad-CAM, to attribute failures to specific image regions and inform a heuristic policy for recovery. All these are achieved without requiring additional training data. Real-world experiments show that InFeR enables informed failure recovery across two different policy architectures, supporting long-range navigation in complex environments.
Sources
- Deep Learning for Anomaly Detection: A Survey
- Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies
- IntentionNet: Map-Lite Visual Navigation at the Kilometre Scale
- Uncertainty-Aware Reinforcement Learning for Collision Avoidance
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation