InFeR: Informed Failure Resilience in Learned Visual Navigation Control

arXiv:2510.24680 · cs.RO · Submitted 2025-10-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "InFeR: Informed Failure Resilience in Learned Visual Navigation Control".

Paper summary: Dev: They claim this method allows the policy to detect when it sees something out of distribution and also localizes the specific image region causing that issue, which then informs a recovery policy.

Rosa: The authors show that they use Grad-CAM for this localization step, which essentially maps gradients back to the image pixels to show where the failure is originating.

Taro: The mechanism relies on maximizing a mutual information objective called LVIB, which balances retaining information about the action with keeping the latent representation close to a simple prior distribution.

Dev: So at inference time, when an out of distribution input comes in, the resulting latent feature diverges significantly from that simple prior distribution because it has captured something unexpected.

Rosa: The results show this works in practice on real robots, like Boston Dynamics Spot, where InFeR enables informed recovery across different robot control policies.

Taro: The paper demonstrates that this approach is effective against four types of critical navigation failures observed in the experiments.

Conclusion: Dev: The authors are Zishuo Wang, Joel Loo and David Hsu. The main implication here is moving beyond just training systems for success to building systems that can handle failure gracefully in complex real-world settings without needing new data for every potential problem.

Rosa: It means we can deploy these navigation policies with a much higher degree of trust because they are designed to recognize and react intelligently when the world misbehaves unexpectedly.

Taro: The paper shows this works across different architectures, which is important because it’s not tied to one specific way you've built your learning model.

Dev: It suggests that we can achieve safer, more robust navigation in the real world by focusing on how the learned system understands its own state and potential errors.

Zishuo Wang, Joel Loo, David Hsu

School of Computing & Smart Systems Institute, National University of Singapore

cs.RO

Submitted: 2025-10-28

Updated: 2026-11-17

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: While imitation learning (IL) has enabled successful visual navigation in many common environments, IL policies are prone to unpredictable failures under out-of-distribution (OOD) scenarios InFeR…

Key concepts

Variational Information Bottleneck (VIB) Loss
This loss function is used during retraining to structure the policy's latent space. It balances maximizing information about actions with minimizing information about the input, specifically designed to distinguish between normal inputs and out-of-distribution (OOD) inputs. This makes the policy better at recognizing when it encounters something it hasn't seen before.
Grad-CAM
Grad-CAM is a visual explainability technique used to locate where in an input image a specific decision was made. By calculating gradients with respect to the feature maps, it highlights the exact image region that most influenced the policy's output, which helps identify what caused a failure.
OOD Recognition
This process uses conformal prediction and KL-based scores to establish thresholds for detecting inputs that are significantly different from the training data. This detection mechanism is crucial because it signals when an input is likely out-of-distribution, triggering the recovery process.
Informed Failure Resilience
This goes beyond just detecting a failure; it means recognizing *why* it happened by pinpointing the source (using Grad-CAM) and then using that knowledge to autonomously recover. This allows the policy to handle diverse real-world failures without needing new demonstrations.

Terminology

Summary

While imitation learning (IL) has enabled successful visual navigation in many common environments, IL policies are prone to unpredictable failures under out-of-distribution (OOD) scenarios InFeR proposes a general framework for building IL policies with informed failure resilience without failure or recovery demonstrations This framework retrains an IL policy with a Variational Information Bottleneck (VIB) loss to structure its latent space for OOD failure detection It applies a visual explainability technique, Grad-CAM, to localise an image region as the source of failure and inform a heuristic policy for recovery Real-world experiments show that InFeR enables informed failure recovery across two different policy architectures, yielding robust longrange navigation in complex environments. p. 0

Where the paper says it

InFeR retrains an IL policy with a Variational Information Bottleneck (VIB) loss to structure its latent space for OOD failure detection. It applies a visual explainability technique, Grad-CAM, to localise an image region as the source of failure and inform a heuristic policy for recovery.

Framework Overview

InFeR operates within the standard offline IL setting, learning policies from a single, static dataset of successful demonstrations where failures are naturally approximated as scenarios outside the dataset’s distribution. The framework introduces three desiderata for failure-resilient policies: (i) the ability to autonomously recover from failures; (ii) the ability to recognise failure sources to inform recovery; and (iii) detection and recognition grounded in the policy’s internal representations. p. 0

Where the paper says it

These limitations motivate three desiderata for failure-resilient policies: (i) the ability to autonomously recover from failures; (ii) the ability to recognise failure sources to inform recovery; and (iii) detection and recognition grounded in the policy’s internal representations to capture its true failure modes. We propose InFeR, a general framework for building informed, failure-resilient visual navigation policies without any failure or recovery demonstrations.

InFeR pairs the IL policy with a recovery policy guided by OOD recognition results. p. 0
Where the paper says it

regularised training objective to equip the policy to detect and recognise out-of-distribution (OOD) elements in its inputs without additional data or changes to its architecture. To recover from OOD scenarios, InFeR pairs the IL policy with a recovery policy guided by OOD recognition.

How the training signal is built

InFeR retrains an IL policy on the original data, into an OOD-aware policy that detects OOD image inputs and localises the image regions responsible. This is achieved with a Variational Information Bottleneck (VIB) loss which structures the latent space between vision encoder and action decoder to distinguish in- and out-of-distribution inputs. p. 0

Where the paper says it

Specifically, InFeR retrains an IL policy on the original data, into an OOD-aware policy that detects OOD image inputs and localises the image regions responsible. This is done with a Variational Information Bottleneck (VIB) loss, which structures the latent space between vision encoder and action decoder to distinguish in- and out-of-distribution inputs.

The VIB loss maximizes the objective LVIB = I(A;Z) − βI(Z; O) (1) This regularization means that at inference, an OOD input oˆ is likely to produce latent feature zˆ that diverges significantly from the prior N (0,I). p. 2
Where the paper says it

The first term simplifies to a behaviour cloning loss, while the second term regularises the latent representation Z to stay close to prior p(z), which we define as a unit Gaussian N (0,I). This regularisation means that at inference, an OOD input oˆ is likely to produce latent feature zˆ that diverges significantly from the prior.

Where the gains come from

OOD detection involves applying conformal prediction (CP) to obtain time-varying thresholds on KL scores OOD recognition is achieved with Grad-CAM, which computes gradients of the prediction with respect to feature map activations in preceding convolutional layers. The recognition result is then computed as MGrad-CAM = ReLU X k αkA k! p. 3

Where the paper says it

Specifically, we compute the weight for the k-th feature map of the layer as: αk = 1 N X i X j ∂KL[qϕ(zt|ot)∥N (0,I)] ∂Ak ij (4) where Ak is the k-th feature map and N normalises across all Ak . The recognition result is: MGrad-CAM = ReLU X k αkA k !

(5). These methods are shown to perform best when using KL-based OOD scores for detection and Grad-CAM for recognition InFeR-DEC achieves the best performance by a clear margin, with high TPR and relatively few false positives. p. 5
Where the paper says it

Both InFeR methods achieve high AUC scores across both detection and recognition compared with density-based approaches (NF), one-class discriminators (RND), and reconstruction-based approaches (AE, VAE-R). InFeR-DEC achieves the best performance by a clear margin, with high TPR and relatively few false positives.

Failure Resilience in Practice

In real-world experiments on a Boston Dynamics Spot robot, InFeR enables two policies—DECISION [2] and GNM [3]—to detect and recover from critical failures in the real world. The framework successfully handles four types of critical Navigation Failures (NF1–NF4). p. 3

Where the paper says it

We train our policies and baselines on the training dataset from [2], comprising 30 hours of teleoperated trajectories demonstrating the task of path-following navigation in indoor and outdoor scenes. We identify four types of critical Navigation Failures (NF) to detect and recover from, which are OOD for this dataset: NF1 Sensor failure.

InFeR effectively recovers from diverse real-world failures and enables the DECISION policy to autonomously complete a 300m route with varied terrain despite critical failures. p. 5
Where the paper says it

Specifically, we test whether InFeR policies can detect and recover from the key failure types identified (NF1–NF4). To illustrate the importance of informed failure resilience, we show that InFeR allows the DECISION policy to autonomously complete a 300m route with varied terrain (cluttered indoor plaza, outdoor park with forested paths) despite critical failures.

The performance of InFeR-DEC consistently outperforms InFeR-DEC-Blind, especially in recovering from close-range dynamic obstacles (NF3). p. 6
Where the paper says it

We compare InFeR-DEC with an uninformed variant (InFeR-DEC-Blind) that ignores the recognition heatmap Mt, and randomly selects corrective actions. InFeR-DEC consistently outperforms InFeR-DEC-Blind, especially in recovering from close-range dynamic obstacles (NF3), where timely, informed actions are critical.

Conclusion

We argue that informed failure resilience in IL policies extends beyond detection to encompass recognising failure sources and autonomously recovering from them. InFeR requires no additional data and no changes to the IL policy’s architecture, generalizing across two distinct policy architectures and a range of diverse failure modes. p. 7

Where the paper says it

It requires no additional data, and no changes to the IL policy’s architecture. InFeR generalises across two distinct policy architectures, and a range of diverse failure modes across different scenes, enabling IL policies to succeed at long-range navigation in complex environments without intervention.

This shows that InFeR is a clear step toward safe, robust navigation with IL. p. 7
Where the paper says it

InFeR generalises across two distinct policy architectures, and a range of diverse failure modes across different scenes, enabling IL policies to succeed at long-range navigation in complex environments without intervention. This shows that InFeR is a clear step toward safe, robust navigation with IL.

The paper concludes that InFeR enables high detection and recovery success across varied failure types and demonstrates its generalisability across different policy architectures. p. 5
Where the paper says it

Overall, we draw three specific conclusions: InFeR effectively recovers from diverse real-world failures. Table II shows that InFeR enables high detection and recovery success across varied failure types, supporting H2.

The results support the hypothesis that InFeR produces OOD-aware policies effective at task-relevant OOD detection and recognition. p. 5
Where the paper says it

This yields the per-bin recognition score only when the input is flagged as OOD. The results in Figure 3 support H1: InFeR produces OOD-aware policies effective at task-relevant OOD detection/recognition.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Variational Information Bottleneck (VIB) loss for latent space structure allows InFeR to distinguish in-distribution from out-of-distribution inputs, enabling OOD detection without additional training data or architectural changes.

  2. Grad-CAM is applied to the KL-based OOD scores predicted by the model to localize visual regions responsible for the detection, providing a spatial source of failure information for recovery.

  3. The system can autonomously recover from critical failures in real-world scenarios, such as sensor failures (NF1), blocked pathways (NF2), and close-range dynamic obstacles (NF3), without requiring human intervention.

  4. InFeR enables informed recovery by pairing the IL policy with a heuristic recovery policy that selects corrective actions based on the OOD recognition heatmap to maximally avoid OOD features in the observation ot.

  5. The system can generalize across different visual navigation policies, specifically demonstrating effectiveness for both DECISION and GNM architectures, supporting H3.

Abstract

While imitation learning (IL) has enabled successful visual navigation in many common environments, IL policies are prone to unpredictable failures under out-of-distribution (OOD) scenarios. This necessitates failure-resilient policies, which not only detect failures, but also recognise their sources and recover from them autonomously. We propose InFeR, a general framework for building IL policies with informed failure resilience without failure or recovery demonstrations. Given an IL policy's architecture and training dataset, InFeR retrains a new OOD-aware policy with a latent space structured for OOD failure detection, with a modified objective based on the Variational Information Bottleneck. It applies a visual explainability technique, Grad-CAM, to attribute failures to specific image regions and inform a heuristic policy for recovery. All these are achieved without requiring additional training data. Real-world experiments show that InFeR enables informed failure recovery across two different policy architectures, supporting long-range navigation in complex environments.

Sources

Related papers