Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning".
Jane: Here's the gist of the paper. Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning proposes a sensor-guided adaptive contrastive learning framework to enable robust scene transfer in vision-based PointGoal navigation During training, privileged LiDAR sensing guides the contrastive objective through a geometry-aware similarity metric and adaptive temperature scaling, encouraging visual embeddings to capture navigation-relevant structure rather than scene-specific appearance The resulting encoder is pretrained independently, frozen, and used as the perceptual backbone for reinforcement learning.
Tom: Core Methodology The framework separates visual representation learning from policy optimization by pretraining a vision encoder with a privileged sensor–guided contrastive objective using replay correspondences and geometric observations available only during representation learning.
Jane: Once pretraining is complete, the encoder is frozen and used as a feature extractor for a lightweight policy trained via model-free RL.
Tom: The navigation policy receives an observation consisting of the visual embedding zt extracted from the RGB image It, the robot’s linear and angular velocities, vt, ψ˙t respectively, and the relative position of the goal in polar coordinates gt.
Title and authors: Tom: What the paper actually summarizes is that they decouple the visual representation learning from the policy optimization, which is a smart structural move for this kind of task <ref:2606.05506#pg1>.
Jane: That decoupling means you train this vision encoder once using all that geometry information, and then you freeze it before training a separate policy network with reinforcement learning <ref:2606.05506#pg3>.
Lu: They do this to ensure the encoder is pretrained independently, which they say helps in suppressing environment-specific shortcuts during the policy learning phase <ref:2606.05506#pg1>.
Meng: But how does that decoupling help with deployment? If you freeze it, isn't it still tied to that training environment?
Tom: That’s where they introduce a cross-stage domain mismatch between the pretraining and the policy learning, which is a specific design choice to suppress those environment-specific shortcuts <ref:2606.05506#pg1>.
Jane: So, by training the robot in a different environment than how it learned its vision features, they force the policy to rely on features that stay predictive across different scenes <ref:2606.05506#pg1>.
Lu: And that cross-stage domain mismatch is key because it tests how well those learned representations truly capture navigation structure instead of just looking like a specific room <ref:2606.05506#pg3>.
The paper's summary: Tom: The core improvement they highlight is the geometry-aware contrastive representation learning objective, which uses that adaptive temperature scaling we talked about earlier <ref:2606.05506#pg3>.
Jane: So, instead of a fixed similarity score for positive and negative pairs in contrastive learning, they make the repulsive forces change based on how geometrically similar two images are <ref:2606.05506#pg4>.
Lu: They define this geometry-based similarity metric using LiDAR readings available during training to create a task-aware similarity that steers the embedding away from appearance features <ref:2606.05506#pg4>.
Meng: That sounds mathematically complex to implement, but if it works, does it actually translate into better control commands?
Tom: The results show that this leads to substantially higher control alignment in both training and hold-out scenes compared to baselines <ref:2606.05506#pg8>.
Jane: And the paper shows that even when they introduce a global appearance shift, where the floor and background are just changed to two uniform colors, their method still maintains eighty-two point six percent success rate <ref:2606.05506#pg11>.
The paper's improvements: Tom: So to wrap up, the big takeaway from this paper is that by using privileged LiDAR sensing only during training to guide contrastive learning, you get a vision encoder that learns geometry-aligned representations <ref:2606.05506#pg1>.
Jane: And because they used decoupling and cross-stage domain mismatch, you end up with a navigation policy that generalizes robustly across indoor and outdoor environments without needing those privileged sensors at deployment <ref:2606.05506#pg2>.
Lu: I think the most important part is how it measures the representation quality using embedding–action distance correlation, showing stability even under scene shift <ref:2606.05506#pg8>.
Meng: From a deployment standpoint, that decoupling is crucial because it means we are shipping a lightweight policy that doesn't rely on hardware we can't guarantee will be present in the field <ref:2606.05506#pg2>.
Lalam: I can confirm, based on the literature I review, this approach successfully learns representations invariant to appearance while still encoding the necessary geometric structure for task performance <ref:2606.05506#pg1>.
Tom: That’s the summary of "Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning". It’s a solid method, but remember, it still requires that geometry information during training to get those strong results <ref:2606.05506#pg3>.
Jane: Yeah, and they've released a multimodal dataset to support future research on this privileged-guided visual representation learning for navigation <ref:2606.05506#pg1>.
Lu: We should keep watching how they explore reducing reliance on explicit task inputs like the goal position in future work <ref:2606.05506#pg1>.
Conclusion: Tom: So we've been talking about "Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning." Basically, they figured out how to train a vision encoder using LiDAR data during training to make it good at navigation even when the robot sees a totally different room later on.
Jane: Right, so the big idea is that they use those privileged LiDAR readings not just as extra data, but they actively guide how the vision encoder learns its features through this geometry-aware similarity metric <ref:2606.05506#pg4>.
Lu: What’s interesting is how they modulate the temperature parameter based on that geometric similarity; it means geometrically similar images get weaker repulsion, which is a clever way to make sure the embeddings capture structure instead of just random scene texture <ref:2606.05506#pg4>.
Meng: But from what I’m seeing at scale, if you freeze that encoder after pretraining, how do you stop the policy network from just overfitting to the training environment's specific quirks?
Tom: That's where the cross-stage domain mismatch comes in; they train it in one set of environments and then test it on a completely different set during policy learning <ref:2606.05506#pg6>. It’s a deliberate way to force the policy to rely on features that actually work across those shifts.
Jane: And the result is they show this method can keep an eighty-two point six percent success rate when tested under simple visual changes, like just swapping out the floor color, which is pretty strong <ref:2606.05506#pg11>.
Lu: It proves that representation learning doesn't have to be scene-specific if you give it the right geometric supervision during training <ref:2606.05506#pg8>.
Meng: For deployment, that means the robot only needs its camera and standard stuff like goal position, no need to carry a LiDAR unit on board for navigation <ref:2606.05506#pg2>. That lowers the hardware cost significantly.
Tom: It's about making sure the underlying understanding is geometric structure rather than just memorizing what a specific room looks like <ref:2606.05506#pg3>.
Jane: It’s a really important step toward making robot navigation truly robust for real-world deployment, especially when dealing with those unpredictable visual changes we see every day <ref:2606.05506#pg11>.
Lu: The dataset they released is also helpful because it gives others a chance to build on this privileged-guided approach for navigation <ref:two thousand six hundred six point zero five five zero six#pg1.
Meng: I just wonder what the cost is for running this whole setup, but if it's as cheap as just using a standard model, then that’s how we deploy it <ref:2606.05506#pg2>.
Tom: Exactly. So that's the gist of "Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning." It shows us how to make vision models learn what matters for navigation, not just what matters for a single training scene <ref:two thousand six hundred six point zero five five zero six#pg1.
Jane: Next time, we'll look at how this contrasts with other representation learning methods, and why that cross-stage mismatch is so fundamental <ref:2606.05506#pg4>.
cs.CV
Submitted: 2026-06-03
Updated: 2026-11-02
Comments: 8 pages, Accepted to IEEE Robotics and Automation Letters (RA-L)
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning proposes a sensor-guided adaptive contrastive learning framework to enable robust scene transfer in…
Key concepts
- Geometry-Aware Contrastive Representation Learning
- This method modifies standard contrastive learning by incorporating geometric similarity derived from LiDAR readings into the loss function. This ensures that visual embeddings are aligned based on the underlying spatial structure of the scene, making them invariant to changes in surface appearance.
- Cross-Stage Domain Mismatch (CSDM)
- The training environment for representation learning is intentionally different from where the policy is trained. This mismatch prevents the encoder from memorizing environment-specific visual shortcuts. It forces the learned features to capture general, task-relevant geometry that remains predictive across different scenes.
- Geometry-Adaptive Temperature
- The temperature parameter in contrastive learning is dynamically adjusted based on geometric similarity. If two images are geometrically similar, the repulsive force between their embeddings is weakened. This means that geometrically similar but visually distinct images are treated less harshly, promoting better structural alignment.
- Decoupled Pretraining and Policy Optimization
- The system separates learning the visual features (pretraining) from learning how to navigate (policy optimization). The visual encoder is trained first using geometric cues, then frozen. A separate reinforcement learning policy is then trained on this stable feature set, ensuring the policy relies on robust, scene-agnostic representations.
Terminology
Summary
Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning proposes a sensor-guided adaptive contrastive learning framework to enable robust scene transfer in vision-based PointGoal navigation During training, privileged LiDAR sensing guides the contrastive objective through a geometry-aware similarity metric and adaptive temperature scaling, encouraging visual embeddings to capture navigation-relevant structure rather than scene-specific appearance The resulting encoder is pretrained independently, frozen, and used as the perceptual backbone for reinforcement learning. p. 0 p. 0
Core Methodology
The framework separates visual representation learning from policy optimization by pretraining a vision encoder with a privileged sensor–guided contrastive objective using replay correspondences and geometric observations available only during representation learning. Once pretraining is complete, the encoder is frozen and used as a feature extractor for a lightweight policy trained via model-free RL. p. 1 p. 1 The navigation policy receives an observation consisting of the visual embedding zt extracted from the RGB image It, the robot’s linear and angular velocities, vt, ψ˙t respectively, and the relative position of the goal in polar coordinates gt. p. 3
Geometry-Aware Contrastive Representation Learning
The objective of the representation learning stage is to train a vision encoder that produces embeddings aligned with navigation-relevant geometry rather than scene appearance. This involves building upon standard contrastive learning but incorporating privileged geometric information available during training to modulate the loss in a task-aware manner. p. 2 Key components include:
-
Contrastive Learning Objective, which minimizes the InfoNCE objective [24] for each positive pair in the batch. p. 2
-
Task-Aware Similarity Metrics, where normalized geometric similarity siml(I1, I2) is computed using LiDAR readings l available during training to define task-relevant similarity. p. 2
-
Geometry-Adaptive Temperature, where the adaptive temperature τ (I1, I2) is defined as α if (I1, I2) is a positive pair and siml(I1, I2) if (I1, I2) is a negative pair. p. 2
Where the Gains Come From
The gains stem from several key design choices that enforce scene-agnostic decision-making. The authors emphasize decoupling representation learning from policy optimization to learn task-relevant visual features first, then train a policy on the frozen representation. p. 1 Furthermore, they introduce a cross-stage domain mismatch (CSDM) between representation pretraining and policy learning to suppress environment-specific shortcuts and promote reliance on task-relevant features. p. 0 This mismatch forces the policy to exploit features that remain predictive under domain shift, providing a stronger test of geometry-aligned representation learning. p. 3
Evaluation and Robustness Analysis
Extensive experiments in high-fidelity simulation demonstrate that our approach significantly improves policy-level scene transfer across diverse indoor and outdoor environments At deployment, the agent relies only on monocular RGB observations together with standard task-related inputs such as goal position and proprioceptive signals, without access to LiDAR or other privileged sensors. p. 0 The method outperforms large pretrained vision models, navigation-specific representation learning approaches, domain randomization methods, and standard contrastive baselines under severe appearance and semantic shifts. p. 0 The results show that our method maintains 82. p. 66% SR when tested against controlled visual perturbations with fixed floor appearance. p. 4 Additionally, the cross-stage domain mismatch consistently yields substantial gains across configurations, demonstrating its fundamental importance in learning robust representations. p. 6
Conclusion and Future Work
We presented a sensor-guided contrastive representation learning framework for robust scene transfer in vision-based PointGoal navigation. By leveraging privileged geometric sensing only during training to adaptively modulate the contrastive objective, our method learns navigation-relevant visual representations invariant to scene appearance. p. 6 Extensive experiments demonstrate substantial policylevel generalization across unseen indoor/outdoor environments under severe appearance shifts. p. 7 Future work will explore reducing reliance on explicit task inputs like goal position, the interaction between geometry-aware objectives and image augmentation strategies, and which privileged sensors most effectively guide representation learning. p. 7
--- Page 1 ---
Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning Amirhossein Zhalehmehrabi, Tiziano Tezze, Alberto Castelini, and Alessandro Farinelli Abstract— We propose a sensor-guided adaptive contrastive learning framework for visual representation learning in PointGoal navigation. During training, privileged LiDAR sensing guides the contrastive objective through a geometry-aware similarity metric and adaptive temperature scaling, encouraging visual embeddings to capture navigation-relevant structure rather than scene-specific appearance. p. 0 The resulting encoder is pretrained independently, frozen, and used as the perceptual backbone for reinforcement learning. We further introduce a crossstage domain mismatch between representation pretraining and policy learning to suppress environment-specific shortcuts and promote reliance on task-relevant features. p. 0 Extensive experiments in high-fidelity simulation demonstrate that our approach significantly improves policy-level scene transfer across diverse indoor and outdoor environments. p. 0 At deployment, the agent relies only on monocular RGB observations together with standard task-related inputs such as goal position and proprioceptive signals, without access to LiDAR or other privileged sensors. Our method outperforms large pretrained vision models, navigation-specific representation learning approaches, domain randomization methods, and standard contrastive baselines under severe appearance and semantic shifts. p. 0 We also release a multimodal dataset to support future research on privilegedguided visual representation learning for navigation. p. 0 The code is available at: https://anonymous. p. 04open.science/r/privilegedsensor-contrastive-nav-E278/README. p. 0md I INTRODUCTION Vision-based mobile navigation promises scalable autonomy by enabling robots to operate using only onboard cameras, avoiding reliance on additional sensing modalities such as depth or LiDAR. p. 0 Despite this appeal, vision-based navigation policies often fail when deployed in visually different environments, even when task dynamics and robot embodiment remain unchanged [1]. p. 0 This lack of scene transfer—the ability to reliably perform the same task across visually distinct environments—remains a major barrier to real-world deployment of learning-based navigation systems. p. 0 At the core of this challenge lies representation learning. p. 0 To generalize across environments, a navigation policy must rely on visual representations that suppress scene-specific appearance while preserving task-relevant structure such as goal geometry, free space, and obstacle layout. p. 0 While prior work has demonstrated strong performance within fixed environments [2]–[4], learning representations that generalize across environments without retraining remains challenging, particularly under large visual distribution shifts. p. 0 In contrast to robotic manipulation, where scene transfer has been extensively studied [5], vision-based navigation methods are typically evaluated in visually similar environments or rely on explicit geometric representations, such as mapping, SLAM, or depth sensing [6]–[9]. p. 0 In another line of research, contrastive representation learning provides a principled mechanism for enforcing invariance by aligning semantically similar observations [10]. p. 0 However, standard contrastive methods rely on heuristic augmentations and do not explicitly incorporate task structure, often resulting in representations that remain entangled with scene appearance under large cross-environment shifts [11]. p. 0 Recent work [12] incorporates privileged supervision into contrastive learning for agile flight. p. 0 However, this approach is trajectory-centric and demonstrated only for executing predefined motion patterns, limiting its applicability to general closed-loop navigation. p. 0 These limitations highlight a key gap: policy-level scene transfer requires representations that support scene-agnostic decision-making. p. 0 More broadly, this work explores a research direction in which privileged sensing modalities guide the representation learning of deployable sensors. p. 0 In this work, we instantiate this idea by using LiDAR observations to guide the learning of an RGB encoder. p. 0 We use privileged geometric sensing available during training to guide representation learning, arXiv:2606. p. 005506v2 [cs. p. 0CV] 6 Aug 2026
--- Page 2 ---
without requiring such sensors at deployment. Specifically, we incorporate geometric observations into the contrastive objective as an additional supervisory signal, encouraging the learned representation to encode the underlying geometric structure of the scene. p. 1 This geometry-guided regularization steers the representation away from appearance-specific features and toward features that remain stable across environments, resulting in more robust and transferable policies. p. 1 Specifically, we propose a sensor-guided adaptive contrastive learning framework in which privileged LiDAR observations modulate the contrastive objective through geometry-aware adaptive temperature scaling. p. 1 The resulting visual encoder is pretrained independently, frozen, and used as the perceptual backbone for reinforcement learning (RL). p. 0 At deployment, the navigation policy operates using only deployable sensing modalities, including monocular RGB observations and standard task inputs such as goal-relative pose and proprioceptive measurements, without access to the privileged sensing used during representation pretraining. p. 1 We summarize our contributions as follows: • Sensor-guided adaptive contrastive learning. p. 1 We propose a framework in which privileged sensors guide the representation learning of a target sensor by modulating the contrastive objective. p. 1 We instantiate this paradigm using LiDAR observations to guide an RGB encoder. p. 1 • Policy-level scene transfer with decoupled pretraining and RL. p. 1 We demonstrate that freezing a sensorguided pretrained visual-based encoder and training navigation policies via RL yields robust closed-loop generalization across visually distinct environments. p. 1 • Comprehensive evaluation and dataset release. p. 1 We demonstrate strong cross-scene generalization on PointGoal navigation and release a multimodal dataset to support future research on privileged-guided visual representation learning for navigation. p. 1
--- Page 3 ---
We propose a vision-based robot navigation framework that separates visual representation learning from policy optimization. The visual encoder is pretrained with a privileged sensor–guided contrastive objective using replay correspondences and geometric observations available only during representation learning. p. 1 Once pretraining is complete, the encoder is frozen and used as a feature extractor for a lightweight policy trained via model-free RL. A. Task Formulation We study the PointGoal Navigation task, where a robot must reach a target relative to its start, avoiding obstacles. p. 1 At each timestep, it receives partial observations and outputs continuous linear and angular velocities. p. 2 Episodes end on success or collision. p. 2 During deployment, the robot observes the goal in its local frame, velocities, and an RGB image, without access to obstacle geometry or absolute position. p. 2 The aim is to learn a navigation policy that generalizes across visually diverse environments using only these sensors. p. 2
--- Page 4 ---
B. Geometry-Aware Contrastive Representation Learning We build upon standard contrastive learning, but incorporate privileged geometric information available during training to modulate the loss in a task-aware manner. p. 2 Training batches are sampled uniformly. p. 2 Positives are augmented views of scene-invariant counterparts, while all other samples in the batch are treated as negatives. p. 2 Instead of altering this pair assignment, we adjust the contribution of each pair to the contrastive objective through a geometryadaptive temperature mechanism. p. 2 1) Contrastive Learning Objective: Let ϕθ denote a vision encoder parameterized by θ that maps an RGB image Ii to a latent embedding zi = ϕθ(Ii) ∈ R d. p. 2 Given a random batch of N images, we train ϕθ to minimize the InfoNCE objective [24] for each positive pair in the batch: Li,j = − log exp sim(zi, zj)/τ PN k=1 1[k̸=i] exp sim(zi, zk)/τ (1) where sim(·, ·) denotes cosine similarity and τ is a temperature parameter. p. 2 In contrast to standard formulations, τ is not fixed but adaptively modulated based on geometric similarity, as described in Section III-B. p. 23. 2) Task-Aware Similarity Metrics: To address this, we compute a geometry-based similarity using LiDAR readings l available during training. p. 2 Given two observations I1 and I2 with corresponding vectors l1 and l2, we define their normalized geometric similarity as siml(I1, I2) = 1 − qPl i=1 wi(l1,i − l2,i) 2 qPl j=1 wj (2) where l = l is the number of LiDAR beams, and wi ≥ 0 is the weight assigned to the i-th beam; their selection is detailed in Section V-A. p. 21. The subtracted term corresponds to a normalized weighted Euclidean distance in LiDAR space, bounded in [0, 1], such that higher values of siml indicate greater similarity in the local spatial structure surrounding the robot. p. 2 3) Geometry-Adaptive Temperature: Contrastive learning objectives are known to be sensitive to the temperature parameter [12], [25], [26], which controls the strength of attraction and repulsion in embedding space. p. 2 We therefore modulate the temperature on a per-pair basis according to geometric similarity, allowing repulsive forces to vary continuously with navigation-relevant structure. p. 2 Given a contrastive pair (I1, I2) with geometric similarity siml(I1, I2), we define the adaptive temperature as τ (I1, I2) = (α if (I1, I2) is a positive pair, siml(I1, I2) if (I1, I2) is a negative pair. p. 2 The resulting value is mapped to a bounded interval [τmin, τmax] to ensure numerical stability: τ′(I1, I2) = τmin + (τmax − τmin) · τ (I1, I2) (4). p. 2 This adaptive formulation ensures that geometrically similar observations, even when treated as negatives, apply weaker repulsive forces. p. 2
--- Page 5 ---
C. Action Net Learning The navigation policy is composed of a frozen vision encoder and a lightweight action network. p. 2 At each timestep t, the policy receives an observation ot consisting of the visual embedding zt = ϕθ(It) extracted from the RGB image It, the robot’s linear and angular velocities, vt, ψ˙t respectively, and the relative position of the goal in polar coordinates gt. p. 3 Formally, the observation is defined as: ot = zt, vt, ψ˙t, gt (5). p. 3 The action network maps observation to continuous control commands: linear velocity ut and angular velocity omegat. p. 3 The action network is optimized using a model-free RL algorithm. p. 3 By fixing the visual representation during policy learning, the action network is trained on a stable and sceneinvariant input distribution, which helps reduce overfitting to the training environments and isolates the effect of representation learning on policy generalization. p. 3 The reward function is inspired by [27] and is defined as R(st, at, st+1) = rsuc if success, rcollide if collision, −α1∆d − α2rback − α3rangular otherwise. p. 3
--- Page 6 ---
D. Cross-Stage Domain Mismatch (CSDM) The environments used for representation pretraining (Fig. p. 3 2 (a)) differ from those used during action net learning (Fig. p. 3 2 (b)). This cross-stage domain mismatch is intentional and serves to isolate the effect of representation learning on policy generalization. p. 3 By freezing the pretrained encoder, the visual feature space remains fixed during RL. p. 3 Training the policy in a distinct environment discourages reliance on scene-specific correlations that may arise during pretraining, as such correlations do… p. 3
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Visual encoder pretraining now captures geometry-aligned representations by using privileged LiDAR observations to modulate the contrastive objective through a geometry adaptive temperature mechanism, which ensures that
geometrically similar observations, even when treated as negatives, apply weaker repulsive forces.
-
The navigation policy decouples representation learning from policy optimization by freezing the pretrained encoder and training a lightweight action network via model-free RL on distinct environments to suppress
environment-specific shortcuts and promote reliance on task-relevant features.
-
The system achieves robust closed-loop generalization across visually distinct indoor and outdoor environments, retaining high success rates even under severe appearance shifts because it
does not require access to LiDAR or other privileged sensors
at deployment.
Sources
- Representation Learning with Contrastive Predictive Coding
- DINOv2: Learning Robust Visual Features without Supervision
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs