CHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies
summary
The gist
CHOP introduces Counterfactual Human Preference Labels to align visuomotor navigation policies with human intuition of safety and obstacle avoidance in navigation The method leverages counterfactual…
In short
CHOP uses counterfactual human preference labels to teach navigation policies how humans intuitively avoid obstacles and prioritize safety. By comparing different potential paths, it aligns robotic movement with human intuition regarding collision risk and efficiency, leading to significantly safer navigation in both simulation and real-world robot tests.
Key concepts
- Counterfactual Human Preference Labels
- These are binary labels where human annotators compare two possible future actions or trajectories. They tell the system which path is preferred based on anticipated outcomes like avoiding a collision or reaching a goal efficiently, allowing the policy to learn from what humans would choose in specific situations.
- Counterfactual Preference Dataset
- This is a large dataset created by comparing many pairs of alternative navigation trajectories. This dataset helps train the policy by showing it which trajectory is superior based on human preference labels, enabling the robot to learn safer movement patterns than just following pre-set rules.
- Policy Fine-tuning
- This involves adjusting the robot's movement algorithm using the preference data. The system selects trajectories that win most human comparisons and uses a loss function to train the policy to mimic these preferred, safer actions, thus aligning its behavior with human intuition.
Terminology used across episodes
This episode discusses
- CHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies · Paper Radio
- RT-1: Robotics Transformer for Real-World Control at Scale
- VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation
- NaVILA: Legged Robot Vision-Language-Action Model for Navigation
- CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
- OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation
- GuideNav: User-Informed Development of a Vision-Only Robotic Navigation Assistant For Blind Travelers
- pi 0.5: a Vision-Language-Action Model with Open-World Generalization
- Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback
- Semi-parametric Topological Memory for Navigation
- Proximal Policy Optimization Algorithms
- HALO: Human Preference Aligned Offline Reward Learning for Robot Navigation
- GNM: A General Navigation Model to Drive Any Robot
- ViNT: A Foundation Model for Visual Navigation
- Parameter Efficient Reinforcement Learning from Human Feedback
- OpenBench: A New Benchmark and Baseline for Semantic Navigation in Smart Logistics
- CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidance
The paper
CHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies · Read on arXiv
Gershom Seneviratne, Jianyu An, Vaibhav Shende, Sahire Ellahy, Yaxita Amin, Kondapi Manasanjani, Samarth Chopra, Jonathan Deepak Kannan, Dinesh Manocha
University of Maryland
Visuomotor navigation policies have shown strong perception-action coupling for embodied agents, yet they often struggle with safe navigation and dynamic obstacle avoidance in complex real-world environments. We introduce CHOP, a novel approach that leverages Counterfactual Human Preference Labels to align visuomotor navigation policies towards human intuition of safety and obstacle avoidance in navigation. In CHOP, for each visual observation, the robot's executed trajectory is included among a set of counterfactual navigation trajectories: alternative trajectories the robot could have followed under identical conditions. Human annotators provide pairwise preference labels over these trajectories based on anticipated outcomes such as collision risk and path efficiency. These aggregated preferences are then used to fine-tune visuomotor navigation policies, aligning their behavior with human preferences in navigation. Experiments on the SCAND dataset show that visuomotor navigation policies fine-tuned with CHOP reduce near-collision events by 49.7%, decrease deviation from human-preferred trajectories by 45.0%, and increase average obstacle clearance by 19.8% on average across multiple state-of-the-art models, compared to their pretrained baselines. These improvements transfer to real-world deployments on a Ghost Robotics Vision60 quadruped, where CHOP-aligned policies improve average goal success rates by 24.4%, increase minimum obstacle clearance by 6.8%, reduce collision and intervention events by 45.7%, and improve normalized path completion by 38.6% on average across navigation scenarios, compared to their pretrained baselines. Our results highlight the value of counterfactual preference supervision in bridging the gap between large-scale visuomotor policies and human-aligned, safety-aware embodied navigation.
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "CHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies".
Dev: Here's the gist of the paper. CHOP introduces Counterfactual Human Preference Labels to align visuomotor navigation policies with human intuition of safety and obstacle avoidance in navigation The method leverages counterfactual human preference labels to align visuomotor navigation policies towards human preferences in navigation The Core Methodology CHOP instantiates counterfactual human preference supervision at scale via large-scale counterfactual preference data annotation, enabling the alignment of visuomotor policies with human preferences The training pipeline consists of three stages: (1) Counterfactual Preference Dataset Generation, (2) Policy Fine-tuning, and (3) Deployment in real-world scenarios.
Rosa: In the generation phase, counterfactual alternatives are created using two complementary mechanisms: human-guided target annotation and structured trajectory perturbations.
Dev: These mechanisms result in a set of trajectories denoted as Tcf = τ0, τ1,..., τm−1.
Rosa: The Training Signal Construction Human preference labels are gathered by having annotators provide pairwise preference labels over these counterfactual trajectories based on anticipated outcomes such as collision risk and path efficiency.
Title and authors: Rosa: So we're looking at this paper called CHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies. It sounds like they're trying to get robot policies to actually match what humans instinctively feel is safe when navigating.
Dev: Yeah, that’s the core idea. They’re using these counterfactual labels—basically showing the AI what other paths could have been taken—to train the policy to choose the path humans prefer, instead of just following whatever path it already learned from pretraining.
Taro: I wonder if that actually works when things get messy in the real world. Because right now, these navigation policies are often safe but just not efficient or intuitive for a human driver.
Rosa: Exactly, and CHOP aims to fix that by feeding the robot data where humans explicitly show preference over different trajectories, like which one minimizes collision risk or keeps you moving smoothly.
Dev: The way they build this data is pretty clever. They create these counterfactual alternatives using two methods: having humans directly label them, and also adding structured perturbations to the existing paths to generate new ones for comparison.
Taro: So it’s not just about showing a robot what to do, but forcing it to reason about what *could* have happened if it had chosen something else. That seems like a big jump from just following pre-trained rules.
Rosa: It is because they gather these binary preference labels by having annotators compare pairs of these counterfactual trajectories based on things like collision risk or path efficiency, which results in this large scale dataset.
Dev: This leads to their training signal construction where they select the most preferred trajectory, the one that wins the most pairwise comparisons for a given observation, and then fine-tune the existing navigation policy towards that winning path using a supervised imitation loss.
Taro: So it’s like teaching the robot not just *how* to move, but *why* a human would prefer one move over another in that exact situation. That’s a different kind of learning signal than just rewarding successful movement.
Rosa: Right, and this process is repeated across their entire dataset to form this large scale counterfactual preference dataset which they call the CHOP counterfactual preference dataset.
Title and authors: Dev: The results on offline evaluation are pretty telling too; for models like OmniVLA, CHOP fine-tuning reduced near collision events by sixty three point four percent and deviation from human preferred trajectories by forty nine point five percent.
Taro: That’s a significant shift in the distribution of behavior, not just a small tweak to the performance numbers. I mean, that suggests they aren't just getting slightly better at avoiding crashes; they’re changing their fundamental approach to safety.
Rosa: It does, and this isn't just about avoiding crashes in simulation; they also showed improvements in average obstacle clearance by nineteen point eight percent on average across multiple state of the art models when compared to their pretraining baselines.
Dev: When we look at the real world deployments on a Ghost Robotics Vision60 quadruped, those safety improvements hold up consistently, and it’s not just in the lab environment.
Taro: What I found interesting there is how these models start internalizing behaviors that weren't explicitly taught in the original training data. For instance, they sometimes show stopping and waiting when faced with tight dynamic obstacles, which is something baseline models often struggle with or don't do gracefully.
Rosa: That’s a key point because it suggests that these human preference labels allow the policies to internalize stopping and waiting as valid safety-oriented behaviors even when those specific actions weren't the most frequent in the initial data.
Dev: And their distributional analysis confirms this, showing a clear leftward shift in near collision distributions, which really indicates a systematic reduction in unsafe interactions across the whole population of tests.
Taro: So it’s not just about hitting fewer obstacles; it’s about shifting the entire way the robot navigates toward what we consider safer and more human-like movement patterns. That speaks to a deeper alignment with intuition.
Rosa: So, to wrap up, CHOP uses counterfactual human preference labels to align visuomotor navigation policies with human intuition of safety by making them select trajectories that are better than the operator action sequence.
Dev: It’s powerful because it scales this supervision via large scale counterfactual preference data and shows consistent safety improvements in real world deployments like on the Ghost Robotics Vision60 quadruped.
Taro: And maybe most importantly, it allows these policies to internalize complex safety behaviors like stopping and waiting when those actions are preferred by human annotators.
Title and authors: Rosa: So, this paper is showing us a way to inject human intuition about safety directly into the training loop by using counterfactual reasoning for obstacle avoidance and planning.
Dev: It moves beyond just optimizing a single path to aligning the entire policy toward a distribution of paths that humans deem safer and more efficient in ambiguous navigation environments.
Taro: The limitation they flag is that this method relies entirely on getting high quality human preference labels, which can be time consuming to generate at scale.
Rosa: Yeah, so while it’s very effective, the practical challenge for deploying this kind of supervision is getting that large scale data annotation done reliably enough to make the system robust outside of the controlled lab setting.
Dev: That makes sense from a controls engineer's view; you can have a perfect loss function, but if you can't generate enough high quality preference pairs, the alignment won't stick when things go wrong in real time.
Taro: It definitely sounds like the next step is making the generation of those counterfactual data points more automated or robust to noise so we don't get stuck needing human annotators for every single test case.
Rosa: Exactly, and that’s where future work probably needs to focus—making this large scale counterfactual preference supervision applicable in a way that doesn't rely entirely on slow, manual labeling processes.
Dev: It seems like the implication is that we can move away from purely reward-based or geometric safety constraints toward incorporating more nuanced, qualitative human safety judgments into the policy training itself.
Taro: That’s what makes it interesting; we’re moving closer to a system that understands not just collision geometry, but the context of risk as perceived by a human operator.
Rosa: So we've seen how CHOP uses counterfactual human preference labels to improve obstacle avoidance in visuomotor navigation policies and it shows consistent safety gains when deployed on real platforms.
Dev: It’s a solid piece of work that demonstrates how incorporating reasoning about alternative actions can lead to policies that are not just safe, but also aligned with human intuition about what’s right in ambiguous situations.
Taro: We should keep an eye on how they handle those edge cases where the world misbehaves unexpectedly, because that's where we really see if this internalization of safety works when the robot encounters something it hasn't seen before.
The paper's summary: Rosa: So, to wrap up what we just talked about, CHOP is basically this method that uses counterfactual human preference labels to fine-tune navigation policies so they actually match what a human would instinctively do when avoiding obstacles.
Dev: Right, and the core idea is that instead of just training the AI on what works geometrically, you're feeding it examples where humans have already made choices about which path is safer or more efficient.
Taro: It’s not just about picking a better pre-trained route; it’s about forcing the policy to understand the *reason* behind human intuition.
Rosa: Exactly. They generate these counterfactual alternatives—those are hypothetical paths that could have been taken instead of the actual one taken—and then human annotators pick which one they prefer based on things like how much collision risk or how smooth the path is.
Dev: That preference data builds this huge dataset, and then the AI learns by minimizing a loss function that pushes its predicted path to match that preferred human trajectory.
Taro: The results in simulation are pretty big too; they showed a substantial drop in near-collision events, like forty-nine point seven percent reduction on the SCAND dataset for some models.
Rosa: And what’s really interesting is that this isn't just a small tweak to performance; the distributional analysis shows these policies shift their behavior towards safer zones systematically across the whole set of tests.
Dev: That means they aren't just slightly better at avoiding crashes in a specific lab setup; they’re changing the fundamental way the robot plans its movements in ambiguous situations.
Taro: And when you move it to real-world testing, like on that Ghost Robotics Vision60 quadruped, those safety gains stay consistent and it works even when things get messy outdoors.
Rosa: What’s really telling is that these models start internalizing behaviors they weren't explicitly programmed to do, like stopping or waiting when faced with tight dynamic obstacles.
Dev: That’s a big deal for the safety case because those kinds of hesitation behaviors aren't always easy to program directly; this method seems to help the AI learn that stopping is a valid safety action.
Taro: It suggests that human preference labels allow policies to internalize these complex safety behaviors, even when those actions are rare in the original training data.
Rosa: So, essentially, CHOP takes qualitative human judgment about what’s safe and turns it into a quantitative signal that directly shapes how the navigation policy behaves in real-time loops.
Dev: It moves us past just geometric constraints and reward functions toward incorporating nuanced, human-like safety judgments right into the training process.
Taro: That opens up possibilities for creating systems that navigate not just efficiently, but with an intuitive sense of risk that mirrors how a person would operate a machine.
Rosa: And while this is powerful, the authors do flag that the whole thing relies heavily on getting those high-quality human preference labels at scale, which can be really time consuming to generate.
Dev: That’s the practical hurdle we have to consider; you can have a perfect training signal, but if you can't generate enough reliable preference pairs for the AI to learn from, it won't stick when things get complicated in the real world.
Taro: So the next big question is how we make those counterfactual data points generated by humans more automated or robust so we don't always need a human annotator for every single tricky situation.
Rosa: That’s where the future work needs to focus, making this large scale supervision applicable in a way that doesn't rely entirely on slow, manual labeling processes.
The paper's improvements: Rosa: So, we’ve talked about how CHOP uses human preference labels to align robot movement with intuition, and now let's talk about what this paper actually suggests we should do next.
Dev: The authors are pointing out that they need to focus on making this whole supervision process much more scalable than it is right now, because relying on slow human annotation just won’t cut it for real deployment.
Taro: Right, and I think the biggest improvement suggested is moving away from just labeling trajectories to something that captures the actual *risk* involved in those choices.
Rosa: They are suggesting we need a way to quantify those preferences so the AI isn't just learning which path looks aesthetically pleasing, but which path minimizes potential harm in a measurable way.
Dev: That means developing better metrics for measuring collision risk and efficiency that aren't just simple geometric calculations, something that reflects what a human actually considers dangerous or good.
Taro: It’s about making the signal richer so the policy learns deeper safety rules rather than just surface-level avoidance patterns.
Rosa: And on the deployment side, they are emphasizing that we need to test this in much more complex, messy real-world environments than just controlled lab settings to see if these gains hold up under pressure.
Dev: Exactly; the simulation results are great, but I want to know how the system behaves when it encounters an unexpected obstacle or a situation that falls outside the scenarios we used for training.
Taro: The paper implies that we need a more robust way for the AI to handle those edge cases, so instead of just failing or defaulting to a blind move, it should have learned from those preferences how to react gracefully.
Rosa: So what this means for us is that we need to build systems where the human preference data informs a much broader set of safety rules, not just specific path choices.
Dev: It’s about creating a framework where the AI can generalize these nuanced human judgments across different types of navigation tasks, not just one specific obstacle avoidance scenario.
Taro: And looking at this alongside other papers, it seems like the real next step is combining this preference-based learning with those dynamic motion planning tools we see in other research to make the planning itself more context-aware.
Rosa: So we’re moving toward a system that doesn't just react to what’s in front of it, but understands the human concept of safety and risk as a whole.
Conclusion: Rosa: So to wrap up this discussion on CHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies, we’ve seen how using human intuition to supervise AI policies can lead to consistent safety improvements in real-world navigation tasks.
Dev: It really shows that injecting qualitative human judgment directly into the training loop through counterfactual reasoning works well for improving obstacle avoidance and goal adherence.
Taro: I think the biggest thing we’re seeing here is how this method allows AI to develop behaviors like stopping and waiting because humans have shown a preference for those actions in certain situations.
Rosa: That internalization of safety behaviors is pretty significant, especially when we look at real-world trials on platforms like the Vision60 quadruped where it maintained its performance even when things got tricky.
Dev: The implication for controls engineering is that we might be able to design safety constraints that aren't just hard geometric boundaries but are informed by a broader understanding of what’s intuitive and safe for a human operator.
Taro: And the limitation, as the authors pointed out, is the reliance on high-quality human preference labels which can be slow and expensive to generate at scale for every scenario.
Rosa: Right, so while it works beautifully in controlled settings, making that data generation faster or more automated is definitely the next major hurdle for this approach.
Dev: I think that’s where the focus should shift—on developing better ways to create those large scale counterfactual preference datasets without needing a human to label every single path comparison.
Taro: That brings us right back to making the generation of those counterfactual data points more robust so we don't get stuck needing slow, manual labeling for every single tricky situation.
Rosa: So that’s where we head off to next week, where we’ll be looking at some other papers on how AI is handling complex planning problems in unstructured outdoor environments.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies