HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping

arXiv:2602.16705 · cs.RO, cs.CV · Submitted 2026-02-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping".

Dev: The gist:

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper called "HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping," and the authors are Runpei Dong, Ziyan Li, Arjun Gupta, and Xialin He. What's the main point here in plain language?

Dev: It’s about giving humanoids the ability to pick up stuff they haven't seen before in any room. The title says it combines visual whole-body open-vocabulary object grasping with an accurate end-effector tracking policy, and that sounds pretty ambitious for real life application.

Taro: It suggests these systems can handle reaching across tables, rotating the torso for objects on the side, or squatting for low tables while keeping things balanced. That level of dexterity in novel scenes is what they are aiming for with this work.

Rosa: Exactly, it’s moving beyond just picking up known items to actually reaching into an environment and figuring out how to grasp something new based on a simple text command like "pick up the orange mug." It tackles that fundamental problem of generalizable scene understanding and precise physical control all at once.

Dev: It suggests they are breaking the monolithic end-to-end learning approach that most people are using right now because that makes scaling way too hard for real-world use. Instead, they separate the planning from the execution, which sounds like a much more manageable architecture for building robust systems.

Taro: Separating those parts means if the vision model gets confused about an object, you only have to worry about correcting the grasp prediction part, not re-learning how to move your whole body from scratch every time.

The paper's summary: Rosa: So what is the actual framework they propose? They use open-vocabulary large vision models like Grounding DINO and SAM to first segment out the object of interest from a picture, and then they use an AnyGrasp model to predict parallel jaw grasps for that object.

Dev: That part handles the perception and grasping strategy—figuring out what to grab and how to grab it—and then they retarget that grasp prediction onto a specific hand, like a Dex-three hand <ref:2602.16705#pg3>. They are using this visual information to decide the action plan.

Taro: Then they take that whole-body reference motion, which comes from something called cuRobo, and use their proposed whole-body end-effector tracker to guide the robot arm toward that predicted grasp before it actually picks up the object.

Rosa: That tracking policy is where the core technical innovation lies, which they call HERO. It’s designed to convey the robot arm to that specific predicted grasp using a combination of classical robotics and machine learning techniques.

Dev: The paper claims this HERO policy reduces end-effector tracking error down to two point four four centimeters, which they say is five point five times better than the strongest prior method they looked at. That’s a very concrete number for control accuracy in the real world.

Taro: That tracking policy sounds like it’s not just trying to hit a target blindly; it involves sending motion planner output, current and target joint angles, and also the current and target end-effector position as inputs to make the control more informed.

Rosa: It suggests that by doing this decomposition—separating identifying which object to pick from the actual movement of the robot arm—they inherit a lot of general visual knowledge from those pre-trained models while building very strong control capabilities for training their tracking policy in simulation.

The paper's improvements: Dev: The authors point out several things they did to make this system better, and I want to run through those improvements now. First, they used two learned neural forward models to fix problems with the analytical kinematics that often causes inaccuracies in robot modeling.

Taro: They learn a residual leg odometry model, which learns the gap between what MOCAP odometry says about the legs and what their analytical forward kinematics calculates for the legs. This correction helps improve how they estimate where their feet are.

Rosa: And they also learn a residual end-effector pose model, which learns that specific gap between what their analytical FK predicts for the end-effector and where it actually ends up, using MOCAP data as ground truth. They train both models with mean squared error losses on translations and rotations to make sure they are accurate.

Dev: On top of those learned models, they have this closed-loop replanning system that periodically replans from the current state every three hundred steps, or six seconds, using cuRobo and an odometry-corrected goal. This is meant to compensate for drift during long operations.

Taro: And they also have a specific mechanism to fix steady-state end-effector tracking offset. They maintain a commanded grasp initialized to TEE and apply a damped translational update, where the grasp position gets nudged by negative beta times the end-effector position, only when that positional error is less than fifteen centimeters.

Rosa: So these improvements show they are tackling both modeling errors in the kinematics and operational drift during long tasks by adding these specific learning and replanning layers to their core tracking policy.

Conclusion: Dev: So to wrap this up, the paper presents a modular humanoid system that successfully separates open-vocabulary action planning from simulator-trained action controllers, which they show achieves ninety percent success on daily object grasping without needing large-scale real-world imitation.

Taro: The main lesson here seems to be that the task space state has to be really accurate; they found that the analytical forward kinematics has a one point three zero centimeter end-effector error, while their learned model reaches a zero point five zero centimeter error.

Rosa: And another big thing is how explicit error feedback during closed-loop execution really cuts down on real-world drift; their tracker reaches a two point five centimeter error in those tests. That’s pretty tight control when you're out there manipulating things without constant supervision.

Dev: It proves that having accurate task-space state is important, and the explicit feedback during closed-loop execution helps reduce those drifts that you see in practice.

Taro: And it shows that even with all these learning components, the system can be language sensitive under distractors in cluttered layouts, which suggests good resilience when things get messy.

Rosa: So "HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping" gives us a way to build these systems by focusing on accurate state representation and using explicit feedback loops to keep the control stable. We’re ready to see what the next paper is about.

Runpei Dong, Ziyan Li, Arjun Gupta, Xialin He, Saurabh Gupta

University of Illinois Urbana-Champaign

cs.RO, cs.CV

Submitted: 2026-02-18

Updated: 2026-10-14

Comments: CoRL 2026

Project page: https://hero-humanoid.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: This work develops a modular system that equips humanoids with autonomous locomotion and open-vocabulary object grasping capabilities by combining large vision models for generalizable

Key concepts

HERO End-Effector Tracking Policy: HERO is the core innovation for accurate robotic arm control. It blends classical robotics with machine learning to track the end-effector precisely. This policy significantly reduces tracking errors compared to previous methods, enabling reliable physical interaction even in complex, real-world scenarios.
Open-Vocabulary Perception Pipeline: This pipeline uses large vision models like Grounding DINO and SAM to understand natural language queries about objects. These models segment the scene and identify the target object, translating a text description into a specific grasp plan. This allows the robot to interact with any object it has been trained on.
Learned Forward Models: The system uses neural networks to correct errors in traditional mathematical models of robot movement (forward kinematics). These learned models predict the actual end-effector pose, compensating for inaccuracies in the analytical calculations. This improves the accuracy of where the robot thinks its hand will land.
Robust Closed-Loop Replanning: The system maintains stability during execution by periodically replanning and adjusting goals based on real-time sensor data and odometry corrections. This loop compensates for accumulated errors (drift) between simulation and reality, ensuring the robot stays on track even when facing challenging environments.

Terminology

Core — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes

Summary

The gist: This work develops a modular system that equips humanoids with autonomous locomotion and open-vocabulary object grasping capabilities by combining large vision models for generalizable scene understanding with an accurate end-effector tracking policy.

System Overview

The proposed framework is a modular system that combines large vision models for visual generalization with simulated training for accurate end-effector control, leading to an overall modular loco-manipulation system that exhibits strong generalization. Given a free-form natural language text query indicating which object needs to be picked, we use open-vocabulary large vision models (LVMs: Grounding DINO [16] and SAM [17]) to segment out the object of interest and predict parallel jaw grasps (using the AnyGrasp model [18]). By decomposing action planning (i.e. identifying which object to pick and using what grasp) from action execution (i.e. actual control of the robot), we inherit the strong visual generalization from pre-trained models as well as strong control capabilities for simulated training of the tracking policy.

HERO End-Effector Tracking Policy

The core technical innovation is HERO, an accurate residual-aware EE tracking policy made possible by combining classical robotics with machine learning. This policy achieves a reduction in the end-effector tracking error to 2.44cm, outperforming the strongest prior method by 5.5×. The accurate end-effector control algorithm is based on multiple innovations. First, rather than directly trying to get the end-effector to the target location, we use a motion planner to generate an upper-body reference motion that gets the end-effector to the desired target. Second, the policy receives as input not just the current and target joint angles (output of the motion planner), but also the current and target endeffector position. Third, it is important to obtain a high-quality estimate of the current end-effector position.

Learned Forward Models

The system utilizes two learned neural forward models to address inaccuracies in analytical kinematics. Specifically, given the current proprioceptive state of one operating arm and waist xt ∈ R10 at timestep t, and output from analytical FK, FK(xt), the final end-effector pose fEE(xt) is obtained as: fEE(xt) = FK(xt) ⊕ η xt, FK(xt) The residual neural leg odometry model ξ learns the residual between MOCAP odometry and analytical leg-FK odometry. We train both models on MOCAP data [67] with MSE losses on residual translations and rotations, using MOCAP-derived ∆T EE t and ∆On−m as ground truth. We periodically replan from the current state and odometry-corrected goal in a closed loop. To further compensate the steady-state EE tracking offset, we maintain a commanded grasp g initialized to TEE and apply the damped translational update g p ← g p − β, ep t, where g p, e p t ∈ R3 denote the translational components of g and the SE(3) residual et = fEE(xt) ⊖ TEE, β = 0.6, only when e p t ≤ 15cm, clipped to 5mm/step, and stopped once e p t < 1.5cm. In cluttered layouts (Fig. 6(b)), HERO achieves 80% success without teleoperation demonstrations, remaining language-sensitive under distractors. In simulation, HERO attains the lowest errors; SONIC lacks in EE tracking feedback and joint position only baselines fall behind.

Conclusion

We present a modular humanoid system that separates open-vocabulary action planning from simulator-trained action controller, achieving 90% success on daily-object grasping without largescale real-world imitation. The experiments show two lessons: task-space state must be accurate (analytical FK has 1.30cm EE error while ours reaches 0.50cm), and explicit error feedback during closed-loop execution effectively reduces real-world drifts (our tracker reaches 2.5cm error)<ref:6pg6,The experiments show two lessons: task-space state must be accurate (analytical FK has 1.30cm EE error while ours reaches 0.50cm), and explicit error feedback during closed-loop execution effectively reduces real-world drifts (our tracker reaches 2.5cm error).

Improvements for AI systems

Bold header: Accurate End-Effector Control via HERO policy

The improved system can achieve an accurate residual-aware EE tracking policy by combining classical robotics with machine learning, which reduces end-effector tracking error to 2.44cm, outperforming the strongest prior method by 5.5×.

Bold header: Modular Open-Vocabulary Perception Pipeline

The system can perform open-vocabulary grasping using a pipeline where Large vision models for generalizable scene understanding translate high-level instructions into actionable plans, allowing it to handle novel objects and scenes via Grounding DINO [16] and SAM [17] for segmentation.

Bold header: Learned Forward Kinematics Correction

The AI can mitigate inaccuracies in robot modeling by using learned residual FK where the model learns a correction to the analytical forward kinematic function, specifically predicting residuals for both end-effector pose and base odometry.

Bold header: Robust Closed-Loop Replanning and Goal Adjustment

The system gains robustness during execution by implementing replanning every k = 300 steps (6 sec) with cuRobo [63], using the current state and the odometry-corrected goal, which compensates for drift, and a goal adjustment loop iteratively nudges the commanded pose to remove the systematic sim2real gap.

Bold header: Enhanced Whole-Body Workspace Utilization

By enabling torso motion, the system can achieve significantly larger reachability; specifically, Bending and twisting the torso effectively repositions the shoulder frame, growing workspace from 0.248 m3 to 0.523 m3 (∼ 2.1×).