DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy".
Dev: Here's the gist of the paper. Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics.
Rosa: To address these limitations, we propose a tactile-driven adaptation strategy designed to enable fine-grained manipulation on top of teleoperation pipelines.
Dev: This strategy introduces a real-time optimization loop that bridges the embodiment gap by translating coarse human tracking intents into precise, force-compliant robotic commands with tactile sensing.
Rosa: By estimating accurate contact points and leveraging a tactile-enabled fingertip force-sensing profile, the system dynamically computes localized corrections using the operational space Jacobian with respect to joint angle updates.
Paper summary: Rosa: So we've covered how DexTeleop-zero uses a real time optimization loop and tactile sensing to translate human intent into stable force compliant commands for bimanual tasks, bridging the embodiment gap in teleoperation <ref:2606.23431#pg1,a real time optimization loop>.
Dev: The authors of DexTeleop-zero are trying to solve that fundamental problem where traditional systems lack the necessary feedback during contact, and they use this approach to get better results in robust grasping and disturbance resilient manipulation compared to other methods <ref:2606.23431#pg1>.
Taro: What it really means for the world is that if you want robots to do intricate assembly or delicate manipulation tasks on your behalf based on human demonstrations, you need a system that can handle those physical forces correctly without needing perfect pre-programming of every single interaction.
Rosa: It shifts the focus from just getting the hand position right in space to getting the *feel* and the *force* right during contact, which is a massive step for building truly dexterous robot systems that interact safely with things.
Conclusion: Rosa: So we're wrapping up DexTeleop-zero which is this whole system that uses tactile feedback to make bimanual robot manipulation feel much more like you're actually doing it yourself while you teleoperate it.
Dev: Yeah, so the paper itself calls it DexTeleop-zero and the authors are pushing a lot about how they closed that control gap using this force-balanced residual action optimization loop.
Rosa: It’s really about taking that raw human tracking intent from your VR headset and translating it into precise physical movements on a robot by constantly checking the forces at the fingertips.
Dev: And what I see is this shared autonomy layer being built right into the control cycle, using a quadratic programming framework to balance local finger force regulation with keeping the whole object stable.
Rosa: The main implication here for me is that we’re moving past just mapping hand positions in space to actually understanding and reacting to the physical feel of touching things during complex tasks like assembly or grasping delicate items.
Dev: That means when the world throws a little unexpected resistance at your robot, this system isn't just going to freeze or overshoot; it’s supposed to adjust its commands based on those contact forces in real time.
Rosa: It’s about making the robot more robust and less fragile when it's actually interacting physically with something, which is huge for real-world applications outside of a perfect simulation.
Dev: So, if we look at the results from their testing, they showed better success rates on tasks like gear meshes and peg insertion compared to older control methods.
Rosa: And they kept the contact force profile really low across all those tests, which is important because it means the robot isn't accidentally crushing what it’s supposed to be working on.
Dev: It sounds like they’ve managed a pretty tight loop running at thirty hertz, and that level of real-time force adjustment is what makes the difference in their performance metrics.
Rosa: It really shows how crucial that ego-centric perception pipeline—using those vision inputs to guide the mapping—is for getting this tactile feedback working reliably.
Dev: So while they’ve shown great success in simulation and on physical gear tasks, the question remains how long this control loop stays stable when you introduce a lot of unexpected noise or sudden external disturbances in a messy real-world environment.
Haichao Liu, Yuyao Jiang, Hyunsun Park, Yuanjiang Xue, Ziwei Wang
Nanyang Technological University · OOJU, USA
cs.RO
Submitted: 2026-06-22
Updated: 2026-11-05
Comments: 16 pages, 9 figures, 6 tables
Project page: https://henryhcliu.github.io/dexteleop-0
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 99/100
The gist: Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics.
Key concepts
- Force-Balanced Residual Action Optimization Loop
- This is a core strategy that estimates exact contact points and measures continuous fingertip forces. It uses these measurements to dynamically calculate necessary force variations using the operational space Jacobian relative to joint angle updates. This loop ensures the robot maintains stable force regulation while adjusting its movements.
- Egocentric Vision Pipeline
- The system uses a commercial VR headset for egocentric vision, meaning it captures a view from the perspective of the human operator. These visual inputs are then mapped onto the robot's 56-DoF bimanual assembly using an inverse kinematics model to translate human hand movements into robotic joint angles.
- Project-and-Escape Tracking Logic
- This is a tracking method applied to specific tip-to-tip vectors. When a vector is marked as 'Projected,' it signals the operator's intention for a close grasp or pinch. This logic increases the weight of the optimization, forcing the system to prioritize maintaining that intended contact during manipulation.
Terminology
Summary
Fine-grained, bimanual dexterous manipulation remains a foundational challenge in robotics. p. 0 To address these limitations, we propose a tactile-driven adaptation strategy designed to enable fine-grained manipulation on top of teleoperation pipelines. p. 0 This strategy introduces a real-time optimization loop that bridges the embodiment gap by translating coarse human tracking intents into precise, force-compliant robotic commands with tactile sensing. p. 0 By estimating accurate contact points and leveraging a tactile-enabled fingertip force-sensing profile, the system dynamically computes localized corrections using the operational space Jacobian with respect to joint angle updates. p. 0 Compared with representative baselines, the proposed method consistently achieves higher task success rates and improved execution efficiency in robust grasping, disturbance-resilient manipulation, and complex dexterous tasks. p. 0
Pruning redundant reasoning tokens
The core problem addressed is the lack of tactile and force feedback in traditional teleoperation systems which hinders accurate kinematic mapping and data collection efficiency. The proposed DexTeleop-0 framework introduces a force-balanced residual action optimization loop
to overcome this gap. This loop estimates precise contact points and measures continuous fingertip forces to dynamically compute force variations using the operational space Jacobian relative to joint angle updates. p. 1 The final command is formulated as qfinal = qtele + Δq, where Δq is derived from a unified Quadratic Programming (QP) optimization framework that balances localized fingertip force regulation with global object-centric force-torque stability. p. 5
How the training signal is built
The system utilizes an accessible egocentric vision pipeline via a commercial VR headset to capture human tracking intents, which are then mapped onto a high-dimensional 56-Degree-of-Freedom (DoF) bimanual robot assembly through an inverse kinematics (IK) retargeting model. The mapping Fhand: S E(3) M → R N maps the M human hand joint vectors to the N robotic joint angles while preserving structural synergy and finger-tip positioning qh = Fhand (H). p. 3 To ensure stable finger-to-finger contact closures, a project-andescape
tracking logic is applied to a designated subset of Kp tip-to-tip vectors. This logic augments the optimization weight wk to a high value (wk = whigh) when a vector is marked as Projected, signifying active intention of a close-range pinch or grasp. p. 4
Where the gains come from
The primary contribution is the tactile-driven shared autonomy adaptation strategy that explicitly bridges the control and interaction gap during physical contact. The framework utilizes a contact-rich force sensing and predictive interaction modeling framework to systematically close the visual-tactile feedback loop during delicate physical engagements. p. 1 This is achieved through a unified box-constrained QP solved at every control cycle at 30 Hz, which minimizes the cost function min Δq 1/2 Δq⊤H0Δq + g⊤0 Δq + lF (Δq) + lB(Δq) s. p. 6t. qmin − q ≤ Δq ≤ qmax − q. p. 6
Experimental Validation
The full-stack solution is comprehensively validated across an aligned simulation environment and physical robotic platform, demonstrating substantial improvements in both data-collection efficiency and task success rates compared to representative baselines. Simulation evaluations show that DexTeleop-0 exhibits a significant performance advantage in multi-stage execution rates and force regulation over the No Residual and PD control methods. p. 8 Real-world experiments confirm the capability of the shared autonomy layer to mitigate tracking errors, resulting in superior assembly success rates for tasks like Gear Mesh and Peg Insertion compared to baselines. p. 9 The framework archives the lowest overall contact profile across tested tasks, preventing damage to delicate surfaces. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Tactile-driven adaptation strategy for bimanual dexterous manipulation: This strategy enables
fine-grained manipulation on top of teleoperation pipelines
by introducing areal-time optimization loop that bridges the embodiment gap by translating coarse human tracking intents into precise, force-compliant robotic commands with tactile sensing.
-
Contact-rich force sensing and predictive interaction modeling framework: The system develops this framework to
systematically close the visual-tactile feedback loop during delicate physical engagements
by using aforce-balanced residual action optimization loop
that estimates contact points and measures continuous fingertip forces. -
Hardware-agnostic teleoperation framework: This design utilizes
egocentric vision perception and IK-based dexterous hand retargeting to translate human tracking intent into coordinated robotic commands,
allowing it to be deployed on various hardware without requiring restrictive mechanical rigs. -
Joint space optimization for force compliance: The system solves a unified box-constrained Quadratic Programming problem at 30 Hz, which
balances localized fingertip force regulation with global object-centric force-torque stability
to ensure thatthe execution commands are projected onto a force-compliant, physically stable manifold.
-
Enhanced task success rates and efficiency: Compared to baselines, the proposed method consistently achieves higher task success rates and improved execution efficiency in
robust grasping, disturbance-resilient manipulation, and complex dexterous tasks,
as demonstrated by achieving a97% success rate
on specific challenging manipulation routines.
Sources
- GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
- ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation
- Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning
- AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiency
- DexterityGen: Foundation Controller for Unprecedented Dexterity
- TacDiffusion: Force-domain Diffusion Policy for Precise Tactile Manipulation
- Improving Low-Cost Teleoperation: Augmenting GELLO with Force
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation