AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection

arXiv:2606.23689 · cs.RO, cs.LG · Submitted 2026-06-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection".

Dev: Here's the gist of the paper. Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts <ref:2606.23689#pg2> Such data are difficult to obtain at scale <ref:2606.23689#pg2> A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop—perception, execution, labeling, and reset—runs without human intervention <ref:2606.23689#pg2> We present AutoDex, an automated real-world data-collection system that performs this entire validation cycle without human intervention <ref:2606.23689#pg2> Given generated candidate grasps, it estimates the object’s initial 6D pose with dense 20-camera perception <ref:2606.23689#pg2>, executes selected candidates with collision-monitored robot motions <ref:2606.23689#pg2>, tracks the object during execution despite severe hand–object occlusion <ref:2606.23689#pg2>, labels lift-and-hold success or failure <ref:2606.23689#pg2>, and actively resets the object between trials to continue testing candidates across stable poses <ref:2606.23689#pg2> The result is a reusable database of physically labeled grasp trials that downstream systems can retrieve and filter for feasibility <ref:2606.23689#pg2> Pruning redundant reasoning tokens The paper introduces AutoDex, an end-to-end automated system designed to turn generated grasp candidates into physically labeled robot trials <ref:2606.23689#pg2> This system automates the entire validation cycle, which includes obtaining candidate grasps from a modular generator, executing them on physical hardware, and recording success or failure labels without human intervention <ref:2606.23689#pg2> The system is realized through four components: dense 20-camera perception for initial object pose estimation and execution-time tracking under hand–object occlusion <ref:2606.23689#pg2>, collision-monitored motion execution for safe unattended grasp attempts <ref:2606.23689#pg2>, physical success/failure labeling from lift-and-hold trials <ref:2606.23689#pg2>, and active object reset for continued testing across stable poses <ref:2606.23689#pg2> How the training signal is built The process begins with Grasp Candidate Generation, where AutoDex obtains candidate grasps from a modular candidate generator which takes an object model and a scene specification as input to return a set of kinematically and geometrically feasible candidates <ref:2606.23689#pg2> The generation uses BODex

8: to propose candidates based on scene primitives like wall, shelf, and box constraints <ref:2606.23689#pg2> Before real-robot execution, a simulation sanity pre-filter is applied where candidates are screened in MuJoCo

37: by applying a weak external force to reject those that translate more than 5 cm under perturbation <ref:2606.23689#pg2> The system selects an unattempted feasible candidate for the current stable pose and plans a collision-free arm trajectory to approach the candidate wrist pose <ref:2606.23689#pg2> Where the gains come from The primary throughput gain is demonstrated by comparing AutoDex with manual teleoperation, which requires 49.4 h for a matched 500-trajectory collection study compared to AutoDex's 10.3 h <ref:2606.23689#pg2> This yields a 4.8× throughput improvement <ref:2606.23689#pg2> Furthermore, grasps retrieved from the AutoDex-validated database achieve 76% real-world success, significantly higher than the 34% success rate for grasps selected without real-world validation <ref:2606.23689#pg2> This improvement is consistent across material, scene, and weight categories <ref:2606.23689#pg2> Impact of physical validation on dataset quality Physical validation substantially improves database quality by filtering out grasps that satisfy geometric constraints but fail under real-world contact effects such as slip, compliance, and multi-finger instability <ref:2606.23689#pg2> The AutoDex validated database achieves 76% grasp success when retrieved in deployment settings, compared to 34% for model-screened databases <ref:2606.23689#pg2> This indicates that kinematic and collision feasibility alone are insufficient for constructing a reliable dexterous-grasp database <ref:2606.23689#pg2> Reset Strategy Evaluation The system employs an explicit reset module to handle pose transitions between trials, which is critical because inter-trial reset cannot rely on passive settling alone <ref:2606.23689#pg2> The Stable Reorient Placement strategy maintains high success across the full difficulty range, including transitions with a passive transition probability of approximately 0 <ref:2606.23689#pg2> This method avoids the failure modes incompatible with unattended operation that occur in other reset strategies <ref:2606.23689#pg2> Impact of Camera Density on Pose Reliability Dense multi-view perception improves pose reliability for unattended collection because it provides redundant observations for robust initial pose estimation and helps recover object motion when the hand occludes much of the object <ref:2606.23689#pg2> The mean ADD-S between the full 20-camera reference pose and a k-camera subset poses decreases as camera count increases, showing that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606.23689#pg2> This result indicates that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606.23689#pg2> Discussion and Limitations AutoDex currently targets stable grasps for a fixed arm-hand setup, and the system also inherits the blind spots of its upstream generator which may miss grasps requiring dynamic finger motion during contact <ref:2606.23689#pg2> While dense multi-view perception improves pose reliability, the highest throughput setting still relies on a multi-camera workcell <ref:2606.23689#pg2> Bimanual coordination, mobile manipulation, finger-rolling regrasps, and functional grasps remain out of scope for the current system <ref:2606.23689#pg2> References

1: Y.

Rosa: Liu, Y.

Dev: Yang, Y.

Rosa: Wang, X.

Paper summary: Rosa: So, to recap, AutoDex tackles the problem of needing real physical outcome data for grasping by creating an automated system that does the whole validation process without a human in the loop.

Dev: The thesis is that you can generate many grasp candidates and test them on hardware to get reliable labels, provided you automate the entire collection loop—perception, execution, labeling, and reset.

Rosa: They show how this works by taking generated candidates from a modular generator and running them through a whole pipeline.

Dev: This pipeline involves using dense twenty camera perception for the initial six dimensional pose estimation, executing those selected candidates with collision-monitored robot motions, and then labeling the lift-and-hold success or failure.

Taro: I’m interested in how they manage the part where a candidate might be geometrically fine but still fail because of real physics like slip or compliance.

Rosa: The paper points out that kinematic and collision feasibility checks only confirm if a grasp can be executed, but they don't certify whether the contact will stably support the object under real contact effects.

Dev: That’s where the physical validation becomes crucial; it filters out grasps that look good on paper but fail because of things like friction or multi-finger instability.

Rosa: They demonstrate this by comparing their validated database to a baseline set of grasps selected without real-world validation, showing a success rate jump from thirty-four percent up to seventy-six percent when retrieved in deployment settings.

Dev: That improvement is consistent across different materials, scene types, and object weights, which suggests the physical validation really improves the quality of the resulting dataset.

Conclusion: Rosa: So what does this whole AutoDex system actually mean for us beyond just having a better dataset?

Dev: It means we can scale up data collection far past what manual teleoperation allows, which is only about ten point three hours for five hundred trials.

Rosa: That throughput gain is significant; they compared AutoDex’s ten point three hours to forty-nine point four hours for the same study with human teleoperation.

Dev: But more importantly, it means that the data you get from this system is much more reliable than what you get from just running candidates through simulation and planning tools.

Taro: The authors acknowledge that while dense multi-view perception helps with pose reliability, there are still limitations, especially when the object is textureless or thin.

Rosa: They admit that kinematic and collision feasibility alone aren't enough to build a reliable database; you need that real-world contact physics validation.

Dev: It also notes that the system currently targets stable grasps for a fixed arm hand setup, and it doesn't cover more complex things like bimanual coordination or finger-rolling regrasps right now.

Rosa: The implication is that this system is a powerful tool for building a foundation of physically verified data, even if it stops short of covering every kind of advanced manipulation we could ever imagine.

Mingi Choi, Gunhee Kim, Jisoo Kim, Taeksoo Kim, Taeyun Ha, Jongbin Lim, Hanbyul Joo

Seoul National University

cs.RO, cs.LG

Submitted: 2026-06-22

Updated: 2026-11-04

Comments: 9 pages + references, 6 figures. Accepted to CoRL 2026

Project page: https://willi19.github.io/AutoDex

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts <ref:2606.23689#pg2> Such data are difficult to obtain at scale <ref:2606.23689#pg2> A…

Terminology

Summary

Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts <ref:2606. p. 023689#pg2> Such data are difficult to obtain at scale <ref:2606. p. 023689#pg2> A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop—perception, execution, labeling, and reset—runs without human intervention <ref:2606. p. 023689#pg2> We present AutoDex, an automated real-world data-collection system that performs this entire validation cycle without human intervention <ref:2606. p. 023689#pg2> Given generated candidate grasps, it estimates the object’s initial 6D pose with dense 20-camera perception <ref:2606. p. 023689#pg2>, executes selected candidates with collision-monitored robot motions <ref:2606. p. 023689#pg2>, tracks the object during execution despite severe hand–object occlusion <ref:2606. p. 023689#pg2>, labels lift-and-hold success or failure <ref:2606. p. 023689#pg2>, and actively resets the object between trials to continue testing candidates across stable poses <ref:2606. p. 023689#pg2> The result is a reusable database of physically labeled grasp trials that downstream systems can retrieve and filter for feasibility <ref:2606. p. 023689#pg2>

Pruning redundant reasoning tokens

The paper introduces AutoDex, an end-to-end automated system designed to turn generated grasp candidates into physically labeled robot trials <ref:2606.23689#pg2> This system automates the entire validation cycle, which includes obtaining candidate grasps from a modular generator, executing them on physical hardware, and recording success or failure labels without human intervention <ref:2606. p. 223689#pg2> The system is realized through four components: dense 20-camera perception for initial object pose estimation and execution-time tracking under hand–object occlusion <ref:2606. p. 123689#pg2>, collision-monitored motion execution for safe unattended grasp attempts <ref:2606. p. 123689#pg2>, physical success/failure labeling from lift-and-hold trials <ref:2606. p. 123689#pg2>, and active object reset for continued testing across stable poses <ref:2606. p. 123689#pg2>

How the training signal is built

The process begins with Grasp Candidate Generation, where AutoDex obtains candidate grasps from a modular candidate generator which takes an object model and a scene specification as input to return a set of kinematically and geometrically feasible candidates <ref:2606.23689#pg2> The generation uses BODex [8] to propose candidates based on scene primitives like wall, shelf, and box constraints <ref:2606. p. 323689#pg2> Before real-robot execution, a simulation sanity pre-filter is applied where candidates are screened in MuJoCo [37] by applying a weak external force to reject those that translate more than 5 cm under perturbation <ref:2606. p. 623689#pg2> The system selects an unattempted feasible candidate for the current stable pose and plans a collision-free arm trajectory to approach the candidate wrist pose <ref:2606. p. 423689#pg2>

Where the gains come from

The primary throughput gain is demonstrated by comparing AutoDex with manual teleoperation, which requires 49.4 h for a matched 500-trajectory collection study compared to AutoDex's 10.3 h <ref:2606. p. 023689#pg2> This yields a 4.8× throughput improvement <ref:2606. p. 023689#pg2> Furthermore, grasps retrieved from the AutoDex-validated database achieve 76% real-world success, significantly higher than the 34% success rate for grasps selected without real-world validation <ref:2606. p. 023689#pg2> This improvement is consistent across material, scene, and weight categories <ref:2606. p. 623689#pg2>

Impact of physical validation on dataset quality

Physical validation substantially improves database quality by filtering out grasps that satisfy geometric constraints but fail under real-world contact effects such as slip, compliance, and multi-finger instability <ref:2606.23689#pg2> The AutoDex validated database achieves 76% grasp success when retrieved in deployment settings, compared to 34% for model-screened databases <ref:2606. p. 623689#pg2> This indicates that kinematic and collision feasibility alone are insufficient for constructing a reliable dexterous-grasp database <ref:2606. p. 723689#pg2>

Reset Strategy Evaluation

The system employs an explicit reset module to handle pose transitions between trials, which is critical because inter-trial reset cannot rely on passive settling alone <ref:2606.23689#pg2> The Stable Reorient Placement strategy maintains high success across the full difficulty range, including transitions with a passive transition probability of approximately 0 <ref:2606. p. 723689#pg2> This method avoids the failure modes incompatible with unattended operation that occur in other reset strategies <ref:2606. p. 723689#pg2>

Impact of Camera Density on Pose Reliability

Dense multi-view perception improves pose reliability for unattended collection because it provides redundant observations for robust initial pose estimation and helps recover object motion when the hand occludes much of the object <ref:2606.23689#pg2> The mean ADD-S between the full 20-camera reference pose and a k-camera subset poses decreases as camera count increases, showing that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606. p. 723689#pg2> This result indicates that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606. p. 823689#pg2>

Discussion and Limitations

AutoDex currently targets stable grasps for a fixed arm-hand setup, and the system also inherits the blind spots of its upstream generator which may miss grasps requiring dynamic finger motion during contact <ref:2606.23689#pg2> While dense multi-view perception improves pose reliability, the highest throughput setting still relies on a multi-camera workcell <ref:2606. p. 823689#pg2> Bimanual coordination, mobile manipulation, finger-rolling regrasps, and functional grasps remain out of scope for the current system <ref:2606. p. 823689#pg2>

References

[1] Y. Liu, Y. Yang, Y. Wang, X. Wu, J. Wang, Y. Yao, S. Schwertfeger, S. p. 8 Yang, W. Wang, J Yu et al Realdex: Towards human-like grasping for robotic dexterous hand arXiv preprint arXiv:240213853 2024

[2] C Wang, H Shi, W Wang, R Zhang, L Fei-Fei and C K Liu Dexcap: Scalable and portable mocap data collection system for dexterous manipulation arXiv preprint arXiv:240307788 2024

[3] Z Chen, K Van Wyk Y-W Chao W Yang A Mousavian A Gupta and D Fox Dextransfer: Real world multi-fingered dexterous grasping with minimal human demonstrations arXiv preprint arXiv:220914284 2022

[4] J Zhang, H Liu D Li X Yu H Geng Y Ding J Chen and H Wang DexGraspNet 2.0 Learning generative dexterous grasping in large-scale synthetic cluttered scenes In Conference on Robot Learning CoRL 2024

[5] A Bicchi and V Kumar Robotic grasping and contact A review In Proceedings of the IEEE International Conference on Robotics and Automation ICRA volume 1 pages 348–353 2000

[6] Y Qin, W Yang B Huang K Van Wyk H Su X Wang Y-W Chao and D Fox Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system In Robotics: Science and Systems 2023

[7] R Wang J Zhang J Chen Y Xu P Li T Liu and H Wang Dexgraspnet: A largescale robotic dexterous grasp dataset for general objects based on simulation arXiv preprint arXiv:221002697 2022

[8] J Chen Y Ke and H Wang Bodex: Scalable and efficient robotic dexterous grasp synthesis using bilevel optimization arXiv preprint arXiv:241216490 2024

[9] D Turpin T Zhong S Zhang G Zhu J Liu R Singh E Heiden M Macklin S Tsogkas S Dickinson et al Fast-grasp’d: Dexterous multi-finger grasp generation through differentiable simulation arXiv preprint arXiv:230608132 2023

[10] D Huang T Zhang Y Li L Zhao J Li Z Fang C Xia and X He Dexterous grasping with real-world robotic reinforcement learning arXiv preprint arXiv:250304014 2025

[11] Y Park J S Bhatia L Ankile and P Agrawal Dart: Dexterous augmented reality teleoperation platform for large-scale robot data collection in simulation In 2025 IEEE International Conference on Robotics and Automation ICRA pages 13883–13889 IEEE 2025

[12] T Liu Z Liu Z Jiao Y Zhu and S-C Zhu Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator IEEE Robotics and Automation Letters 7(1):470–477 Jan. 2022 ISSN 2377-3774 doi:10. p. 91109/lra.2021.3129138 URL http://dx. p. 9doi.org/10.1109/LRA.2021.3129138

[35] H Lin S Chen J Liew D Y Chen Z Li G Shi J Feng and B Kang Depth anything 3: Recovering the visual space from any views arXiv:251110647 2025

[40] B Sundaralingam S K Hari A Fishman C Garrett K Van Wyk V Blukis A Millane H Oleynikova A Handa F Ramos et al Curobo: Parallelized collision-free robot motion generation In 2023 IEEE International Conference on Robotics and Automation ICRA pages 8112–8119 IEEE 2023

[47] S Hinterstoisser V Lepetit S Ilic S Holzer G Bradski K Konolige and N Navab Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes In Asian conference on computer vision 2012

[48] E Todorov T Erez and Y Tassa Mujoco: A physics engine for model-based control In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems pages 5026–5033 IEEE 2012 doi:10.1109/IROS.2012.6386109

[49] V N Nguyen C Forster B Tekin S Shkodrani V Lepetit C Keskin and T Hodan Gotrack: Generic 6dof object pose refinement and tracking Computer Vision and Pattern Recognition Workshops CVPRW 2025

[50] B Sundaralingam S K Hari A Fishman C Garrett K Van Wyk V Blukis A Millane H Oleynikova A Handa F Ramos et al Curobo: Parallelized collision-free robot motion generation In 2023 IEEE International Conference on Robotics and Automation ICRA pages 8112–8119 IEEE 2023

[51] S Mirchandani S Belkhale J Hejna E Choi M S Islam and D Sadigh So you think you can scale up autonomous robot data collection? In Conference on Robot Learning CoRL 2024

[53] N Carion L Gustafson Y-T Hu S Debnath R Hu D Suris C Ryali K V Alwala H Khedr A Huang et al Sam 3: Segment anything with concepts arXiv:251116719 2025

--- Page 6 ---

Figure 4 Left: Throughput comparison AutoDex collects 500 trials in 10.3 h compared with teleoperation requires 49. p. 04 h for the same workcell

--- Page 7 ---

Figure 5 Left: Reset strategy comparison Naive Drop follows y = x by construction its success rate equals the passive transition probability

--- Page 8 ---

Figure 5 Right: Pose self-consistency relative to the 20-camera reference as a function of camera count

--- Page 9 ---

Figure 9 Left: AutoDex object library and diversity

The paper presents AutoDex, an automated system for collecting physically labeled dexterous-grasp trials through real-world execution, labeling, and reset <ref:2606.23689#pg2> The results show that real-world validation improves collection throughput over teleoperation and produces a grasp database that transfers more reliably than candidates screened without physical execution <ref:2606. p. 823689#pg2>

The system operates by estimating the object’s 6D pose from multi-view observations, identifying the current stable pose Pi ∈ P, and transforming each object-relative candidate in Gcand(Pi) into the robot frame <ref:2606.23689#pg2> AutoDex then selects an unattempted feasible candidate and plans a collision-free arm trajectory to approach the candidate wrist pose <ref:2606. p. 423689#pg2> The hand then moves through the candidate pre-grasp and grasp configurations, after which the robot lifts the object and holds for a fixed duration <ref:2606. p. 423689#pg2>

The throughput gain stems from eliminating human idle time and enabling sustained unattended collection across extended runs <ref:2606.23689#pg2> The database quality is improved because physical validation filters out grasps that satisfy geometric constraints but fail under real contact effects such as slip, compliance, and multi-finger instability <ref:2606. p. 723689#pg2> The reset module's success in maintaining high success across the full difficulty range demonstrates that explicit reset placement is critical for reliable unattended dexterous-grasp collection <ref:2606.23689#pg2>

Dense multi-view perception improves pose reliability because it provides redundant observations for robust initial pose estimation and helps recover object motion when the hand occludes much of the object <ref:2606.23689#pg2> The result is that kinematic and collision feasibility alone are insufficient for constructing a reliable dexterous-grasp database <ref:2606. p. 723689#pg2> The high error at k = 2 is driven by occasional catastrophic pose failures on textureless or thin objects, which are largely eliminated by k = 8 <ref:2606. p. 8

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated real-world data collection system AutoDex can perform end-to-end automated real-world data collection at scale by combining candidate generation, execution, labeling, and active object reset without human intervention.

  2. The improved AI system can create a reusable database of physically labeled grasp trials that downstream systems can retrieve and filter for feasibility in new scenes, achieving 76% real-world success for retrieved grasps compared to 34% for model-screened ones.

  3. The system gains a throughput improvement over teleoperation by eliminating human idle time, enabling sustained unattended accumulation across extended runs.

  4. The improved AI can filter out geometrically feasible but physically unstable candidates by using the simulation sanity pre-filter in MuJoCo, which rejects candidates that fail a minimal plausibility check under weak perturbation.

  5. The system utilizes dense 20-camera perception to enable robust initial pose estimation and tracking despite severe hand–object occlusion, improving object-pose reliability during collection.

  6. The active reset module allows the system to move the object between stable poses, ensuring unattended collection across object poses by generating a reset grasp that is collision-free in a combined pickup-placement scene.

  7. Downstream systems can perform in-the-wild execution without policy training or additional real-world trial collection by retrieving successful grasps and checking their feasibility against the current workspace obstacles.

Sources

Related papers