AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection".
Dev: Here's the gist of the paper. Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts <ref:2606.23689#pg2> Such data are difficult to obtain at scale <ref:2606.23689#pg2> A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop—perception, execution, labeling, and reset—runs without human intervention <ref:2606.23689#pg2> We present AutoDex, an automated real-world data-collection system that performs this entire validation cycle without human intervention <ref:2606.23689#pg2> Given generated candidate grasps, it estimates the object’s initial 6D pose with dense 20-camera perception <ref:2606.23689#pg2>, executes selected candidates with collision-monitored robot motions <ref:2606.23689#pg2>, tracks the object during execution despite severe hand–object occlusion <ref:2606.23689#pg2>, labels lift-and-hold success or failure <ref:2606.23689#pg2>, and actively resets the object between trials to continue testing candidates across stable poses <ref:2606.23689#pg2> The result is a reusable database of physically labeled grasp trials that downstream systems can retrieve and filter for feasibility <ref:2606.23689#pg2> Pruning redundant reasoning tokens The paper introduces AutoDex, an end-to-end automated system designed to turn generated grasp candidates into physically labeled robot trials <ref:2606.23689#pg2> This system automates the entire validation cycle, which includes obtaining candidate grasps from a modular generator, executing them on physical hardware, and recording success or failure labels without human intervention <ref:2606.23689#pg2> The system is realized through four components: dense 20-camera perception for initial object pose estimation and execution-time tracking under hand–object occlusion <ref:2606.23689#pg2>, collision-monitored motion execution for safe unattended grasp attempts <ref:2606.23689#pg2>, physical success/failure labeling from lift-and-hold trials <ref:2606.23689#pg2>, and active object reset for continued testing across stable poses <ref:2606.23689#pg2> How the training signal is built The process begins with Grasp Candidate Generation, where AutoDex obtains candidate grasps from a modular candidate generator which takes an object model and a scene specification as input to return a set of kinematically and geometrically feasible candidates <ref:2606.23689#pg2> The generation uses BODex
8: to propose candidates based on scene primitives like wall, shelf, and box constraints <ref:2606.23689#pg2> Before real-robot execution, a simulation sanity pre-filter is applied where candidates are screened in MuJoCo
37: by applying a weak external force to reject those that translate more than 5 cm under perturbation <ref:2606.23689#pg2> The system selects an unattempted feasible candidate for the current stable pose and plans a collision-free arm trajectory to approach the candidate wrist pose <ref:2606.23689#pg2> Where the gains come from The primary throughput gain is demonstrated by comparing AutoDex with manual teleoperation, which requires 49.4 h for a matched 500-trajectory collection study compared to AutoDex's 10.3 h <ref:2606.23689#pg2> This yields a 4.8× throughput improvement <ref:2606.23689#pg2> Furthermore, grasps retrieved from the AutoDex-validated database achieve 76% real-world success, significantly higher than the 34% success rate for grasps selected without real-world validation <ref:2606.23689#pg2> This improvement is consistent across material, scene, and weight categories <ref:2606.23689#pg2> Impact of physical validation on dataset quality Physical validation substantially improves database quality by filtering out grasps that satisfy geometric constraints but fail under real-world contact effects such as slip, compliance, and multi-finger instability <ref:2606.23689#pg2> The AutoDex validated database achieves 76% grasp success when retrieved in deployment settings, compared to 34% for model-screened databases <ref:2606.23689#pg2> This indicates that kinematic and collision feasibility alone are insufficient for constructing a reliable dexterous-grasp database <ref:2606.23689#pg2> Reset Strategy Evaluation The system employs an explicit reset module to handle pose transitions between trials, which is critical because inter-trial reset cannot rely on passive settling alone <ref:2606.23689#pg2> The Stable Reorient Placement strategy maintains high success across the full difficulty range, including transitions with a passive transition probability of approximately 0 <ref:2606.23689#pg2> This method avoids the failure modes incompatible with unattended operation that occur in other reset strategies <ref:2606.23689#pg2> Impact of Camera Density on Pose Reliability Dense multi-view perception improves pose reliability for unattended collection because it provides redundant observations for robust initial pose estimation and helps recover object motion when the hand occludes much of the object <ref:2606.23689#pg2> The mean ADD-S between the full 20-camera reference pose and a k-camera subset poses decreases as camera count increases, showing that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606.23689#pg2> This result indicates that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606.23689#pg2> Discussion and Limitations AutoDex currently targets stable grasps for a fixed arm-hand setup, and the system also inherits the blind spots of its upstream generator which may miss grasps requiring dynamic finger motion during contact <ref:2606.23689#pg2> While dense multi-view perception improves pose reliability, the highest throughput setting still relies on a multi-camera workcell <ref:2606.23689#pg2> Bimanual coordination, mobile manipulation, finger-rolling regrasps, and functional grasps remain out of scope for the current system <ref:2606.23689#pg2> References
1: Y.
Rosa: Liu, Y.
Dev: Yang, Y.
Rosa: Wang, X.
Paper summary: Rosa: So, to recap, AutoDex tackles the problem of needing real physical outcome data for grasping by creating an automated system that does the whole validation process without a human in the loop.
Dev: The thesis is that you can generate many grasp candidates and test them on hardware to get reliable labels, provided you automate the entire collection loop—perception, execution, labeling, and reset.
Rosa: They show how this works by taking generated candidates from a modular generator and running them through a whole pipeline.
Dev: This pipeline involves using dense twenty camera perception for the initial six dimensional pose estimation, executing those selected candidates with collision-monitored robot motions, and then labeling the lift-and-hold success or failure.
Taro: I’m interested in how they manage the part where a candidate might be geometrically fine but still fail because of real physics like slip or compliance.
Rosa: The paper points out that kinematic and collision feasibility checks only confirm if a grasp can be executed, but they don't certify whether the contact will stably support the object under real contact effects.
Dev: That’s where the physical validation becomes crucial; it filters out grasps that look good on paper but fail because of things like friction or multi-finger instability.
Rosa: They demonstrate this by comparing their validated database to a baseline set of grasps selected without real-world validation, showing a success rate jump from thirty-four percent up to seventy-six percent when retrieved in deployment settings.
Dev: That improvement is consistent across different materials, scene types, and object weights, which suggests the physical validation really improves the quality of the resulting dataset.
Conclusion: Rosa: So what does this whole AutoDex system actually mean for us beyond just having a better dataset?
Dev: It means we can scale up data collection far past what manual teleoperation allows, which is only about ten point three hours for five hundred trials.
Rosa: That throughput gain is significant; they compared AutoDex’s ten point three hours to forty-nine point four hours for the same study with human teleoperation.
Dev: But more importantly, it means that the data you get from this system is much more reliable than what you get from just running candidates through simulation and planning tools.
Taro: The authors acknowledge that while dense multi-view perception helps with pose reliability, there are still limitations, especially when the object is textureless or thin.
Rosa: They admit that kinematic and collision feasibility alone aren't enough to build a reliable database; you need that real-world contact physics validation.
Dev: It also notes that the system currently targets stable grasps for a fixed arm hand setup, and it doesn't cover more complex things like bimanual coordination or finger-rolling regrasps right now.
Rosa: The implication is that this system is a powerful tool for building a foundation of physically verified data, even if it stops short of covering every kind of advanced manipulation we could ever imagine.
Mingi Choi, Gunhee Kim, Jisoo Kim, Taeksoo Kim, Taeyun Ha, Jongbin Lim, Hanbyul Joo
Seoul National University
cs.RO, cs.LG
Submitted: 2026-06-22
Updated: 2026-11-04
Comments: 9 pages + references, 6 figures. Accepted to CoRL 2026
Project page: https://willi19.github.io/AutoDex
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 100/100
The gist: Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts <ref:2606.23689#pg2> Such data are difficult to obtain at scale <ref:2606.23689#pg2> A…
Terminology
Summary
Learning robust dexterous grasping requires real-world data that records the physical outcomes of grasp attempts <ref:2606. p. 023689#pg2> Such data are difficult to obtain at scale <ref:2606. p. 023689#pg2> A natural solution is to generate candidate grasps and verify them on real hardware, but this scales only if the entire collection loop—perception, execution, labeling, and reset—runs without human intervention <ref:2606. p. 023689#pg2> We present AutoDex, an automated real-world data-collection system that performs this entire validation cycle without human intervention <ref:2606. p. 023689#pg2> Given generated candidate grasps, it estimates the object’s initial 6D pose with dense 20-camera perception <ref:2606. p. 023689#pg2>, executes selected candidates with collision-monitored robot motions <ref:2606. p. 023689#pg2>, tracks the object during execution despite severe hand–object occlusion <ref:2606. p. 023689#pg2>, labels lift-and-hold success or failure <ref:2606. p. 023689#pg2>, and actively resets the object between trials to continue testing candidates across stable poses <ref:2606. p. 023689#pg2> The result is a reusable database of physically labeled grasp trials that downstream systems can retrieve and filter for feasibility <ref:2606. p. 023689#pg2>
Pruning redundant reasoning tokens
The paper introduces AutoDex, an end-to-end automated system designed to turn generated grasp candidates into physically labeled robot trials <ref:2606.23689#pg2> This system automates the entire validation cycle, which includes obtaining candidate grasps from a modular generator, executing them on physical hardware, and recording success or failure labels without human intervention <ref:2606. p. 223689#pg2> The system is realized through four components: dense 20-camera perception for initial object pose estimation and execution-time tracking under hand–object occlusion <ref:2606. p. 123689#pg2>, collision-monitored motion execution for safe unattended grasp attempts <ref:2606. p. 123689#pg2>, physical success/failure labeling from lift-and-hold trials <ref:2606. p. 123689#pg2>, and active object reset for continued testing across stable poses <ref:2606. p. 123689#pg2>
How the training signal is built
The process begins with Grasp Candidate Generation, where AutoDex obtains candidate grasps from a modular candidate generator which takes an object model and a scene specification as input to return a set of kinematically and geometrically feasible candidates <ref:2606.23689#pg2> The generation uses BODex [8] to propose candidates based on scene primitives like wall, shelf, and box constraints <ref:2606. p. 323689#pg2> Before real-robot execution, a simulation sanity pre-filter is applied where candidates are screened in MuJoCo [37] by applying a weak external force to reject those that translate more than 5 cm under perturbation <ref:2606. p. 623689#pg2> The system selects an unattempted feasible candidate for the current stable pose and plans a collision-free arm trajectory to approach the candidate wrist pose <ref:2606. p. 423689#pg2>
Where the gains come from
The primary throughput gain is demonstrated by comparing AutoDex with manual teleoperation, which requires 49.4 h for a matched 500-trajectory collection study compared to AutoDex's 10.3 h <ref:2606. p. 023689#pg2> This yields a 4.8× throughput improvement <ref:2606. p. 023689#pg2> Furthermore, grasps retrieved from the AutoDex-validated database achieve 76% real-world success, significantly higher than the 34% success rate for grasps selected without real-world validation <ref:2606. p. 023689#pg2> This improvement is consistent across material, scene, and weight categories <ref:2606. p. 623689#pg2>
Impact of physical validation on dataset quality
Physical validation substantially improves database quality by filtering out grasps that satisfy geometric constraints but fail under real-world contact effects such as slip, compliance, and multi-finger instability <ref:2606.23689#pg2> The AutoDex validated database achieves 76% grasp success when retrieved in deployment settings, compared to 34% for model-screened databases <ref:2606. p. 623689#pg2> This indicates that kinematic and collision feasibility alone are insufficient for constructing a reliable dexterous-grasp database <ref:2606. p. 723689#pg2>
Reset Strategy Evaluation
The system employs an explicit reset module to handle pose transitions between trials, which is critical because inter-trial reset cannot rely on passive settling alone <ref:2606.23689#pg2> The Stable Reorient Placement strategy maintains high success across the full difficulty range, including transitions with a passive transition probability of approximately 0 <ref:2606. p. 723689#pg2> This method avoids the failure modes incompatible with unattended operation that occur in other reset strategies <ref:2606. p. 723689#pg2>
Impact of Camera Density on Pose Reliability
Dense multi-view perception improves pose reliability for unattended collection because it provides redundant observations for robust initial pose estimation and helps recover object motion when the hand occludes much of the object <ref:2606.23689#pg2> The mean ADD-S between the full 20-camera reference pose and a k-camera subset poses decreases as camera count increases, showing that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606. p. 723689#pg2> This result indicates that dense multi-view redundancy improves pose reliability for unattended collection <ref:2606. p. 823689#pg2>
Discussion and Limitations
AutoDex currently targets stable grasps for a fixed arm-hand setup, and the system also inherits the blind spots of its upstream generator which may miss grasps requiring dynamic finger motion during contact <ref:2606.23689#pg2> While dense multi-view perception improves pose reliability, the highest throughput setting still relies on a multi-camera workcell <ref:2606. p. 823689#pg2> Bimanual coordination, mobile manipulation, finger-rolling regrasps, and functional grasps remain out of scope for the current system <ref:2606. p. 823689#pg2>
References
[1] Y. Liu, Y. Yang, Y. Wang, X. Wu, J. Wang, Y. Yao, S. Schwertfeger, S. p. 8 Yang, W. Wang, J Yu et al Realdex: Towards human-like grasping for robotic dexterous hand arXiv preprint arXiv:240213853 2024
[2] C Wang, H Shi, W Wang, R Zhang, L Fei-Fei and C K Liu Dexcap: Scalable and portable mocap data collection system for dexterous manipulation arXiv preprint arXiv:240307788 2024
[3] Z Chen, K Van Wyk Y-W Chao W Yang A Mousavian A Gupta and D Fox Dextransfer: Real world multi-fingered dexterous grasping with minimal human demonstrations arXiv preprint arXiv:220914284 2022
[4] J Zhang, H Liu D Li X Yu H Geng Y Ding J Chen and H Wang DexGraspNet 2.0 Learning generative dexterous grasping in large-scale synthetic cluttered scenes In Conference on Robot Learning CoRL 2024
[5] A Bicchi and V Kumar Robotic grasping and contact A review In Proceedings of the IEEE International Conference on Robotics and Automation ICRA volume 1 pages 348–353 2000
[6] Y Qin, W Yang B Huang K Van Wyk H Su X Wang Y-W Chao and D Fox Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system In Robotics: Science and Systems 2023
[7] R Wang J Zhang J Chen Y Xu P Li T Liu and H Wang Dexgraspnet: A largescale robotic dexterous grasp dataset for general objects based on simulation arXiv preprint arXiv:221002697 2022
[8] J Chen Y Ke and H Wang Bodex: Scalable and efficient robotic dexterous grasp synthesis using bilevel optimization arXiv preprint arXiv:241216490 2024
[9] D Turpin T Zhong S Zhang G Zhu J Liu R Singh E Heiden M Macklin S Tsogkas S Dickinson et al Fast-grasp’d: Dexterous multi-finger grasp generation through differentiable simulation arXiv preprint arXiv:230608132 2023
[10] D Huang T Zhang Y Li L Zhao J Li Z Fang C Xia and X He Dexterous grasping with real-world robotic reinforcement learning arXiv preprint arXiv:250304014 2025
[11] Y Park J S Bhatia L Ankile and P Agrawal Dart: Dexterous augmented reality teleoperation platform for large-scale robot data collection in simulation In 2025 IEEE International Conference on Robotics and Automation ICRA pages 13883–13889 IEEE 2025
[12] T Liu Z Liu Z Jiao Y Zhu and S-C Zhu Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator IEEE Robotics and Automation Letters 7(1):470–477 Jan. 2022 ISSN 2377-3774 doi:10. p. 91109/lra.2021.3129138 URL http://dx. p. 9doi.org/10.1109/LRA.2021.3129138
[35] H Lin S Chen J Liew D Y Chen Z Li G Shi J Feng and B Kang Depth anything 3: Recovering the visual space from any views arXiv:251110647 2025
[40] B Sundaralingam S K Hari A Fishman C Garrett K Van Wyk V Blukis A Millane H Oleynikova A Handa F Ramos et al Curobo: Parallelized collision-free robot motion generation In 2023 IEEE International Conference on Robotics and Automation ICRA pages 8112–8119 IEEE 2023
[47] S Hinterstoisser V Lepetit S Ilic S Holzer G Bradski K Konolige and N Navab Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes In Asian conference on computer vision 2012
[48] E Todorov T Erez and Y Tassa Mujoco: A physics engine for model-based control In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems pages 5026–5033 IEEE 2012 doi:10.1109/IROS.2012.6386109
[49] V N Nguyen C Forster B Tekin S Shkodrani V Lepetit C Keskin and T Hodan Gotrack: Generic 6dof object pose refinement and tracking Computer Vision and Pattern Recognition Workshops CVPRW 2025
[50] B Sundaralingam S K Hari A Fishman C Garrett K Van Wyk V Blukis A Millane H Oleynikova A Handa F Ramos et al Curobo: Parallelized collision-free robot motion generation In 2023 IEEE International Conference on Robotics and Automation ICRA pages 8112–8119 IEEE 2023
[51] S Mirchandani S Belkhale J Hejna E Choi M S Islam and D Sadigh So you think you can scale up autonomous robot data collection? In Conference on Robot Learning CoRL 2024
[53] N Carion L Gustafson Y-T Hu S Debnath R Hu D Suris C Ryali K V Alwala H Khedr A Huang et al Sam 3: Segment anything with concepts arXiv:251116719 2025
--- Page 6 ---
Figure 4 Left: Throughput comparison AutoDex collects 500 trials in 10.3 h compared with teleoperation requires 49. p. 04 h for the same workcell
--- Page 7 ---
Figure 5 Left: Reset strategy comparison Naive Drop follows y = x by construction its success rate equals the passive transition probability
--- Page 8 ---
Figure 5 Right: Pose self-consistency relative to the 20-camera reference as a function of camera count
--- Page 9 ---
Figure 9 Left: AutoDex object library and diversity
The paper presents AutoDex, an automated system for collecting physically labeled dexterous-grasp trials through real-world execution, labeling, and reset <ref:2606.23689#pg2> The results show that real-world validation improves collection throughput over teleoperation and produces a grasp database that transfers more reliably than candidates screened without physical execution <ref:2606. p. 823689#pg2>
The system operates by estimating the object’s 6D pose from multi-view observations, identifying the current stable pose Pi ∈ P, and transforming each object-relative candidate in Gcand(Pi) into the robot frame <ref:2606.23689#pg2> AutoDex then selects an unattempted feasible candidate and plans a collision-free arm trajectory to approach the candidate wrist pose <ref:2606. p. 423689#pg2> The hand then moves through the candidate pre-grasp and grasp configurations, after which the robot lifts the object and holds for a fixed duration <ref:2606. p. 423689#pg2>
The throughput gain stems from eliminating human idle time and enabling sustained unattended collection across extended runs <ref:2606.23689#pg2> The database quality is improved because physical validation filters out grasps that satisfy geometric constraints but fail under real contact effects such as slip, compliance, and multi-finger instability <ref:2606. p. 723689#pg2> The reset module's success in maintaining high success across the full difficulty range demonstrates that explicit reset placement is critical for reliable unattended dexterous-grasp collection <ref:2606.23689#pg2>
Dense multi-view perception improves pose reliability because it provides redundant observations for robust initial pose estimation and helps recover object motion when the hand occludes much of the object <ref:2606.23689#pg2> The result is that kinematic and collision feasibility alone are insufficient for constructing a reliable dexterous-grasp database <ref:2606. p. 723689#pg2> The high error at k = 2 is driven by occasional catastrophic pose failures on textureless or thin objects, which are largely eliminated by k = 8 <ref:2606. p. 8
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated real-world data collection system AutoDex can perform
end-to-end automated real-world data collection at scale
by combining candidate generation, execution, labeling, and active object reset without human intervention. -
The improved AI system can create a reusable database of physically labeled grasp trials that downstream systems can retrieve and filter for feasibility in new scenes, achieving
76% real-world success
for retrieved grasps compared to 34% for model-screened ones. -
The system gains a throughput improvement over teleoperation by eliminating human idle time, enabling
sustained unattended accumulation across extended runs.
-
The improved AI can filter out geometrically feasible but physically unstable candidates by using the
simulation sanity pre-filter
in MuJoCo, which rejects candidates thatfail a minimal plausibility check
under weak perturbation. -
The system utilizes dense 20-camera perception to enable robust initial pose estimation and tracking
despite severe hand–object occlusion,
improving object-pose reliability during collection. -
The active reset module allows the system to move the object between stable poses, ensuring
unattended collection across object poses
by generating areset grasp
that is collision-free in a combined pickup-placement scene. -
Downstream systems can perform
in-the-wild execution without policy training or additional real-world trial collection
by retrieving successful grasps and checking their feasibility against the current workspace obstacles.
Sources
- RealDex: Towards Human-like Grasping for Robotic Dexterous Hand
- DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
- DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations
- DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation
- BODex: Scalable and Efficient Robotic Dexterous Grasp Synthesis Using Bilevel Optimization
- Fast-Grasp'D: Dexterous Multi-finger Grasp Generation Through Differentiable Simulation
- Dexterous Grasping with Real-World Robotic Reinforcement Learning
- SpringGrasp: Synthesizing Compliant, Dexterous Grasps under Shape Uncertainty
- DDGC: Generative Deep Dexterous Grasping in Clutter
- Learning Robust Real-World Dexterous Grasping Policies via Implicit Shape Augmentation
- Solving Rubik's Cube with a Robot Hand
- AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
- Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware
- SAM 3: Segment Anything with Concepts
- Depth Anything 3: Recovering the Visual Space from Any Views
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation