TARGO and TARGO-Net: Benchmarking Target-driven Object Grasping under Occlusions
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "TARGO and TARGO-Net".
Paper summary: Rosa: So, we just talked about the basics of "TARGO and TARGO-Net: Benchmarking Target-driven Object Grasping under Occlusions." The paper sets up this new benchmark dataset called TARGO to study target-driven grasping when there's occlusion.
Dev: Yeah, it’s important because it investigates how the level of occlusion affects five previous state-of-the-art models for the target-driven grasping task. They aim to assess how well those models generalize when they face different levels of occlusion in both synthetic and natural environments.
Taro: The key finding they highlight is that the current state of the art models have an apparent decrease in their grasp success rate as the occlusion level increases. This shows that grasping under occlusion is still a challenge even for the best existing methods.
Rosa: And to address this, they introduce TARGO-Net, which is a transformer based model incorporating a shape completion module. They claim this new approach employs more effective shape completion than models like GIGA when dealing with scene completion.
Dev: The methodology involves using the TARGO-Synthetic and TARGO-Real datasets to validate this effectiveness across both synthetic and natural environments, demonstrating that TARGO-Net is robust in target-driven grasp under occlusion.
Taro: They also found that incorporating single scenes where only the target object is visible can increase the robustness of grasping models against occlusion, which they draw inspiration from previous work.
Rosa: It seems like they’re making a case for this new approach by showing that it handles occlusion better than existing methods on these specific datasets and in these specific environments.
Dev: The paper also mentions that the target object size influences the success rate when it is at the same occlusion level, and the gripper width has an impact depending on that size.
Taro: So, what this means for us in real-world testing is that we can use these benchmarks to test if our systems handle these varying degrees of occlusion gracefully.
Conclusion: Rosa: To wrap up "TARGO and TARGO-Net: Benchmarking Target-driven Object Grasping under Occlusions," the authors establish a new benchmark dataset called TARGO and propose TARGO-Net to handle occlusions in target-driven grasping.
Dev: The title itself tells you the focus: it’s about benchmarking target-driven object grasping specifically when there are occlusions involved, which is a niche that hasn't been studied this way before.
Taro: The implication is that we now have a standard way to measure how much occlusion affects these grasping systems across different environments, both synthetic and real-world.
Rosa: They show that the current state of the art models still struggle when occlusion increases, but TARGO-Net shows better performance because of its shape completion module.
Dev: It’s about moving towards a model that can handle clutter and partial visibility more reliably so we can deploy robots in these challenging scenarios with a higher safety case.
Yan Xia, Ran Ding, Ziyuan Qin, Guanqi Zhan, Kaichen Zhou, Long Yang, Hao Dong, Daniel Cremers
Technical University of Munich Department of Computer Science, Peking University, Munich Center for Machine Learning
cs.RO, cs.CV
Submitted: 2024-07-08
Updated: 2026-11-15
Comments: Published at International Journal of Computer Vision (IJCV), 2026
Code: https://github.com/ethz-asl/vgn
Project page: https://targo-benchmark.github.io
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: Recent advances in predicting 6D grasp poses from a single depth image have led to promising performance in robotic grasping However…
Key concepts
- TARGO
- This is a new benchmark dataset specifically designed to study the robustness of target-driven grasping when objects are partially hidden or occluded in cluttered environments. It consists of both large-scale synthetic data (TARGO-Synthetic) and real-world data (TARGO-Real) featuring natural occlusion.
- TARGO-Net
- This is a proposed transformer-based grasping model that incorporates a 3D shape completion module. This module helps the network reconstruct missing parts of an object's shape, which makes the model much more robust and effective at detecting grasps when objects are heavily occluded.
- Shape Completion Module
- This is a component within TARGO-Net that uses deep learning to infer or complete the 3D geometry of an object based on available visual information. It is crucial because it allows the model to predict accurate grasp poses even when parts of the target object are not visible due to other objects in the scene.
- Target-driven Grasping
- This refers to a robotic grasping task where the system is specifically trained or focused on successfully grasping a designated target object, rather than just finding any graspable point in an unstructured scene. The model learns to prioritize and accurately predict grasps for the specific intended target.
Terminology
Summary
Recent advances in predicting 6D grasp poses from a single depth image have led to promising performance in robotic grasping However, previous grasping models face challenges in cluttered environments where nearby objects impact the target object’s grasp In this paper, we first establish a new benchmark dataset for TARget-driven Grasping under Occlusions, named TARGO We propose a transformer-based grasping model involving a shape completion module, termed TARGO-Net, which performs most robustly as occlusion increases
Benchmark Dataset Establishment
The authors establish the TARGO benchmark dataset to study the robustness of target-driven grasp models against occlusion This benchmark consists of large-scale synthetic data and part of real-world data TARGO includes synthetic and real-world datasets named TARGO-Synthetic and TARGO-Real, respectively The synthetic dataset is derived from VGN [1] and created by synthetically arranging multiple objects on a tabletop in a disorganized fashion Furthermore, we introduce real-world grasping data, TARGO-Real, which comprises real-world objects with natural occlusion
Model Architecture and Training Signal
The proposed model is TARGO-Net, which involves a 3D shape completion module The TARGO-Net pipeline consists of instance segmentation, shape completion, and grasp detection modules The input to the grasp detection network involves concatenating the cluttered scene point cloud and the target point cloud The model is trained end-to-end on simulated ground-truth grasps with the loss function L(ˆg, g) = Lq(ˆq, q) + q(Lr(ˆr, r) + Lw(ˆw, w))
Performance Against Occlusion
Experimental results show that the current SOTA models have an apparent decrease in grasp success rate as the occlusion level increases TARGO-Net drops the least, only about 7%, and is less affected by severe occlusion In real-world experiments, TARGO-Net achieves higher success rates and picks more targets at each occlusion level compared to GIGA
Key Findings and Contributions
The authors make the following contributions: 1) We are the first to study the occlusion level of grasping 2) We set up a new benchmark dataset, TARGO, to study occlusion in target-driven robotic grasping task 3) We propose an effective transformer-based grasping model, TARGO-Net, involving a 3D shape completion module Extensive experiments in both simulation and real-world environments show it performances better than the state-of-the-art under occlusion The work demonstrates that augmentation with single scenes benefits all models, target as input improves performance, and shape completion is beneficial but does not generalize to real-world environments Additionally, the target object size influences the success rate when it is at the same occlusion level The gripper width also has an impact on success rate depending on the target object size The paper concludes by proposing TARGO-Net as a robust solution to grasping under occlusion in robotics.
REFERENCES
[1] Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Siegwart, and Juan Nieto. Volumetric grasping network: Real-time 6 dof grasp detection in clutter. p. 9 In Conference on Robot Learning, pages 1602–1611. p. 9 PMLR, 2021
[2] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. p. 9 In 2015 international conference on advanced robotics (ICAR), pages 510–517. p. 9 IEEE, 2015
[3] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield et al. Dexycb: A benchmark for capturing hand grasping of objects. p. 9 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pages 9044–9053, 2021
[4] Nikhil Chavan-Dafle, Sergiy Popovych, Shubham Agrawal, Daniel D Lee, and Volkan Isler. Simultaneous object reconstruction and grasp prediction using a camera-centric object shell representation. p. 9 In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 1396–1403, IEEE, 2022
[5] Sang Hun Cheong, Brian Y Cho, Jinhwi Lee, ChangHwan Kim, and Changjoo Nam. Where to relocate? p. 9: Object rearrangement inside cluttered and confined environments for robotic manipulation. p. 9 In 2020 IEEE International Conference on Robotics and Automation ICRA pages 7791–7797, IEEE, 2020
[6] Woojin Chung, Seokgyu Kim, Minki Choi, Jaesik Choi, Hoyeon Kim, Chang-bae Moon, and Jae-Bok Song. Safe navigation of a mobile robot considering visibility of environment. p. 9 IEEE Transactions on Industrial Electronics 56(10):3941–3950, 2009
[7] Clemens Eppner, Arsalan Mousavian, and Dieter Fox. Acronym: A large-scale grasp dataset based on simulation. p. 9 In 2021 IEEE International Conference on Robotics and Automation ICRA pages 6222–6227, IEEE, 2021
[8] Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A largescale benchmark for general object grasping. p. 9 In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pages 11444–11453, 2020
[9] Shresth Grover, Vibhav Vineet, and Yogesh Rawat. Revealing the unseen: Benchmarking video action recognition under occlusion. p. 9 Advances in Neural Information Processing Systems 36, 2024
[10] Basile Van Hoorick, Pavel Tokmakov, Simon Stent, Jie Li, and Carl Vondrick. Tracking through containers and occluders in the wild, 2023
[11] Cheng-Yen Hsieh, Tarasha Khurana, Achal Dave, and Deva Ramanan. Tracking any object amodally. p. 9 arXiv preprint arXiv:2312. p. 9 p. 912433, 2023
[12] Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, Robert Platt. Edge grasp network: A graph-based se (3)-invariant approach to grasp detection. p. 9 In 2023 IEEE International Conference on Robotics and Automation ICRA pages 3882–3888, IEEE, 2023
[15] Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. p. 9 Robotics: science and systems 2021
[16] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion-aware instance segmentation with overlapping bilayers. p. 1 In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition CVPR pages 4019–4028, 2021
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. p. 6 The 3rd International Conference for Learning Representations, 2015
[18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. Segment anything. p. 4 arXiv preprint arXiv:2304. p. 902643, 2023
[19] Adam Kortylewski, Ju He, Qing Liu, and Alan L Yuille. Compositional convolutional neural networks: A deep architecture with innate robustness to partial occlusion. p. 1 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pages 8940–8949, 2020
[20] James J Kuffner. Effective sampling and distance metrics for 3d rigid body path planning. p. 1 In IEEE International Conference on Robotics and Automation ICRA, 2004. p. 9 Proceedings ICRA’04, volume 4 pages 3993–3998, IEEE, 2004
[21] Andrey Kurenkov, Joseph Taglic, Rohun Kulkarni et al. Visuomotor mechanical search: Learning to retrieve target objects in clutter. p. 0 In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 8408–8414, IEEE, 2020
[22] Jinhwi Lee, Younggil Cho, Changjoo Nam, Jonghyeon Park et al. Efficient obstacle rearrangement for object manipulation tasks in cluttered environments in 2019 International Conference on Robotics and Automation ICRA pages 183–189, IEEE, 2019
[23] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. p. 1 The International journal of robotics research 37(4-5):421–436, 2018
[24] Yu Li, Kai Cheng, Ruihai Wu, Yan Shen, Kaichen Zhou, Hao Dong. Mobileafford: Mobile robotic manipulation through differentiable affordance learning. p. 9 In 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024, 2024
[25] Zhan Liu, Ziwei Wang, Sichao Huang, Jie Zhou, Jiwen Lu. Ge-grasp: Efficient targetoriented grasping in dense clutter. p. 9 In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 1388–1395, IEEE, 2022
[26] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for realtime object recognition. p. 1 In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 922–928, IEEE, 2015
[27] Rajat Modi, Vibhav Vineet, and Yogesh Rawat. On occlusions in video action detection: Benchmark datasets and training recipes. p. 1 Advances in Neural Information Processing Systems 36, 2023
[28] Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner et al. 6-dof grasping for target-driven object manipulation in clutter. p. 1 In 2020 IEEE International Conference on Robotics and Automation ICRA pages 6232–6238, IEEE, 2020
[29] Ege Ozguroglu, Ruoshi Liu, Dídac Surís, Dian Chen et al. pix2gestalt: Amodal segmentation by synthesizing wholes. p. 1 arXiv preprint arXiv:2401. p. 914398, 2024
[30] Songyou Peng, Michael Niemeyer, Lars Mescheder et al. Convolutional occupancy networks. p. 2 In Computer Vision–ECCV 2020: 16th European Conference Glasgow UK, August 23–28, 2020 Proceedings Part III 16 pages 523–540, Springer, 2020
[31] Abhishek Sharma, Oliver Grau, and Mario Fritz. Vconv-dae: Deep volumetric shape learning without object labels. p. 1 In European Conference on Computer Vision pages 236–250, Springer, 2016
[32] Maximilian Stölzle, Takahiro Miki, Levin Gerdes et al. Reconstructing occluded elevation information in terrain maps with self-supervised learning. IEEE Robotics and Automation Letters 7(2):1697–1704, 2022
[33] Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. p. 1 In 2021 IEEE International Conference on Robotics and Automation ICRA pages 13438–13444, IEEE, 2021
[34] Lyne P Tchapmi, Vineet Kosaraju, Hamid Rezatofighi et al. Topnet: Structural point cloud decoder. p. 2 In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pages 383–392, 2019
[35] Angtian Wang, Yihong Sun, Adam Kortylewski, and Alan L Yuille. Robust object detection under occlusion with context-aware compositionalnets. p. 8 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pages 12645–12654, 2020
[36] Lizi Wang, Hongkai Ye, Qianhao Wang et al. Learning-based 3d occupancy prediction for autonomous navigation in occluded environments in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 4509–4516, IEEE, 2021
[37] Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for deformable object manipulation. p. 5 In IEEE International Conference on Computer Vision ICCV 2023
[38] Ruihai Wu, Yan Zhao, Kaichun Mo et al. VAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects in International Conference on Learning Representations 2022
[39] Zhirong Wu, Shuran Song, Aditya Khosla et al. 3d shapenets: A deep representation for volumetric shapes. p. 1 In Proceedings of the IEEE conference on computer vision and pattern recognition pages 1912–1920, 2015
[40] Yan Xia, Mariia Gladkova, Rui Wang, Qianyun Li et al. Casspr: Cross attention single scan place recognition. p. 5 In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 8461–8472, 2023
[41] Yan Xia, Yusheng Xu, Cheng Wang, and Uwe Stilla. Vpc-net: Completion of 3d vehicles from mls point clouds. p. 2 ISPRS Journal of Photogrammetry and Remote Sensing 174:166–181, 2021
[42] Yaqi Xia, Yan Xia, Wei Li, Rui Song et al. Asfm-net: Asymmetrical siamese feature matching network for point completion. p. 2 In Proceedings of the 29th ACM international conference on multimedia pages 1938–1947, IEEE, 2021
[43] Katherine Xu, Lingzhi Zhang, and Jianbo Shi. Amodal completion via progressive mixed context diffusion. p. 0 arXiv preprint arXiv:2312.15540, 2023
[44] Yang Yang, Hengyue Liang, and Changhyun Choi. A deep learning approach to grasping the invisible. p. 1 IEEE Robotics and Automation Letters 5(2):2232–2239, 2020
[45] Xumin Yu, Yongming Rao, Ziyi Wang et al. Adapointr: Diverse point cloud completion with adaptive geometry-aware transformers. p. 2 IEEE Transactions on Pattern Analysis and Machine Intelligence 174:166–181, 2023
[46] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pcn: Point completion network. p. 5 In 2018 International Conference on 3D Vision (3DV) pages 728–737, IEEE, 2018
[47] Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. p. 1 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pages 11141–11150, 2021
[48] Andy Zeng, Shuran Song, Kuan-Ting Yu et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. p. 0 The International Journal of Robotics Research 41(7):690–705, 2022
[49] Guanqi Zhan, Weidi Xie, and Andrew Zisserman. A tri-layer plugin to improve occluded detection. p. 5 British Machine Vision Conference BMVC, 2022
[50] Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. Amodal ground truth and completion in the wild. p. 5 The IEEE Conference on Computer Vision and Pattern Recognition CVPR 2024
A.1 Overview
All benchmark datasets and the associated code are accessible to the public via https://TARGO-benchmark.github.io/
A.2 License
We utilize the standard CC BY-NC 4.0 licenses from the community and offer the following links to the non-commercial licenses for the datasets used in this paper
A.3 Dataset
This section details generating our cluttered and single scenes in our TARGO dataset We first explain why only packed scenes are used in our grasping datasets and then introduce why the single scenes can help the grasping success rates
A.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Effective training data augmentation for models against occlusion can be achieved by incorporating single scenes where
only the target object in a scene can increase the robustness of grasping models against occlusion.
-
The proposed TARGO-Net transformer-based model involving a shape completion module
performs most robustly as occlusion increases,
allowing it to maintain performance even when existing SOTA models suffer an apparent decrease in grasp success rate as occlusion level increases. -
TARGO-Net can be used to
segment and reconstruct the target object, then plans the optimal target grasp, enhancing both accuracy and efficiency in cluttered environments
by directly grasping the target without removing obstructing items. -
The system can be evaluated on real-world data for zero-shot generalization using the TARGO-Real dataset, which comprises
real-world objects with natural occlusion,
enabling validation of models trained in synthetic environments within a real-world setting. -
The model's performance can be optimized by using a specific number of Cross Attention Transformer layers, as
two layers strike a balance between model complexity and the ability to capture essential features without overfitting.
-
TARGO-Net can achieve superior shape completion compared to scene completion approaches, as it demonstrates
better completion performance than GIGA-Geo and GIGA
for reconstructing target shapes like cuboids and hexagonal prisms. -
The system can be improved by incorporating a grasp sampling strategy that involves processing query points from a grid of 64,000 points to generate output voxels for quality, rotation, and width fields.
-
The model can be made more resilient to noise by using a specific Gaussian noise level during testing; for instance,
with more severe Gaussian noise (sigma = 0.005), TARGO-Net w/o SC is, on average 2% better than the second best model and 5% better than the worst model in these bins.
-
The system can be tuned based on target object size; objects with sizes in the range of (0.0509, 0.0626] have the highest grasp success rates, suggesting that
an appropriate object size facilitates grasping success better.
Sources
- TAO-Amodal: A Benchmark for Tracking Any Object Amodally
- Segment Anything
- pix2gestalt: Amodal Segmentation by Synthesizing Wholes
- Amodal Completion via Progressive Mixed Context Diffusion
- ICGNet: A Unified Approach for Instance-Centric Grasping
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation