SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale

arXiv:2606.13497 · cs.RO, cs.CV · Submitted 2026-06-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale".

Dev: The gist The SPARC framework introduces a risk-aware system that automatically labels robot demonstrations with structured spatial annotations and assigns each annotation a reliability score,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So to recap, SPARC is this risk-aware framework designed to automatically label robot demonstrations with structured spatial annotations—bounding boxes, object trajectories, and manipulation phase labels—and critically, it assigns a reliability score to every single one of those annotations.

Dev: The core thesis is that existing automated pipelines fail because detector confidence isn't calibrated correctly for how right or wrong the annotation actually is, forcing you into a choice between noisy data or discarding useful information.

Rosa: SPARC fixes that by leveraging the spatio-temporal structure of robot tasks to generate a reliability signal directly, rather than just using raw detector output. This allows them to reduce those noisy labels and keep more useful samples in the learning process.

Dev: The methodology breaks down into three stages: subtask decomposition based on phase detection, candidate proposal and tracking of objects over time, and finally, a reliability scoring mechanism grounded in physical interaction evidence.

Taro: I want to ask about that phase detection part. How does the system decide what's happening in a frame versus what’s just background noise or irrelevant movement? That seems like a big assumption for any computer vision pipeline.

Rosa: Stage one involves interaction segmentation through phase detection, where they analyze visual frames, proprioceptive states from the robot itself, and language instructions to decompose the demo into short object-centric subtasks. This helps them extract things like the manipulated object name and its initial and target locations based on what the language instruction tells them.

Dev: Then Stage two focuses on candidate proposal and tracking. SPARC runs an LLMDet at a keyframe to propose initial candidates with bounding boxes and segmentation masks, but they then rank these candidates using that Composite Annotation Reliability Score Ri we talked about earlier.

Taro: And the ranking is what’s interesting. They combine appearance reliability from the detector confidence, manipulation consistency from phase-aware object motion, and embodiment consistency based on three dee gripper proximity to get that final score Ri <ref:2606.13497#pg1>.

Rosa: That formula is key: Ri equals wA Abi plus wD(Pbi) Di plus wP Pbi minus Pi, where those weights define how much they trust each piece of physical evidence. It's not just one signal; it’s a weighted combination of appearance, manipulation, and embodiment cues.

Dev: And this whole process is what leads to their large-scale output: 167k annotated trajectories spanning twelve data sources and supporting things like object localization, trajectory prediction, and action generation <ref:2606.13497#pg2>.

Taro: So the paper shows that this structured annotation process doesn't just produce labels; it produces high-quality supervision that significantly helps policies learn to manipulate objects robustly in real-world settings.

Rosa: Exactly. They show that fully automatic SPARC supervision matches or exceeds human-annotated data quality on embodied grounding, and the resulting reasoning policies are more robust when dealing with cluttered real-world manipulation.

Dev: The results on IA-Bench confirm this: SPARC reasoning scores are sixty-four percent, compared to twenty-one percent for detection baselines, showing a significant leap in spatial understanding <ref:2606.13497#pg2>.

Conclusion: Rosa: So, looking at this SPARC paper by Blank, Mattes, Xiling Li, Suliga, Roth, and Reuss—the authors—the main point is that they’ve moved away from just using raw detection confidence to generate a more physically grounded way to supervise robot demonstrations.

Dev: They’ve reframed the automatic spatio-temporal annotation as a selective prediction problem where the reliability score is grounded in physical interaction evidence, which is what makes it so much better than previous methods.

Rosa: The implication for us on the ground is that we can use this system to create massive datasets—167k trajectories—that are much higher quality than what we could gather manually, without needing a huge team of human annotators just to get started <ref:2606.13497#pg2>.

Dev: And for anyone building embodied foundation models or training policies, this means they can use these SPARC annotations to achieve state-of-the-art results on object grounding and pointing benchmarks, even when the environments are visually ambiguous.

Taro: It changes the way we think about creating supervision for embodied agents; instead of just collecting raw video frames, we need to understand the interaction physics—the manipulation phase—to get useful spatial data.

Rosa: Right. And because they retained more usable samples at a fixed precision target, it gives us more robust training data to work with than we ever had before.

Dev: The limitation they flag is that the method relies on off-the-shelf models and detector errors, and it also assumes a mostly static camera setup, meaning the accuracy of those physical cues might be affected by dynamic camera movement or rapid changes.

Taro: That makes sense; if the environment isn't stable, those physical cues for scoring Ri might become less reliable. It’s not perfect in every chaotic scenario yet.

Rosa: So SPARC is a powerful tool for scaling up robot learning, but we have to keep an eye on those assumptions about camera stability and detector performance when we try to deploy this outside of the controlled lab setting.

Nils Blank, Paul Mattes, Maximilian Xiling Li, Jakub Suliga, Thomas Roth, Moritz Reuss, Pankhuri Vanjani, Rudolf Lioutikov

Karlsruhe Institute of Technology 2023 Karlsruhe Institute of Technology 3 NVIDIA Robotics Institute Germany

cs.RO, cs.CV

Submitted: 2026-06-11

Updated: 2026-10-14

Comments: CoRL 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The SPARC framework introduces a risk-aware system that automatically labels robot demonstrations with structured spatial annotations and assigns each annotation a reliability score, which

Key concepts

Structured Spatial Annotations: These are detailed labels like bounding boxes, object trajectories, and manipulation phase markers applied to robot videos. They provide rich spatial and temporal information about what the robot is doing in a scene, which is crucial for training robots to understand physical tasks.
Reliability Calibration: SPARC assigns a numerical score to every annotation based on multiple physical cues—like detector confidence, motion consistency, and gripper proximity. This score quantifies how trustworthy each label is, allowing the system to prioritize high-quality data over noisy labels automatically.
Interaction-Aware Bench (IA-Bench): This is a new benchmark designed to test how well models can accurately locate objects during robot interactions. It uses video and proprioception across different robot setups to measure grounding accuracy, providing a rigorous way to evaluate the quality of SPARC's annotations.
Composite Annotation Reliability Score (Ri): This score is calculated by combining four physically grounded cues: appearance reliability (from detector confidence), manipulation consistency (from object motion phase), embodiment consistency (from gripper proximity), and a penalty term. This comprehensive score determines the final trustworthiness of an annotation.

Terminology

Accuracy — 2× in this paper · explained in 1 episode
Benchmark — 2× in this paper · explained in 2 episodes
Grounding — 2× in this paper · explained in 3 episodes
Calibration — 1× in this paper · explained in 7 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes

Summary

The gist The SPARC framework introduces a risk-aware system that automatically labels robot demonstrations with structured spatial annotations and assigns each annotation a reliability score, which significantly improves downstream embodied reasoning and policy learning.

How it works

SPARC operates as an automatic pipeline that produces spatio-temporal annotations from robot demonstrations by identifying the interacted object and generating its start and target locations, its movement over time as a 3D object trajectory, and a continuous reliability score for each annotation The pipeline proceeds in three stages: subtask decomposition, candidate proposal and tracking, and reliability scoring

Stage 1 involves interaction segmentation through phase detection This stage decomposes the demonstration into short object-centric subtasks by analyzing visual frames, proprioceptive states, and language instructions It extracts information such as the manipulated object name and initial and target locations based on the language instruction

Stage 2 focuses on candidate proposal and tracking SPARC compiles a set of candidate objects by running LLMDet at a keyframe to propose initial candidates with bounding boxes and segmentation masks This stage ranks candidates by combining four physically grounded cues into a Composite Annotation Reliability Score Ri: appearance reliability from detector confidence, manipulation consistency from phase-aware object motion, and embodiment consistency from 3D gripper proximity The final score is calculated as Ri = wA Abi + wD(Pbi) Di + wP Pbi − Πi, where the weights are defined by the formula

Evaluation and Results

To evaluate the quality of the annotations, SPARC introduces Interaction-Aware Bench (IA-Bench) The pipeline achieves high performance on this benchmark Furthermore, models finetuned on SPARC annotations achieve state-of-the-art results on object-grounding and pointing benchmarks among similarly sized models Policies trained on SPARC annotations significantly outperform baselines in cluttered, visually ambiguous real-world scenes

Key Contributions

The contributions of this work include three main points (ii) We introduce IA-Bench, an interaction-aware benchmark with video and proprioception spanning diverse embodiments and bimanual setups.> (iii) We generate a large-scale, high-quality dataset of 167k annotated trajectories and show that fully automatic SPARC supervision matches or exceeds human-annotated data on embodied grounding and yields reasoning policies more robust in cluttered real-world manipulation.>

Downstream Applications

SPARC annotations are useful downstream for various applications Models fine-tuned on SPARC-generated annotations outperform similarly-sized state-of-the-art models trained on human-annotated data across objectgrounding and pointing benchmarks such as Where2Place [15], VABench [17], RoboRefIt [18], RefSpatial [16], and ShareRobot [7]. Furthermore, policies trained on SPARC-annotated data outperform baselines in cluttered, visually ambiguous real-world scenes

Limitations

The limitations of SPARC include reliance on off-the-shelf models and detector errors It also assumes a mostly static camera and relies on accurate phase intervals

Conclusion

SPARC introduces an automatic spatial annotation pipeline that scores candidates with physically grounded motion cues rather than detector confidence The resulting reliability score makes the trade-off between quality and scale controllable without any human review SPARC yields more accurate annotations than detection-based baselines and retains more usable data at a fixed precision target Models trained on our annotations surpass those trained on human-annotated data across object-grounding and motion-aware benchmarks, and reasoning policies trained on them are substantially more robust in cluttered real-world manipulation<ref:2606.13497M,Models trained on our annotations surpass those trained on human-annotated data across objectgrounding and pointing benchmarks such as Where2Place [15], VABench [17], RoboRefIt [18], RefSpatial [16], and ShareRobot [7].

Improvements for AI systems

Bold header: Reliability Calibration via Physical Cues for Annotation Quality

This framework introduces a Composite Annotation Reliability Score that effectively captures how likely an object was interacted with in a demonstration by combining phase-aware motion, gripper proximity, and a robot-overlap filter. This allows users to explicitly trade off annotation quality and dataset scale without human review.

Bold header: Interaction-Aware Benchmarking for Grounding Accuracy

The introduction of the InteractionAware Bench (IA-Bench) measures model accuracy in grounding the locations of interacted objects in robot demonstrations, showing that SPARC achieves 80.2% interacted object localization accuracy at high confidence, compared to 58.1% for a detection-only baseline.

Bold header: Robust Downstream VLM Training Supervision

Models fine-tuned on SPARC annotations achieve state-of-the-art results on object-grounding and pointing benchmarks among similarly sized models and significantly outperform baselines in cluttered, visually ambiguous real-world scenes.

Bold header: Scalable Data Filtering via Controllable Quality Tradeoff

The system enables the creation of a high-quality VQA dataset spanning 511k samples by thresholding SPARC annotations, demonstrating that the score provides a practical operating point for trading dataset scale against annotation quality.

Bold header: Enhanced Policy Robustness in Cluttered Environments

Policies trained on SPARC-generated annotations significantly outperform baselines in cluttered, visually ambiguous real-world scenes, indicating that the reliability score helps select annotations consistent with the actual interaction rather than merely plausible in a single image.

Abstract

This work introduces Spatial Annotations from Robot Demonstrations with Reliability Calibration (SPARC), a risk-aware framework that automatically labels robot demonstrations with structured spatial annotations and assigns each annotation a reliability score. Structured spatial annotations, such as bounding boxes, object trajectories, and manipulation phase labels, benefit a broad range of robotics applications from training grounded robot policies and embodied foundation models to motion planning and hierarchical task composition. Existing automated pipelines generate such annotations at scale but provide no reliable quality signal: detector confidence is poorly calibrated for annotation correctness, forcing a choice between accepting noisy labels or discarding useful samples. In contrast to existing automated pipelines, SPARC leverages the spatio-temporal structure inherent to robot tasks to generate a reliability signal, reducing noisy labels and retaining more useful samples. We further introduce Interaction-Aware Bench (IA-Bench), a benchmark that measures model accuracy in grounding the locations of interacted objects in robot demonstrations. On 1.7k human-annotated demonstrations spanning diverse embodiments and scenarios, SPARC significantly outperforms detection-only baselines in localization accuracy while retaining three times more samples at high-precision operating points. Our experiments demonstrate that models finetuned on our annotations achieve state-of-the-art results on object-grounding and pointing benchmarks among similarly sized models, while remaining competitive on broader spatial-reasoning suites without manually verified or annotated training data. Furthermore, policies trained on SPARC-generated annotations outperform baselines in cluttered, visually ambiguous real-world scenes. Code, data, and models are available at intuitive-robots.github.io/sparc-labeling.