RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning

arXiv:2602.18742 · cs.RO, cs.AI, cs.CV · Submitted 2026-02-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning".

Dev: Here's the gist of the paper. Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos <ref:2602.18742#pg2> Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves <ref:2602.18742#pg4>.

Rosa: To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay <ref:2602.18742#pg6>.

Dev: Framework Overview RoboCurate is a unified framework that (1) generates synthetic robot data and (2) filters neural trajectory by quality via simulator-replay consistency, while supporting diverse scene and task generation <ref:2602.18742#pg7>.

Rosa: The core idea is to leverage a simulator to replay the generated action sequences, producing proxy robot videos with guaranteed correspondence to the generated actions <ref:2602.18742#pg6>.

Paper summary: Rosa: So we've been talking about RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning. The authors are Seungku Kim, Suhyeok Jang, Byungjun Yoon, Dongyoung Kim, John Won, and Jinwoo Shin >

Dev: It seems the core of the work is moving beyond just generating videos and focusing on verifying the actions within those videos by checking them against a simulator replay to ensure physical consistency >

Rosa: And they achieve this while simultaneously increasing observation diversity through image-to-image editing for scenes and video-to-video transfer for appearances, which is what really lets the robot explore new things >

Dev: The implication here is that visual diversity of the neural trajectory matters a lot for how well the downstream policy performs, and that their method of filtering data based on motion consistency is effective both when training policies and when sampling from generated videos >

Rosa: It’s about creating synthetic robot data that's not just visually appealing but physically grounded, which is what helps policies learn novel actions more effectively >

Dev: It suggests that for building robust robot policies, we need a way to verify the action quality itself rather than just relying on visual checks or simulation alone >

Rosa: That’s the main point of RoboCurate. It gives us a system to generate data that is better for learning and deploying robots in the real world >

Conclusion: Rosa: So, RoboCurate is about taking synthetic robot data and actually checking if the actions in those videos make sense by comparing them to what a simulator predicts they should be doing.

Dev: Yeah, it's moving past just making pretty robot videos to making sure the movements are physically realistic according to a simulation.

Rosa: The authors, Kim and Jang and their team, they built this framework to generate data that has more variety in scenes and actions than just standard methods can give you.

Taro: They use image-to-image editing for scenes to get more visual differences, and video-to-video transfer for appearances to change how things look without messing up the actual motion.

Dev: That's the diversity part, but the real trick is this attention probe they put on top of a video encoder to check motion consistency between the generated video and the simulator replay.

Rosa: So, if that probe says they match well, then you know both the visuals and what they were doing are probably correct.

Taro: It sounds like a safety layer for policy learning; it's filtering out junk data that looks plausible but is physically impossible or just wrong.

Dev: The results show some big jumps in success rates, like a seventy point one percent increase on the GR-one Tabletop task when they used this method.

Rosa: And they even showed it works really well when you fine-tune it with data from actual robots, where the improvement is almost eighty percent.

Taro: What's interesting is that they tested it on a really hard environment, like the ALLEX humanoid dexterity task, and the robot got much better at picking and placing new objects.

Dev: It also showed they can even learn entirely new actions that weren't in their original training set, going from zero percent success to twenty-five percent success on novel tasks.

Rosa: So, this whole idea points to the fact that making the synthetic data more diverse isn't just about looking pretty; it directly helps the robot policy learn better actions.

Taro: It seems like verifying action quality is a really important step if you want your policy to actually work when it leaves the lab.

Dev: They also pointed out that their method works not just for one type of robot, but that the knowledge they get about how to generate good actions can be shared across different robot bodies.

Rosa: It's a framework for making synthetic data more trustworthy, which is crucial if we want to use it to train real robots.

KAIST

cs.RO, cs.AI, cs.CV

Submitted: 2026-02-21

Updated: 2026-11-05

Comments: CoRL 2026 G2A Workshop Accepted (Oral); Project page is available at https://seungkukim.github.io/robocurate/

Code: https://github.com/nvidia-cosmos/cosmos-predict2

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline…

Key concepts

Simulator Replay Consistency
This is a core filtering technique where generated robot actions are replayed in a simulator to create proxy videos. By comparing these simulated videos with the original generated videos, researchers can identify and filter out actions that look physically inconsistent or unrealistic, ensuring higher quality training data.
Image-to-Image (I2I) Editing
This strategy is used to artificially increase the diversity of scenes in the synthetic data. By editing input images to generate new visual variations while keeping the underlying action structure intact, it allows the framework to create a wider range of realistic environmental settings for robot training.
Action-Preserving Video-to-Video (V2V) Transfer
This technique augments synthetic videos by changing their appearance without altering the actual motion dynamics. It helps generate more visually varied data while guaranteeing that the movement patterns remain consistent with the original intended actions, enhancing realism.
Attentive Probe
This is a mechanism used to verify motion consistency between two videos. It works by training a specialized probe on video encoders to detect subtle mismatches in motion. If the alignment probability between a generated video and its simulator replay exceeds a certain threshold, the action is deemed high quality.

Terminology

Summary

Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos <ref:2602. p. 018742#pg2> Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves <ref:2602. p. 018742#pg4>. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay <ref:2602. p. 018742#pg6>.

Framework Overview

RoboCurate is a unified framework that (1) generates synthetic robot data and (2) filters neural trajectory by quality via simulator-replay consistency, while supporting diverse scene and task generation <ref:2602.18742#pg7>. The core idea is to leverage a simulator to replay the generated action sequences, producing proxy robot videos with guaranteed correspondence to the generated actions <ref:2602. p. 118742#pg6>. By measuring visual motion consistency between these simulator-replayed videos and the synthesized videos, we can evaluate both the visual quality of the generated data and the correctness of action labels <ref:2602. p. 118742#pg6>.

Data Augmentation Strategies

The framework unlocks observation diversity beyond the available dataset via image-to-image editing and applies action-preserving video-to-video transfer to further augment appearance. Specifically, we employ (1) image-to-image (I2I) editing to substantially increase scene diversity, and (2) action-preserving video-to-video (V2V) transfer to augment appearance while preserving motion dynamics. p. 1 This pipeline generates high-quality data that reflects a wide range of realistic environmental variations. p. 1

Action Verification Mechanism

We propose an attentive probe on top of a frozen video encoder to solve the motion-consistency comparison problem between two videos, (wgen, wsim(aIDM)). We construct positive and negative pairs from real-world demonstrations T =

to train the probe to detect subtle motion mismatches. At inference time, we retain the sample only if the alignment probability p exceeds a threshold c, indicating the two videos are motion-consistent. p. 4

Performance and Results

RoboCurate yields substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos) and +16. p. 01% on DexMimicGen in the pre-training setup. p. 0 In the co-finetuning setup with real-robot evaluation, RoboCurate demonstrates a +179. p. 19% relative improvement in success rate. p. 1 Notably, RoboCurate further demonstrates strong out-ofdistribution generalization on the challenging real-world ALLEX humanoid dexterous manipulation environment, achieving a +162. p. 13% relative improvement on novel object pick-and-place tasks and enabling emergent success on novel action tasks (from 0. p. 20% to 25.0%).

Ablation and Analysis

Component-wise ablation shows that the attentive probe trained on V-JEPA2 yields the best VLA performance, outperforming other baselines. Our action-level filtering outperforms video-only physical plausibility filtering methods, demonstrating that verifying action quality is crucial for policy learning. p. 7 Furthermore, the effectiveness of our neural trajectory is not limited to a single embodiment; neural trajectory’s prior can be transferred across embodiments. p. 5

Conclusion

We present RoboCurate, a synthetic robot data generation framework that improves neural trajectory by (i) verifying IDM-predicted actions through simulator-replay consistency and (ii) expanding observation diversity via I2I editing and action-preserving V2V transfer. Our results show that increasing the visual diversity of neural trajectory is a key driver of downstream policy performance, and that our I2I and V2V pipeline yields more beneficial synthetic data than naive I2V generation. p. 8 Moreover, our action-consistency measure is effective both for filtering training data for policy learning and as a critic for Best-of-N sampling in video generation, consistently boosting performance across simulation and real-world benchmarks. p. 8

Impact Statement

This paper presents RoboCurate, a framework to expand the capabilities of robot policies using synthetic robot data. The framework suggests that a policy can learn novel actions present in synthetic robot data. p. 8 Careful consideration is required to ensure that synthetic robot data and resulting policies align with social values. p. 8

Acknowledgements

This work, including the ALLEX humanoid platform, was supported by RLWRLD.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved data generation pipeline for robot learning by introducing RoboCurate, a framework that evaluates action quality by comparing predicted actions in a simulator with simulation replay to measure motion consistency between the simulator rollout and the generated video. This allows for filtering physically implausible videos and ensuring correctness of action labels, leading to substantial relative improvements in success rates across various benchmarks.

  2. Enhanced observation diversity through a controllable pipeline that expands initial scenes using image-to-image editing and augments generated videos via action-preserving video-to-video transfer. This ensures the generated data reflects a wide range of realistic environmental variations by modifying scene visuals (table, objects, lighting) while preserving motion dynamics.

  3. Robust filtering strategy for neural trajectory by proposing an attentive probe on top of a frozen video encoder to verify IDM-predicted actions against simulator rollouts by measuring motion-consistency comparison between two videos. This mechanism allows the system to retain only samples where the alignment probability exceeds a threshold, effectively curating high-quality data without discarding samples.

  4. Improved data quality during inference through Best-of-N sampling, which uses the action consistency measure as a critic for the video generative model to select the video-action pair with the highest critic score. This strategy is particularly appealing in fine-tuning settings, where it improves the quality of neural trajectory without discarding samples, enabling more effective use of the generation framework in data-scarce settings.

  5. Superior performance on real-world tasks by leveraging co-finetuning on real data and curated neural trajectory, which demonstrates a +179.9% relative improvement in success rate on the challenging ALLEX humanoid dexterous manipulation setting compared to baseline methods. This indicates that action-level filtering is effective for curating neural trajectory and further enhances VLA performance, even with a smaller amount of data.

Sources

Related papers