RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning".
Dev: Here's the gist of the paper. Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos <ref:2602.18742#pg2> Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves <ref:2602.18742#pg4>.
Rosa: To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay <ref:2602.18742#pg6>.
Dev: Framework Overview RoboCurate is a unified framework that (1) generates synthetic robot data and (2) filters neural trajectory by quality via simulator-replay consistency, while supporting diverse scene and task generation <ref:2602.18742#pg7>.
Rosa: The core idea is to leverage a simulator to replay the generated action sequences, producing proxy robot videos with guaranteed correspondence to the generated actions <ref:2602.18742#pg6>.
Paper summary: Rosa: So we've been talking about RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning. The authors are Seungku Kim, Suhyeok Jang, Byungjun Yoon, Dongyoung Kim, John Won, and Jinwoo Shin >
Dev: It seems the core of the work is moving beyond just generating videos and focusing on verifying the actions within those videos by checking them against a simulator replay to ensure physical consistency >
Rosa: And they achieve this while simultaneously increasing observation diversity through image-to-image editing for scenes and video-to-video transfer for appearances, which is what really lets the robot explore new things >
Dev: The implication here is that visual diversity of the neural trajectory matters a lot for how well the downstream policy performs, and that their method of filtering data based on motion consistency is effective both when training policies and when sampling from generated videos >
Rosa: It’s about creating synthetic robot data that's not just visually appealing but physically grounded, which is what helps policies learn novel actions more effectively >
Dev: It suggests that for building robust robot policies, we need a way to verify the action quality itself rather than just relying on visual checks or simulation alone >
Rosa: That’s the main point of RoboCurate. It gives us a system to generate data that is better for learning and deploying robots in the real world >
Conclusion: Rosa: So, RoboCurate is about taking synthetic robot data and actually checking if the actions in those videos make sense by comparing them to what a simulator predicts they should be doing.
Dev: Yeah, it's moving past just making pretty robot videos to making sure the movements are physically realistic according to a simulation.
Rosa: The authors, Kim and Jang and their team, they built this framework to generate data that has more variety in scenes and actions than just standard methods can give you.
Taro: They use image-to-image editing for scenes to get more visual differences, and video-to-video transfer for appearances to change how things look without messing up the actual motion.
Dev: That's the diversity part, but the real trick is this attention probe they put on top of a video encoder to check motion consistency between the generated video and the simulator replay.
Rosa: So, if that probe says they match well, then you know both the visuals and what they were doing are probably correct.
Taro: It sounds like a safety layer for policy learning; it's filtering out junk data that looks plausible but is physically impossible or just wrong.
Dev: The results show some big jumps in success rates, like a seventy point one percent increase on the GR-one Tabletop task when they used this method.
Rosa: And they even showed it works really well when you fine-tune it with data from actual robots, where the improvement is almost eighty percent.
Taro: What's interesting is that they tested it on a really hard environment, like the ALLEX humanoid dexterity task, and the robot got much better at picking and placing new objects.
Dev: It also showed they can even learn entirely new actions that weren't in their original training set, going from zero percent success to twenty-five percent success on novel tasks.
Rosa: So, this whole idea points to the fact that making the synthetic data more diverse isn't just about looking pretty; it directly helps the robot policy learn better actions.
Taro: It seems like verifying action quality is a really important step if you want your policy to actually work when it leaves the lab.
Dev: They also pointed out that their method works not just for one type of robot, but that the knowledge they get about how to generate good actions can be shared across different robot bodies.
Rosa: It's a framework for making synthetic data more trustworthy, which is crucial if we want to use it to train real robots.
KAIST
cs.RO, cs.AI, cs.CV
Submitted: 2026-02-21
Updated: 2026-11-05
Comments: CoRL 2026 G2A Workshop Accepted (Oral); Project page is available at https://seungkukim.github.io/robocurate/
Code: https://github.com/nvidia-cosmos/cosmos-predict2
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 100/100
The gist: Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline…
Key concepts
- Simulator Replay Consistency
- This is a core filtering technique where generated robot actions are replayed in a simulator to create proxy videos. By comparing these simulated videos with the original generated videos, researchers can identify and filter out actions that look physically inconsistent or unrealistic, ensuring higher quality training data.
- Image-to-Image (I2I) Editing
- This strategy is used to artificially increase the diversity of scenes in the synthetic data. By editing input images to generate new visual variations while keeping the underlying action structure intact, it allows the framework to create a wider range of realistic environmental settings for robot training.
- Action-Preserving Video-to-Video (V2V) Transfer
- This technique augments synthetic videos by changing their appearance without altering the actual motion dynamics. It helps generate more visually varied data while guaranteeing that the movement patterns remain consistent with the original intended actions, enhancing realism.
- Attentive Probe
- This is a mechanism used to verify motion consistency between two videos. It works by training a specialized probe on video encoders to detect subtle mismatches in motion. If the alignment probability between a generated video and its simulator replay exceeds a certain threshold, the action is deemed high quality.
Terminology
Summary
Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos <ref:2602. p. 018742#pg2> Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves <ref:2602. p. 018742#pg4>. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay <ref:2602. p. 018742#pg6>.
Framework Overview
RoboCurate is a unified framework that (1) generates synthetic robot data and (2) filters neural trajectory by quality via simulator-replay consistency, while supporting diverse scene and task generation <ref:2602.18742#pg7>. The core idea is to leverage a simulator to replay the generated action sequences, producing proxy robot videos with guaranteed correspondence to the generated actions <ref:2602. p. 118742#pg6>. By measuring visual motion consistency between these simulator-replayed videos and the synthesized videos, we can evaluate both the visual quality of the generated data and the correctness of action labels <ref:2602. p. 118742#pg6>.
Data Augmentation Strategies
The framework unlocks observation diversity beyond the available dataset via image-to-image editing and applies action-preserving video-to-video transfer to further augment appearance. Specifically, we employ (1) image-to-image (I2I) editing to substantially increase scene diversity, and (2) action-preserving video-to-video (V2V) transfer to augment appearance while preserving motion dynamics. p. 1 This pipeline generates high-quality data that reflects a wide range of realistic environmental variations. p. 1
Action Verification Mechanism
We propose an attentive probe on top of a frozen video encoder to solve the motion-consistency comparison problem between two videos, (wgen, wsim(aIDM)). We construct positive and negative pairs from real-world demonstrations T =
to train the probe to detect subtle motion mismatches. At inference time, we retain the sample only if the alignment probability p exceeds a threshold c, indicating the two videos are motion-consistent. p. 4
Performance and Results
RoboCurate yields substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos) and +16. p. 01% on DexMimicGen in the pre-training setup. p. 0 In the co-finetuning setup with real-robot evaluation, RoboCurate demonstrates a +179. p. 19% relative improvement in success rate. p. 1 Notably, RoboCurate further demonstrates strong out-ofdistribution generalization on the challenging real-world ALLEX humanoid dexterous manipulation environment, achieving a +162. p. 13% relative improvement on novel object pick-and-place tasks and enabling emergent success on novel action tasks (from 0. p. 20% to 25.0%).
Ablation and Analysis
Component-wise ablation shows that the attentive probe trained on V-JEPA2 yields the best VLA performance, outperforming other baselines. Our action-level filtering outperforms video-only physical plausibility filtering methods, demonstrating that verifying action quality is crucial for policy learning. p. 7 Furthermore, the effectiveness of our neural trajectory is not limited to a single embodiment; neural trajectory’s prior can be transferred across embodiments. p. 5
Conclusion
We present RoboCurate, a synthetic robot data generation framework that improves neural trajectory by (i) verifying IDM-predicted actions through simulator-replay consistency and (ii) expanding observation diversity via I2I editing and action-preserving V2V transfer. Our results show that increasing the visual diversity of neural trajectory is a key driver of downstream policy performance, and that our I2I and V2V pipeline yields more beneficial synthetic data than naive I2V generation. p. 8 Moreover, our action-consistency measure is effective both for filtering training data for policy learning and as a critic for Best-of-N sampling in video generation, consistently boosting performance across simulation and real-world benchmarks. p. 8
Impact Statement
This paper presents RoboCurate, a framework to expand the capabilities of robot policies using synthetic robot data. The framework suggests that a policy can learn novel actions present in synthetic robot data. p. 8 Careful consideration is required to ensure that synthetic robot data and resulting policies align with social values. p. 8
Acknowledgements
This work, including the ALLEX humanoid platform, was supported by RLWRLD.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved data generation pipeline for robot learning by introducing RoboCurate, a framework that evaluates action quality by comparing predicted actions in a simulator with simulation replay to measure
motion consistency between the simulator rollout and the generated video.
This allows for filteringphysically implausible videos
and ensuringcorrectness of action labels,
leading to substantial relative improvements in success rates across various benchmarks. -
Enhanced observation diversity through a controllable pipeline that expands initial scenes using image-to-image editing and augments generated videos via action-preserving video-to-video transfer. This ensures the generated data reflects a
wide range of realistic environmental variations
by modifying scene visuals (table, objects, lighting) while preserving motion dynamics. -
Robust filtering strategy for neural trajectory by proposing an attentive probe on top of a frozen video encoder to verify IDM-predicted actions against simulator rollouts by measuring
motion-consistency comparison between two videos.
This mechanism allows the system to retain only samples where the alignment probability exceeds a threshold, effectively curating high-quality data without discarding samples. -
Improved data quality during inference through Best-of-N sampling, which uses the action consistency measure as a critic for the video generative model to select
the video-action pair with the highest critic score.
This strategy is particularly appealing in fine-tuning settings, where itimproves the quality of neural trajectory without discarding samples,
enabling more effective use of the generation framework in data-scarce settings. -
Superior performance on real-world tasks by leveraging co-finetuning on real data and curated neural trajectory, which demonstrates a
+179.9% relative improvement in success rate
on the challenging ALLEX humanoid dexterous manipulation setting compared to baseline methods. This indicates thataction-level filtering is effective for curating neural trajectory and further enhances VLA performance, even with a smaller amount of data.
Sources
- Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control
- World Simulation with Video Foundation Models for Physical AI
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- VideoPhy: Evaluating Physical Commonsense for Video Generation
- Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
- AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems
- GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control
- OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- DreamGen: Unlocking Generalization in Robot Learning through Video World Models
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- OpenVLA: An Open-Source Vision-Language-Action Model
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
- Flow Matching for Generative Modeling
- Solving New Tasks by Adapting Internet Video Knowledge
- CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation