RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning

summary

The gist

Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline…

In short

RoboCurate is a framework that generates synthetic robot data and then filters it for quality by comparing generated actions with simulator replays. It uses image-to-image editing and video-to-video transfer to increase data diversity. This method significantly improves robot policy performance on real tasks by ensuring the synthetic data accurately reflects plausible physical movements.

Key concepts

Simulator Replay Consistency
This is a core filtering technique where generated robot actions are replayed in a simulator to create proxy videos. By comparing these simulated videos with the original generated videos, researchers can identify and filter out actions that look physically inconsistent or unrealistic, ensuring higher quality training data.
Image-to-Image (I2I) Editing
This strategy is used to artificially increase the diversity of scenes in the synthetic data. By editing input images to generate new visual variations while keeping the underlying action structure intact, it allows the framework to create a wider range of realistic environmental settings for robot training.
Action-Preserving Video-to-Video (V2V) Transfer
This technique augments synthetic videos by changing their appearance without altering the actual motion dynamics. It helps generate more visually varied data while guaranteeing that the movement patterns remain consistent with the original intended actions, enhancing realism.
Attentive Probe
This is a mechanism used to verify motion consistency between two videos. It works by training a specialized probe on video encoders to detect subtle mismatches in motion. If the alignment probability between a generated video and its simulator replay exceeds a certain threshold, the action is deemed high quality.

Terminology used across episodes

This episode discusses

The paper

RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning · Read on arXiv

KAIST

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning".

Dev: Here's the gist of the paper. Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos <ref:2602.18742#pg2> Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves <ref:2602.18742#pg4>.

Rosa: To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay <ref:2602.18742#pg6>.

Dev: Framework Overview RoboCurate is a unified framework that (1) generates synthetic robot data and (2) filters neural trajectory by quality via simulator-replay consistency, while supporting diverse scene and task generation <ref:2602.18742#pg7>.

Rosa: The core idea is to leverage a simulator to replay the generated action sequences, producing proxy robot videos with guaranteed correspondence to the generated actions <ref:2602.18742#pg6>.

Paper summary: Rosa: So we've been talking about RoboCurate: Harnessing Diversity with Action-Verified Synthetic Data for Robot Learning. The authors are Seungku Kim, Suhyeok Jang, Byungjun Yoon, Dongyoung Kim, John Won, and Jinwoo Shin >

Dev: It seems the core of the work is moving beyond just generating videos and focusing on verifying the actions within those videos by checking them against a simulator replay to ensure physical consistency >

Rosa: And they achieve this while simultaneously increasing observation diversity through image-to-image editing for scenes and video-to-video transfer for appearances, which is what really lets the robot explore new things >

Dev: The implication here is that visual diversity of the neural trajectory matters a lot for how well the downstream policy performs, and that their method of filtering data based on motion consistency is effective both when training policies and when sampling from generated videos >

Rosa: It’s about creating synthetic robot data that's not just visually appealing but physically grounded, which is what helps policies learn novel actions more effectively >

Dev: It suggests that for building robust robot policies, we need a way to verify the action quality itself rather than just relying on visual checks or simulation alone >

Rosa: That’s the main point of RoboCurate. It gives us a system to generate data that is better for learning and deploying robots in the real world >

Conclusion: Rosa: So, RoboCurate is about taking synthetic robot data and actually checking if the actions in those videos make sense by comparing them to what a simulator predicts they should be doing.

Dev: Yeah, it's moving past just making pretty robot videos to making sure the movements are physically realistic according to a simulation.

Rosa: The authors, Kim and Jang and their team, they built this framework to generate data that has more variety in scenes and actions than just standard methods can give you.

Taro: They use image-to-image editing for scenes to get more visual differences, and video-to-video transfer for appearances to change how things look without messing up the actual motion.

Dev: That's the diversity part, but the real trick is this attention probe they put on top of a video encoder to check motion consistency between the generated video and the simulator replay.

Rosa: So, if that probe says they match well, then you know both the visuals and what they were doing are probably correct.

Taro: It sounds like a safety layer for policy learning; it's filtering out junk data that looks plausible but is physically impossible or just wrong.

Dev: The results show some big jumps in success rates, like a seventy point one percent increase on the GR-one Tabletop task when they used this method.

Rosa: And they even showed it works really well when you fine-tune it with data from actual robots, where the improvement is almost eighty percent.

Taro: What's interesting is that they tested it on a really hard environment, like the ALLEX humanoid dexterity task, and the robot got much better at picking and placing new objects.

Dev: It also showed they can even learn entirely new actions that weren't in their original training set, going from zero percent success to twenty-five percent success on novel tasks.

Rosa: So, this whole idea points to the fact that making the synthetic data more diverse isn't just about looking pretty; it directly helps the robot policy learn better actions.

Taro: It seems like verifying action quality is a really important step if you want your policy to actually work when it leaves the lab.

Dev: They also pointed out that their method works not just for one type of robot, but that the knowledge they get about how to generate good actions can be shared across different robot bodies.

Rosa: It's a framework for making synthetic data more trustworthy, which is crucial if we want to use it to train real robots.

More episodes

← Home