DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

summary

The gist

DenseReward introduces a dense vision-language reward model for robotic manipulation that addresses the challenges of acquiring diverse failure data at scale and providing fine-grained reward signals…

In short

DenseReward creates a vision-language reward model for robotic manipulation by automatically generating diverse failure data. It predicts fine-grained, per-timestep rewards instead of sparse labels, allowing downstream reinforcement learning to learn better from task progress and common failure patterns.

Key concepts

Automated Data Generation
A pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling. It covers various mistakes like collisions or missed grasps by decomposing the manipulation into five phases (Reach, Grasp, Lift, Move, Place). This creates a large, diverse dataset for training.
Dense Reward Prediction
Instead of giving a single reward at the end of a task, DenseReward predicts a scalar reward score for every single timestep. This score measures how much progress has been made toward the goal at that exact moment, providing very fine-grained feedback on task completion.
Downstream RL Guidance
The predicted dense rewards are used to improve robot policies in different ways. In model predictive control (MPC), it guides action selection by minimizing distance to the target. For policy learning, it provides intermediate feedback during online reinforcement learning, helping the robot learn complex manipulation skills more effectively.

Terminology used across episodes

This episode discusses

The paper

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation · Read on arXiv

University of North Carolina at Chapel Hill University of North Carolina at Chapel Hill Carnegie Mellon University Shanghai Jiao Tong University Amazon Web Services

Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Existing approaches reuse successful trajectories to construct negative or suboptimal examples through trajectory truncation, instruction relabeling, or trajectory comparisons, without generating new trajectories where physical interaction failures such as collisions, missed grasps, object drops, and recovery actually occur. Meanwhile, many existing reward models predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without manual annotation, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation".

Dev: Here's the gist of the paper. DenseReward introduces a dense vision-language reward model for robotic manipulation that addresses the challenges of acquiring diverse failure data at scale and providing fine-grained reward signals for downstream policy learning The predicted rewards can be used in downstream reinforcement learning for policy improvement Automated Data Generation The authors develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors.

Rosa: This pipeline decomposes manipulation into five canonical phases: 1) Reach, 2) Grasp, 3) Lift, 4) Move, and 5) Place The automated pipeline uses a scene initialized with an object and a container at random positions to ensure diverse spatial configurations across episodes.

Dev: Phase boundaries are automatically detected from simulation state without manual annotation.

Rosa: How the training signal is built DenseReward predicts per-timestep dense reward scores, unlike prior models that predict sparse binary or trajectory-level rewards.

Title and authors: Rosa: Let's start by looking at the paper "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation" and who came up with it. The authors are Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang, Mingxiao Huo, Yao Mu, Huaxiu Yao, Li Erran Li Daniel Szafir and Mingyu Ding.

Dev: They come from a mix of places including UNC Chapel Hill and Carnegie Mellon University alongside some folks at Shanghai Jiao Tong University and Amazon AWS AI. It looks like a solid team with experience in both the lab setting and big-scale data infrastructure.

Taro: I'm curious about what this title actually implies for the field. "Dense Reward Learning via Failure Synthesis" suggests they are combining synthetic failure data with a dense reward approach to help train robots perform manipulation tasks.

Rosa: Exactly, it’s about solving that credit assignment problem in manipulation by synthesizing realistic failure scenarios and then training a model to give fine-grained rewards based on what the robot is actually doing at each moment.

Dev: It means we can teach policies much better because they get continuous guidance instead of just a big reward once the entire task is done. It points toward more sample efficient reinforcement learning.

Taro: If this works well, it could drastically cut down on how much time and physical wear-and-tear is needed to train a robot policy for complex tasks like stacking cups or putting a ball in a basket.

Rosa: That’s the real world implication; if we can get reliable feedback during training, the policies should be much more robust when they go out into the real world.

Dev: It sets up a lot of hope for using reinforcement learning to push vision language action models beyond what simple demonstration data can achieve, which is something we’ve been trying to do.

The paper's summary: Rosa: So, let's dig into the actual summary of "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". It explains that they curate a dataset of twenty-seven thousand episodes spanning both successful and failure trajectories across different tasks and scenes.

Dev: That dataset is what allows them to train DenseReward, which is a vision-language reward model that identifies the robot's current execution state and estimates task progress based on the language instruction and visual observations.

Taro: The summary highlights that this model predicts a per-timestep dense reward score, which is different from previous models because it gives feedback at every moment, not just once at the end.

Rosa: They are showing that this DenseReward model outperforms general purpose vision language models and existing robotic reward models in its ability to predict these dense rewards across both simulated and real-world manipulation.

Dev: The key result they highlight is that when you use DenseReward for model predictive control, it achieves the best overall performance with an average prediction error of zero point zero eight one one.

Taro: And they show this effect extends to online reinforcement learning using DSRL, where adding DenseReward boosts the success rate from forty percent up to eighty percent on tasks like stacking cups.

Rosa: So, in simple terms, the paper says that by synthesizing failure data and training a model that gives continuous progress feedback, you get a reward system that’s much more informative for teaching robots how to manipulate things than what we usually have.

The paper's improvements: Dev: Now let's look at the specific improvements they suggest in "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". They focus on three main areas of improvement.

Rosa: First, they improve dense reward prediction compared to general purpose vision language models and existing robotic reward models in both simulated and real-world manipulation settings.

Taro: Second, they show that this model provides effective guidance for model predictive control by predicting a per-timestep dense reward score that reflects task progress at the current time step.

Dev: That means in MPC, instead of just looking at the next action, you’re optimizing based on how close you are to finishing the task right now, which they say leads to actions that minimize the minimum distance between the end effector and the target object.

Rosa: And third, for policy learning using Proximal Policy Optimization or PPO fine-tuning on LIBERO, they show it provides useful dense reward guidance for online reinforcement learning and improves performance on most LIBERO suites.

Taro: The improvement in real-world online RL using DSRL is also significant; they report increasing the success rate from thirty percent to seventy percent on putting a ball in a basket when you add this DenseReward feedback.

Dev: So, the core improvement is that this failure-aware dense reward modeling provides actionable guidance for both closed-loop control and online policy optimization using standard RL algorithms.

Conclusion: Rosa: Alright, wrapping up the discussion on "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". The authors conclude that their method captures fine-grained task progress, partial completion, and common failure patterns by synthesizing successful and failure trajectories together.

Dev: They confirm that DenseReward outperforms general purpose vision language models and existing robotic reward models in dense reward prediction while providing useful reward guidance for downstream reinforcement learning.

Taro: What this really means is that DenseReward is a practical source of dense rewards for improving pretrained vision language action policies through online reinforcement learning.

Rosa: It’s a practical resource because they release the dataset, the trained reward models, and the evaluation suite to support building failure-aware dense reward modeling for robot learning.

Dev: I think we should also mention that their work points toward using this kind of feedback for more complex manipulation tasks in the future, like tool use or long-horizon tasks.

Taro: If they can extend this idea to those harder things, it could open up a whole new way to train robots for really intricate actions.

Rosa: Well, that's the gist of what we covered today on "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". We’ll be looking at some other papers next time.

More episodes

← Home