DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

arXiv:2607.13033 · cs.RO · Submitted 2026-07-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation".

Dev: Here's the gist of the paper. DenseReward introduces a dense vision-language reward model for robotic manipulation that addresses the challenges of acquiring diverse failure data at scale and providing fine-grained reward signals for downstream policy learning The predicted rewards can be used in downstream reinforcement learning for policy improvement Automated Data Generation The authors develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors.

Rosa: This pipeline decomposes manipulation into five canonical phases: 1) Reach, 2) Grasp, 3) Lift, 4) Move, and 5) Place The automated pipeline uses a scene initialized with an object and a container at random positions to ensure diverse spatial configurations across episodes.

Dev: Phase boundaries are automatically detected from simulation state without manual annotation.

Rosa: How the training signal is built DenseReward predicts per-timestep dense reward scores, unlike prior models that predict sparse binary or trajectory-level rewards.

Title and authors: Rosa: Let's start by looking at the paper "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation" and who came up with it. The authors are Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang, Mingxiao Huo, Yao Mu, Huaxiu Yao, Li Erran Li Daniel Szafir and Mingyu Ding.

Dev: They come from a mix of places including UNC Chapel Hill and Carnegie Mellon University alongside some folks at Shanghai Jiao Tong University and Amazon AWS AI. It looks like a solid team with experience in both the lab setting and big-scale data infrastructure.

Taro: I'm curious about what this title actually implies for the field. "Dense Reward Learning via Failure Synthesis" suggests they are combining synthetic failure data with a dense reward approach to help train robots perform manipulation tasks.

Rosa: Exactly, it’s about solving that credit assignment problem in manipulation by synthesizing realistic failure scenarios and then training a model to give fine-grained rewards based on what the robot is actually doing at each moment.

Dev: It means we can teach policies much better because they get continuous guidance instead of just a big reward once the entire task is done. It points toward more sample efficient reinforcement learning.

Taro: If this works well, it could drastically cut down on how much time and physical wear-and-tear is needed to train a robot policy for complex tasks like stacking cups or putting a ball in a basket.

Rosa: That’s the real world implication; if we can get reliable feedback during training, the policies should be much more robust when they go out into the real world.

Dev: It sets up a lot of hope for using reinforcement learning to push vision language action models beyond what simple demonstration data can achieve, which is something we’ve been trying to do.

The paper's summary: Rosa: So, let's dig into the actual summary of "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". It explains that they curate a dataset of twenty-seven thousand episodes spanning both successful and failure trajectories across different tasks and scenes.

Dev: That dataset is what allows them to train DenseReward, which is a vision-language reward model that identifies the robot's current execution state and estimates task progress based on the language instruction and visual observations.

Taro: The summary highlights that this model predicts a per-timestep dense reward score, which is different from previous models because it gives feedback at every moment, not just once at the end.

Rosa: They are showing that this DenseReward model outperforms general purpose vision language models and existing robotic reward models in its ability to predict these dense rewards across both simulated and real-world manipulation.

Dev: The key result they highlight is that when you use DenseReward for model predictive control, it achieves the best overall performance with an average prediction error of zero point zero eight one one.

Taro: And they show this effect extends to online reinforcement learning using DSRL, where adding DenseReward boosts the success rate from forty percent up to eighty percent on tasks like stacking cups.

Rosa: So, in simple terms, the paper says that by synthesizing failure data and training a model that gives continuous progress feedback, you get a reward system that’s much more informative for teaching robots how to manipulate things than what we usually have.

The paper's improvements: Dev: Now let's look at the specific improvements they suggest in "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". They focus on three main areas of improvement.

Rosa: First, they improve dense reward prediction compared to general purpose vision language models and existing robotic reward models in both simulated and real-world manipulation settings.

Taro: Second, they show that this model provides effective guidance for model predictive control by predicting a per-timestep dense reward score that reflects task progress at the current time step.

Dev: That means in MPC, instead of just looking at the next action, you’re optimizing based on how close you are to finishing the task right now, which they say leads to actions that minimize the minimum distance between the end effector and the target object.

Rosa: And third, for policy learning using Proximal Policy Optimization or PPO fine-tuning on LIBERO, they show it provides useful dense reward guidance for online reinforcement learning and improves performance on most LIBERO suites.

Taro: The improvement in real-world online RL using DSRL is also significant; they report increasing the success rate from thirty percent to seventy percent on putting a ball in a basket when you add this DenseReward feedback.

Dev: So, the core improvement is that this failure-aware dense reward modeling provides actionable guidance for both closed-loop control and online policy optimization using standard RL algorithms.

Conclusion: Rosa: Alright, wrapping up the discussion on "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". The authors conclude that their method captures fine-grained task progress, partial completion, and common failure patterns by synthesizing successful and failure trajectories together.

Dev: They confirm that DenseReward outperforms general purpose vision language models and existing robotic reward models in dense reward prediction while providing useful reward guidance for downstream reinforcement learning.

Taro: What this really means is that DenseReward is a practical source of dense rewards for improving pretrained vision language action policies through online reinforcement learning.

Rosa: It’s a practical resource because they release the dataset, the trained reward models, and the evaluation suite to support building failure-aware dense reward modeling for robot learning.

Dev: I think we should also mention that their work points toward using this kind of feedback for more complex manipulation tasks in the future, like tool use or long-horizon tasks.

Taro: If they can extend this idea to those harder things, it could open up a whole new way to train robots for really intricate actions.

Rosa: Well, that's the gist of what we covered today on "DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation". We’ll be looking at some other papers next time.

University of North Carolina at Chapel Hill University of North Carolina at Chapel Hill Carnegie Mellon University Shanghai Jiao Tong University Amazon Web Services

cs.RO

Submitted: 2026-07-14

Updated: 2026-11-04

Comments: Website: https://dense-reward.github.io/

Project page: https://dense-reward.github.io/Teaser

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 96/100

The gist: DenseReward introduces a dense vision-language reward model for robotic manipulation that addresses the challenges of acquiring diverse failure data at scale and providing fine-grained reward signals…

Key concepts

Automated Data Generation
A pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling. It covers various mistakes like collisions or missed grasps by decomposing the manipulation into five phases (Reach, Grasp, Lift, Move, Place). This creates a large, diverse dataset for training.
Dense Reward Prediction
Instead of giving a single reward at the end of a task, DenseReward predicts a scalar reward score for every single timestep. This score measures how much progress has been made toward the goal at that exact moment, providing very fine-grained feedback on task completion.
Downstream RL Guidance
The predicted dense rewards are used to improve robot policies in different ways. In model predictive control (MPC), it guides action selection by minimizing distance to the target. For policy learning, it provides intermediate feedback during online reinforcement learning, helping the robot learn complex manipulation skills more effectively.

Terminology

Summary

DenseReward introduces a dense vision-language reward model for robotic manipulation that addresses the challenges of acquiring diverse failure data at scale and providing fine-grained reward signals for downstream policy learning The predicted rewards can be used in downstream reinforcement learning for policy improvement

Automated Data Generation

The authors develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. This pipeline decomposes manipulation into five canonical phases: 1) Reach, 2) Grasp, 3) Lift, 4) Move, and 5) Place The automated pipeline uses a scene initialized with an object and a container at random positions to ensure diverse spatial configurations across episodes. p. 3

Where the paper says it

Based on these phases, we design an automated pipeline for trajectory generation, as shown in Fig. The scene is first randomly initialized with an object and a container: the target object and the goal container are placed at a random position on the table, ensuring diverse spatial configurations across episodes.

Phase boundaries are automatically detected from simulation state without manual annotation. p. 3
Where the paper says it

Phase boundaries are automatically detected from simulation state: Grasp begins when the gripper contacts the object; Lift begins when the object is off the table; and Place begins when the end-effector enters a proximity radius dplace around the target. This requires no human annotation.

How the training signal is built

DenseReward predicts per-timestep dense reward scores, unlike prior models that predict sparse binary or trajectory-level rewards. The training dataset consists of 27k episodes with dense frame-level reward labels and failure mode annotations spanning diverse scenes and objects. p. 1

Where the paper says it

We assign per-timestep reward scores that reflect task progress throughout execution, capturing successful progress, partial completion, degradation after failure events, and recovery from temporary failures. Using this pipeline, we construct a dataset of 27k episodes with dense frame-level reward labels and failure mode annotations, spanning diverse scenes and objects from DROID, Isaac Sim, RoboSuite, and LIBERO.

DenseReward is trained on top of Qwen3-VL-4B-Instruct, finetuned to predict dense reward scores from visual observations and language instructions. p. 4
Where the paper says it

DenseReward Models. We build DenseReward on top of Qwen3-VL-4B-Instruct [62] and finetune it to predict dense reward scores from visual observations and language instructions.

The model outputs a scalar reward that measures task progress at the current timestep, retaining three decimal places for fine-grained differences in task progress. p. 4
Where the paper says it

Unlike prior reward models that predict trajectory-level binary success labels, DenseReward is trained with dense frame-level reward supervision. We retain three decimal places for reward values, allowing the model to learn fine-grained differences in task progress, such as moving closer to the target object, or recovering from a failure state.

Where the gains come from

DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. In model predictive control (MPC), DenseReward achieves the best overall performance with an average prediction error of 0. p. 4

Where the paper says it

1 Evaluating Reward Prediction We compare the dense reward prediction performance of DenseReward against a range of generalpurpose vision-language models and existing robotic reward models. 1, DenseReward achieves the best overall performance with an average prediction error of 0.

081, outperforming all baselines across all evaluated data sources. p. 4
Where the paper says it

1, DenseReward achieves the best overall performance with an average prediction error of 0. 081, outperforming all baselines across all evaluated data sources.

Furthermore, in real-world online RL using DSRL, adding DenseReward improves the success rate from 40% to 80% on stack the cups and from 30% to 70% on put ball in basket. p. 7
Where the paper says it

Adding DenseReward improves the success rate from 40% to 80% on stack the cups, and from 30% to 70% on put ball in basket. These results show that DenseReward provides effective dense feedback for real-world policy learning, enabling DSRL to improve the base policy with only a small number of costly real-world rollouts.

Downstream Applications

DenseReward provides effective reward guidance for model predictive control and reinforcement learning. In MPC, DenseReward achieves the lowest minimum distance between the object and the gripper position across three object manipulation tasks, suggesting it provides more effective guidance for selecting actions in closed-loop control. p. 5

Where the paper says it

2, DenseReward achieves the best average performance across the three objects, reducing the average minimum distance to 0. It outperforms RoboReward and VLAC, demonstrating that the predicted dense rewards provide more effective guidance for selecting actions in closed-loop control.

For policy learning with PPO fine-tuning on LIBERO, DenseReward provides useful dense reward guidance for online RL fine-tuning and improves performance on most LIBERO suites. p. 6
Where the paper says it

Following the official LIBERO evaluation protocol, each checkpoint is evaluated over 500 trials, with 50 trials per task across 10 tasks. DenseReward provides useful dense reward guidance for online PPO fine-tuning and improves the performance on most LIBERO suites.

In real-world experiments, adding DenseReward improves the success rate by providing effective dense feedback for real-world policy learning under limited rollout budgets. p. 7
Where the paper says it

Adding DenseReward improves the success rate from 40% to 80% on stack the cups, and from 30% to 70% on put ball in basket. These results show that DenseReward provides effective dense feedback for real-world policy learning, enabling DSRL to improve the base policy with only a small number of costly real-world rollouts.

Conclusion

DenseReward captures fine-grained task progress, partial completion, and common failure patterns through its synthesis of successful and failure trajectories. It outperforms generalpurpose VLMs and existing robotic reward models in dense reward prediction while providing useful reward guidance for downstream reinforcement learning. p. 7

Where the paper says it

DenseReward captures fine-grained task progress, partial completion, and common failure patterns. DenseReward outperforms generalpurpose VLMs and existing robotic reward models in dense reward prediction, and provides useful reward guidance for downstream reinforcement learning.

The authors release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning Future work includes extending this work to more complex manipulation such as tool use and long-horizon tasks. p. 1
Where the paper says it

We release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning. Keywords: Dense Reward Learning, Failure Synthesis, Robotic Manipulation 1 Introduction Robotic manipulation has seen remarkable progress through imitation learning on large-scale demonstration datasets [1, 2, 3, 4].

The paper concludes that DenseReward is a practical reward source for improving pretrained VLA policies through online reinforcement learning. p. 6
Where the paper says it

DenseReward often provides competitive or improved learning curves, suggesting that dense progress rewards can complement sparse task-completion signals during RL optimization. These results demonstrate the potential of DenseReward as a practical reward source for improving pretrained VLA policies through online reinforcement learning.

The authors hope their dataset, models, and evaluation suite will support future research on scalable reward learning for robotic manipulation. p. 7
Where the paper says it

DenseReward outperforms generalpurpose VLMs and existing robotic reward models in dense reward prediction, and provides useful reward guidance for downstream reinforcement learning. We hope our dataset, models, and evaluation suite will support future research on scalable reward learning for robotic manipulation.

References

[1] A. Brohan, N. Brown, J. Carbajal, Y. p. 8

Where the paper says it

References [1] A. Carbajal, Y.

Chebotar, J. p. 8
Where the paper says it

Carbajal, Y. Chebotar, J.

Dabis, C. Finn, K. Gopalakrishnan, K. p. 8
Where the paper says it

Chebotar, J. Gopalakrishnan, K.

Hausman, A Herzog, J Hsu et al Rt-1: Robotics transformer for real-world control at scale arXiv preprint arXiv:2212. p. 8
Where the paper says it

Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.

06817 1

[2] H R Walke K Black T Z Zhao Q Vuong C Zheng P Hansen-Estruch A W He V Myers M J Kim M Du et al Bridgedata v2: A dataset for robot learning at scale In Conference on Robot Learning pages 1723–1736 PMLR 2

[3] A O’Neill A Rehman A Maddukuri A Gupta A Padalkar A Lee A Pooley A Gupta A Mandlekar et al Open x-embodiment: Robotic learning datasets and rt-x models Open x-embodiment collaboration In 2024 IEEE International Conference on Robotics and Automation ICRA pages 6892–6903 IEEE

[4] Droid: A large-scale in-the-wild robot manipulation dataset arXiv preprint arXiv:2403.12945 1

[5] J Schulman F Wolski P Dhariwal A Radford and O Klimov Proximal policy optimization algorithms arXiv preprint arXiv:1707.06347 5

[6] Z Shao P Wang Q Zhu R Xu J Song X Bi H Zhang M Zhang Y Li Y Wu et al Deepseekmath: Pushing the limits of mathematical reasoning in open language models arXiv preprint arXiv:2402.03300 6

[7] K Black N Brown D Driess A Esmail M Equi C Finn N Fusai L Groom K Hausman B Ichter et al π0: A vision-language-action flow model for general robot control arXiv preprint arXiv:2410.24164 7

[8] M Kim K Pertsch S Karamcheti T Xiao A Balakrishna S Nair R Rafailov E Foster G Lam P Sanketi Q Vuong T Kollar B Burchfiel R Tedrake D Sadigh S Levine P Liang and C Finn Openvla: An open-source vision-language-action model arXiv preprint arXiv:2406.09246 8

[9] A Wagenmaker M Nakamoto Y Zhang S Park W Yagoub A Nagabandi A Gupta and S Levine Steering your diffusion policy with latent space reinforcement learning arXiv preprint arXiv:2506.15799 11

[10] P Intelligence A Amin R Aniceto A Balakrishna K Black K Conley G Connors J Darpinian K Dhabalia J DiCarlo et al π∗0.6: a vla that learns from experience arXiv preprint arXiv:2511. p. 8

Where the paper says it

6 : a vla that learns from experience. arXiv preprint arXiv:2511.

14759 12

[11] P Mahmoudieh D Pathak and T Darrell Zero-shot reward specification via grounded natural language In International Conference on Machine Learning pages 14743–14752 PMLR 3

[12] S Sontakke J Zhang S Arnold K Pertsch E Bıyık D Sadigh C Finn and L Itti Roboclip: One demonstration is enough to learn robot policies Advances in Neural Information Processing Systems 36:55681–55693 12

[13] Y Wang Z Sun J Zhang Z Xian E Biyik D Held and Z Erickson Rl-vlm-f Reinforcement learning from vision language foundation model feedback arXiv preprint arXiv:2402.03681 14

[14] W Zhao J Chen Z Meng D Mao R Song and W Zhang Vlmpc: Vision-language model predictive control for robotic manipulation arXiv preprint arXiv:2407.09829 15

[15] Y J Ma J Hejna C Fu D Shah J Liang Z Xu S Kirmani P Xu D Driess T Xiao et al Vision language models are in-context value learners In International Conference on Learning Representations volume 2025 pages 33984–34009 16

[16] H Tan S Chen Y Xu Z Wang Y Ji C Chi Y Lyu Z Zhao X Chen P Co et al Robodopamine: General process reward modeling for high-precision robotic manipulation arXiv preprint arXiv:2512.23703 17

[17] S Zhai Q Zhang T Zhang F Huang H Zhang M Zhou S Zhang L Liu S Lin and J Pang A vision-language-action-critic model for robotic real-world reinforcement learning arXiv preprint arXiv:2509.15937 18

[18] S Venkataraman Y Wang Z Wang N S Ravie Z Erickson and D Held Real-world offline reinforcement learning from vision language model feedback In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 13452–13459 IEEE

[19] S Chen C Harrison Y Lee A J Yang Z Ren L Ratliff J Duan D Fox and R Krishna Topreward: Token probabilities as hidden zero-shot rewards for robotics arXiv preprint arXiv:2602.19313 10

[20] A Liang Y Korkmaz J Zhang M Hwang A Anwar S Kaushik A Shah A S Huang L Zettlemoyer D Fox et al Robometer: Scaling general-purpose robotic reward models via trajectory comparisons arXiv preprint arXiv:2603.02115 10

[21] T Lee A Wagenmaker K Pertsch P Liang S Levine and C Finn Roboreward: Generalpurpose vision-language reward models for robotics arXiv preprint arXiv:2601.00675 9

[22] D Sadigh A Dragan S Sastry and S Seshia Active preference-based learning of reward functions 2017 13

[23] V Kumar E Todorov and S Levine Optimal control with learned local models Application to dexterous manipulation In 2016 IEEE International Conference on Robotics and Automation ICRA pages 378–383 IEEE

[24] A Rajeswaran V Kumar A Gupta G Vezzani J Schulman E Todorov and S Levine Learning complex dexterous manipulation with deep reinforcement learning and demonstrations In Proceedings of Robotics Science and Systems RSS 2018

[25] Y J Ma S Sodhani D Jayaraman O Bastani V Kumar and A Zhang Vip: Towards universal visual reward and representation via value-implicit pre-training arXiv preprint arXiv:2210.00030 13

[26] Y J Ma V Kumar A Zhang O Bastani and D Jayaraman Liv: Language-image representations and rewards for robotic control In International Conference on Machine Learning pages 23301–23320 PMLR 14

[27] J Duan W Pumacay N Kumar Y R Wang S Tian W Yuan R Krishna D Fox A Mandlekar and Y Guo Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation arXiv preprint arXiv:2410.00371 15

[28] J Rocamonde V Montesinos E Nava E Perez and D Lindner Vision-language models are zero-shot reward models for reinforcement learning In International Conference on Learning Representations volume 2024 pages 28446–28463 19

[29] H Zhang Z Zhuang H Zhao P Ding H Lu and D Wang Reinbot: Amplifying robot visual-language manipulation with reinforcement learning arXiv preprint arXiv:2505.07395 10

[30] Q Chen J Yu M Schwager P Abbeel Y Shentu and P Wu Sarm: Stage-aware reward modeling for long horizon robot manipulation arXiv preprint arXiv:2509.25358 11

[31] P Schroeder T Weng K Schmeckpeper E Rosen S Hart and O Biza Sole-r1: Videolanguage reasoning as the sole reward for on-robot reinforcement learning arXiv preprint arXiv:2603.28730 9

[32] A Mandlekar J Booher M Spero A Tung A Gupta Y Zhu A Garg S Savarese and L Fei-Fei Scaling robot supervision to hundreds of hours with roboturk Robotic manipulation dataset through human reasoning and dexterity In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems IROS pages 1048–1055 IEEE

[33] Z Ye W Lu M Ye T Lin S Yang J Yan and B Zhao Robofac: A comprehensive framework for robotic failure analysis and correction arXiv preprint arXiv:2505.12224 9

[34] K Wu C Hou J Liu Z Che X Ju Z Yang M Li Y Zhao Z Xu G Yang et al Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation In Robotics Science and Systems RSS 2025 Robotics: Science and Systems Foundation 2511

[35] C Glossop W Chen A Bhorkar D Shah and S Levine Cast Counterfactual labels improve instruction following in vision-language-action models arXiv preprint arXiv:2508.13446 17

[36] J Zhang Y Luo A Anwar S Sontakke J J Lim J Thomason E Biyik and J Zhang RewiND: Language-guided rewards teach robot policies without new demonstrations In 9th Annual Conference on Robot Learning 2508 URL https://openreview.net/forum?id=XjjXLxfPou

[37] Y Dai J Lee N Fazeli and J Chai Racer: Rich language-guided failure recovery policies for imitation learning 2025 IEEE International Conference on Robotics and Automation ICRA pages 15657–15664 URL https://api.semanticscholar.org/

[38] J Tobin R Fong A Ray J Schneider W Zaremba and P Abbeel Domain randomization for transferring deep neural networks from simulation to the real world In IEEE/RSJ International Conference on Intelligent Robots and Systems IROS 2017

[39] A Mandlekar Y Zhu A Garg L Fei-Fei and S Savarese Adversarially robust policy learning: Active construction of physically-plausible perturbations In IEEE/RSJ International Conference on Intelligent Robots and Systems IROS 2017

[40] P Pacaud R Garcia S Chen and C Schmid Scaling cross-environment failure reasoning data for vision-language robotic manipulation arXiv preprint arXiv:2512.01946 13

[41] T Haarnoja A Zhou P Abbeel and S Levine Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor In International conference on machine learning pages 1861–1870 Pmlr 2

[42] J Zhang Z Huang C Gu Z Ma and L Zhang Reinforcing action policies by prophesying arXiv preprint arXiv:2511.20633 9

[43] H Li Y Zuo J Yu Y Zhang Z Yang K Zhang X Zhu Y Zhang T Chen G Cui et al Simplevla-rl: Scaling vla training via reinforcement learning arXiv preprint arXiv:2509.09674 11

[44] C Li N Liu and K Yang Adaptive group policy optimization: Towards stable training and token-efficient reasoning arXiv preprint arXiv:2503.15952 12

[45] Y Guo J Zhang X Chen X Ji Y J Wang Y Hu and J Chen Improving vision-languageaction model with online reinforcement learning In 2025 IEEE International Conference on Robotics and Automation ICRA pages 15665–15672 IEEE

[46] P Intelligence B Ai A Amin R Aniceto A Balakrishna et al π0.7: a steerable generalist robotic foundation model with emergent capabilities arXiv preprint arXiv:2604. p. 8

Where the paper says it

Rl-vlm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.

15483 13

[47] A Research SOP: Scalable online post-training for general-purpose robots in the real world Technical Report 2026 URL https://agibot.com/research/sop en

[48] M Huo Y Zhang Y Wang T Tian X Zhang Y Xie C Xu P Ji W Zhan M Ding and M Tomizuka Sparse diffusion policy: A sparse, reusable, and flexible policy for robot learning In Conference on Robot Learning CoRL 2024

[49] Y Li X Ma J Xu Y Cui Z Cui Z Han L Huang T Kong Y Liu H Niu et al Gr-rl: Going dexterous and precise for long-horizon robotic manipulation arXiv preprint arXiv:2512.01801 13

[50] Y Chen X Li Rlrc Reinforcement learning-based recovery for compressed visionlanguage-action models arXiv preprint arXiv:2506.17639 14

[51] Y Wang R Yu F Zhang J Lu X Qin T Zhang K Wang and L Zhao Reinforcing vlas in task-agnostic world models arXiv preprint arXiv:2605.12334 14

[52] F Ebert C Finn S Dasari A Xie A Lee and S Levine Visual foresight: Model-based deep reinforcement learning for vision-based robotic control arXiv preprint arXiv:1812.00568 18

[53] D Hafner T Lillicrap J Ba and M Norouzi Dream to control Learning behaviors by latent imagination arXiv preprint arXiv:1912.01603 2

[54] A Escontrela A Adeniji W Yan A Jain X B Peng K Goldberg Y Lee D Hafner and P Abbeel Video prediction models as rewards for reinforcement learning In Advances in Neural Information Processing Systems 2023

[55] Y Guo L X Shi J Chen and C Finn Ctrl-world: A controllable generative world model for robot manipulation arXiv preprint arXiv:2510.10125 14

[56] B Peng W Zhang L Xu Z Qi J Zhang H Liu W Zeng X Jin Reworld: Multidimensional reward modeling for embodied world models arXiv preprint arXiv:2601.12428 14

[57] H He Y Zhang L Lin Z Xu and L Pan Pre-trained video generative models as world simulators In Proceedings of the AAAI Conference on Artificial Intelligence volume 40 pages 4645–4653 19

[58] Z Jiang S Zhou Y Jiang Z Huang M Wei Y Chen T Zhou Z Guo H Lin Q Zhang et al Wovr: World models as reliable simulators for post-training vla policies with rl arXiv preprint arXiv:2602.13977 14

[59] J Lv H Li J Li Y Nie F Kong Y Wang X Wang Z Zhu C Ni Q Deng et al Viva: A video-generative value model for robot reinforcement learning arXiv preprint arXiv:2604.08168 14

[60] H S Fang C Wang M Gou and C Lu Graspnet-1billion: A large-scale benchmark for general object grasping In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pages 11444–11453 2

[61] B Sundaralingam S K S Hari A Fishman C Garrett K Van Wyk V Blukis A Millane H Oleynikova A Handa F Ramos et al Curobo: Parallelized collision-free robot motion generation In 2023 IEEE International Conference on Robotics and Automation ICRA pages 8112–8119 IEEE 15

[62] S Bai Y Cai R Chen K Chen X Chen Z Cheng L Deng W Ding C Gao C Ge W Ge Z Guo Q Huang J Huang F Huang B Hui S Jiang Z Li M Li M Li K Li Z Lin J Lin X Liu J Liu C Liu Y Liu D Liu S Liu D Lu R Luo C Lv R Men L Meng X Ren X Ren S Song Y Sun J Tang J Tu J Wan P Wang P Wang Q Wang Y Wang T Xie Y Xu H Xu J Xu Z Yang M Yang J Yang A Yang B Yu F Zhang H Zhang X Zhang B Zheng H Zhong J Zhou F Zhou Y Zhu and K Zhu Qwen3-vl technical report arXiv preprint arXiv:2511.21631 9

[63] C Clark J Zhang Z Ma J S Park M Salehi R Tripathi S Lee Z Ren C D Kim Y Yang et al Molmo2: Open weights and data for vision-language models with video understanding and grounding arXiv preprint arXiv:2601.10611 13

[64] C Yu Y Wang Z Guo H Lin S Xu H Zang Q …

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated data generation with dense rewards allows for training on physically realistic failure trajectories without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. This enables fine-grained estimation of task progress throughout an episode rather than sparse binary success/failure labels.

  2. DenseReward models can be used in downstream model predictive control (MPC) to guide action selection by predicting a per-timestep dense reward score that reflects task progress at the current timestep, leading to actions that minimize the minimum distance between the end effector and the target object.

  3. Reinforcement policy learning with DenseReward enables online fine-tuning of policies using Proximal Policy Optimization (PPO) by combining simulator success signals with dense intermediate rewards, such as in equation one: rt = α · rsim t + β · rmodel t. This provides informative feedback for policy optimization over long-horizon manipulation tasks.

  4. Real-world online reinforcement learning with DSRL can improve policies under limited rollout budgets by integrating DenseReward to provide intermediate feedback at action chunks, leading to improved success rates on real-world manipulation tasks, such as increasing the success rate from 40% to 80% on stack the cups.

  5. The system can be equipped with a mechanism that reasons about the current execution state (correct or failure mode) and predicts a scalar reward that estimates task progress at the current timestep given a language instruction, visual observation, and historical frames.

Abstract

Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Existing approaches reuse successful trajectories to construct negative or suboptimal examples through trajectory truncation, instruction relabeling, or trajectory comparisons, without generating new trajectories where physical interaction failures such as collisions, missed grasps, object drops, and recovery actually occur. Meanwhile, many existing reward models predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without manual annotation, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning.

Sources

Related papers