Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation

arXiv:2606.13040 · cs.RO · Submitted 2026-06-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation".

Dev: The gist The paper introduces RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation, which decomposes capability into 12 diagnostic question families to test how models judge ongoing execution.

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So, wrapping up this look at "Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation", we're talking about how vision language models can start judging robotic actions based on what’s happening right now during a task instead of just looking at if it finished successfully.

Dev: Right, so the authors are presenting this framework that uses real robotic data to build supervision by breaking long manipulation episodes down into local process units.

Taro: They built this diagnostic benchmark called RoboProcessBench that tests models across twelve question families covering static monitoring and dynamic reasoning.

Rosa: Those twelve families cover everything from just identifying the current phase of the task, like contact detection, to figuring out temporal order and how primitives connect.

Dev: It's important to note that they split those questions into two main groups: static monitoring for what’s happening in this exact moment, and dynamic reasoning for predicting motion or progress over time.

Taro: They showed that the understanding isn't consistent across all those twelve areas; some tasks are just harder for the models right now than others.

Rosa: The dataset they created, ProcessData, has fifty-eight thousand question answer pairs from real manipulation tasks to train and test this idea.

Dev: And using that data, specifically ProcessData-SFT supervision, really helps models get better at recognizing those local states and motions when they're trained on it.

Taro: It means we can start building AI critics for robots that actually monitor if the manipulation is unfolding correctly in real-time based on physical signals.

Rosa: That shifts our focus away from just final success rates to monitoring intermediate action states during the process itself.

Dev: The authors themselves stress that this benchmark isn't a closed-loop policy evaluation tool; it’s a tool to help us develop better critics and evaluators for monitoring consistency.

Taro: They also flag that they need more work on richer video inputs and external progress-ranking validation to truly test these systems in the real world.

Rosa: That means the challenge now is moving from controlled traces to seeing how these models handle messy, unpredictable real-world scenarios where things go wrong.

Conclusion: Rosa: So we’re wrapping up this look at Robo2Process, which is about how vision language models can start judging robotic actions based on what’s happening right now during a task instead of just checking if it finished successfully.

Dev: Right, so the authors are showing that vision language models need to understand the entire execution path of a manipulation, not just the final result.

Taro: They built this framework by breaking long manipulation episodes into local process units and linking those visual observations to the source signals we have from robot data.

Dev: And they created a diagnostic benchmark called RoboProcessBench that splits everything into twelve question families covering static monitoring and dynamic reasoning.

Rosa: So, static monitoring is about checking the current state of a robot right now, while dynamic reasoning tests if the model can predict motion or understand temporal order based on what’s happening.

Taro: They found that this understanding isn't uniform across all those twelve families; some parts are much harder for the models to get right than others.

Dev: The key takeaway is that they created ProcessData, a dataset with 58k question answer pairs from real manipulation tasks to train and test these models.

Rosa: And using that data, specifically the ProcessData-SFT supervision, substantially improves models on recognizing local states and motions across different vision language models.

Taro: It means we can start building AI critics that monitor whether the robot is making correct intermediate decisions based on physical signals over time.

Dev: The authors are clear, though, that this benchmark isn't a closed-loop policy evaluation tool; it’s a way to help us develop better critics and evaluators for monitoring consistency.

Rosa: They also point out they need more work on richer video inputs and external progress-ranking validation to truly test these systems in the real world.

Taro: That means the challenge now is moving from controlled traces to seeing how these models handle messy, unpredictable real-world scenarios where things go wrong.

Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University · China University of Mining Technology

cs.RO

Submitted: 2026-06-11

Updated: 2026-10-15

Project page: https://aistairc.github.io/aist_bimanip_site

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: The gist The paper introduces RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation, which decomposes capability into 12 diagnostic question families

Key concepts

RoboProcessBench: A diagnostic benchmark designed to evaluate vision-language models' ability to judge the execution of robotic manipulation. It tests specific skills like monitoring the current state (static) and inferring future motion or progress (dynamic) by asking targeted questions about the process.
Local Process Units (LPUs): Long robotic episodes are broken down into smaller, temporally localized units called LPUs. Each LPU represents a coherent sub-progress of the manipulation task, allowing researchers to focus on understanding how models track small segments of an action.
Static Monitoring vs. Dynamic Reasoning: These are two main dimensions used to categorize the benchmark questions. Static monitoring checks if a model can identify the current local process state (e.g., which phase or contact is happening now). Dynamic reasoning tests more complex abilities, such as predicting future motion, determining temporal order, or inferring the eventual outcome of a sequence.

Terminology

Benchmark — 3× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Generator — 1× in this paper · explained in 2 episodes
Trace — 1× in this paper · explained in 2 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist The paper introduces RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation, which decomposes capability into 12 diagnostic question families to test how models judge ongoing execution.

RoboProcessBench Decomposition

RoboProcessBench decomposes such capability into two complementary dimensions: static monitoring and dynamic reasoning, instantiated as 12 diagnostic question families covering phase, contact, motion, coordination, primitive-local progress, temporal order, outcome, and primitive-level transitions <ref:2606.13040#pg4>. This benchmark evaluates the VLM-side abilities required for process judging rather than action generation or closed-loop policy success <ref:2606.13040#pg3>. The resulting ProcessData contains ~58k question-answer pairs across 260 manipulation tasks, which is further split into ProcessData-SFT and ProcessData-Eval for post-training and evaluation purposes <ref:2606.13040#pg2>.

Diagnostic Question Families

The 12 task families are organized into two complementary categories: Static Monitoring evaluates whether a VLM can identify the current local process state, while Dynamic Reasoning evaluates whether it can infer motion, progress, temporal relation, outcome, or primitive-level continuation <ref:2606.13040#pg6>. Static monitoring families include phase recognition (T1), contact detection (T2), bimanual coordination state recognition (T4), and current primitive recognition (T10) <ref:2606.13040#pg6>. Dynamic reasoning families require temporal, predictive, progresscentric, or chain-level inference, including motion direction prediction (T3), primitive-local progress recognition (T5), motion state recognition (T6), operation outcome prediction (T7), temporal ordering (T8), temporal priority prediction (T9), next primitive prediction (T11), and primitive chain restoration (T12) <ref:2606.13040#pg6>.

Benchmark Construction and Supervision

A key design choice of RoboProcessBench is that long manipulation episodes are decomposed into local process units (LPUs): U(e) = ⋃ uj for j=1, uj = [t s j, t e j], ts j. Each LPU is a temporally localized decision unit corresponding to a coherent manipulation subprogress <ref:2606.13040#pg5>. A benchmark item for task family m is defined as x(m) i = gi, oi, q(m) i, y(m) i <ref:2606.13040#pg6>. The construction pipeline follows four regimes: T1, T2, T4, and T6 are built from stable state, contact, or motion regimes; T3, T8, and T9 are built from motion and timestamp structure; T5 and T7 are built from local subprogress and eventual outcome signals; and the remaining families (T10–T12) are instantiated from explicit primitive annotations when available <ref:2606.13040#pg6>.

Evaluation Results

Extensive evaluation of various VLMs on ProcessData-Eval reveals broad limitations across 12 diagnostic task families, suggesting current models still lack robust process-aware understanding of manipulation executions <ref:2606.13040#pg2>. Diagnostic results show that related probes cluster strongly (T4/T6, ρ = 0.844; T10–T12, ρ = 0.635–0.737), whereas temporal ordering, temporal priority, primitive restoration, and progress estimation expose separable failure modes <ref:2606.13040#pg9>. Local process-state grounding is easier than local progress estimation <ref:2606.13040#pg9>. The strongest zero-shot result on T5 remains difficult, reaching only 34.4%, close to the 33.3% random baseline <ref:2606.13040#pg9>.

Effect of ProcessData-SFT Supervision

ProcessData-SFT provides a broadly learnable supervision signal across model backbones, substantially improving over their corresponding zero-shot backbones and achieving strong performance on most local state, motion, and primitive-aware tasks <ref:2606.13040#pg10>. The gains are especially consistent on phase recognition (T1), contact detection (T2), bimanual coordination (T4), motion direction prediction (T3), motion-state recognition (T6), current primitive recognition (T10), next primitive prediction (T11), and primitive-chain restoration (T12) <ref:2606.13040#pg10>. Compared with their corresponding base backbones, ProcessData-SFT improves static monitoring by 42.7 and 44.4 points, reaching 77.2% for both models <ref:2606.13040#pg10>. Dynamic reasoning also improves substantially, by 23.6 and 21.4 points, reaching 68.9% and 69.5% <ref:2606.13040#pg10>. The improvement on primitive progress recognition (T5) is particularly important, as ProcessDataSFT-Qwen reaches 45.4% and ProcessData-SFT-Intern reaches 45.3%, improving by roughly 11 points over the strongest zero-shot baseline <ref:2606.13040#pg10>.

Conclusion

RoboProcessBench provides not a replacement for closed-loop policy evaluation, but a capability substrate for developing VLM-based progress-aware critics and process evaluators that can better monitor whether robotic manipulation is unfolding correctly, locally progressing, temporally consistent, and compatible with primitive-level continuation <ref:2606.13040#pg10>. The benchmark decomposes the VLM-side abilities needed by judge-like robotic modules into 12 diagnostic task families across static monitoring and dynamic reasoning <ref:2606.13040#pg8>. The paper demonstrates that many process signals are learnable from structured execution traces, making it useful for supervised adaptation and diagnosing unresolved temporal and outcome-oriented reasoning bottlenecks <ref:2606.13040#pg10>.

Limitations

RoboProcessBench is a diagnostic benchmark and learnable substrate for fine-grained process-aware understanding, not a closed-loop policy benchmark <ref:2606.13040#pg8>. Its multiple-choice QA scores should not be interpreted as direct evidence of downstream policy success or deployable reward quality <ref:2606.13040#pg8>. Future work should extend RoboProcessBench to richer video inputs, more embodiments and viewpoints, failure-rich trajectories, external progress-ranking validation, and closed-loop robotic evaluation <ref:2606.13040#pg8>.

Improvements for AI systems

RoboProcessBench decomposes such capability into two complementary dimensions, static monitoring and dynamic reasoning, allowing VLMs to be evaluated on phase, contact, motion, coordination state recognition (T1-T4) and motion direction prediction (T3), primitive-local progress recognition (T5), operation outcome prediction (T7) (dynamic reasoning tasks).
The improved AI system can act as a visual critic or online reward generator by judging the physical and temporal progression of manipulation, moving beyond just final task success.
Post-training with ProcessData-SFT enables VLMs to show consistent gains on local state, motion, progress, and primitive-aware cues, specifically improving tasks like phase recognition (T1), contact detection (T2), bimanual coordination (T4), motion direction prediction (T3), motion state recognition (T6), current primitive recognition (T10) as noted in Table 3.
The system gains the ability to perform primitive-level transitions and primitive chain restoration by leveraging explicit annotations from sources like REASSEMBLE, as these tasks are explicitly included in the benchmark taxonomy.
The AI can be diagnosed for weaknesses; results show that primitive-local progress recognition (T5) remains difficult, highlighting a specific bottleneck that requires further model development or supervision to solve.

Abstract

Vision-language models (VLMs) used as robotic critics, reward generators, and failure detectors must understand how manipulation unfolds, not only whether it succeeds. Yet fine-grained process supervision is not readily available in a unified form, and existing evaluations do not systematically diagnose these abilities. We introduce Robo2Process, a trace-grounded framework for scalable construction of process-aware supervision from heterogeneous robotic execution data. It aligns visual observations with source-native signals and annotations, extracts local process units (LPUs), and generates structured QA supervision through task-specific sampling, deterministic label construction, and ambiguity filtering. We instantiate this framework as RoboProcessBench, a diagnostic benchmark with 12 question families spanning static monitoring and dynamic reasoning, and ProcessData, a corpus of 57,892 QA pairs across four data sources and 260 manipulation tasks, split into ProcessData-SFT and ProcessData-Eval. Evaluation of a broad suite of VLMs reveals uneven process understanding. Supervision from ProcessData-SFT improves Qwen2.5-VL-7B and InternVL-3-8B on local state, motion, progress, and primitive-aware cues. External evaluation on RoboRewardBench shows Robo2Process enables out-of-distribution process-aware reasoning abilities. Overall, Robo2Process provides a source-aware pipeline, diagnostic benchmark, and learnable substrate for process-aware understanding in robotic manipulation.