ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

arXiv:2610.12089 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks".

Rosa: The gist The work introduces ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities,

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So we’re looking at "ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks," which essentially sets up the whole infrastructure—that dataset of one hundred thirty-seven thousand eight hundred ninety-nine segments and the benchmark with its controlled tests—to let us break down these complex long-horizon tasks into smaller, manageable skill units <ref:2610.12089#pg2>.

Dev: The main thing they prove is that when you treat these operations as distinct skills and pair them with explicit subtask instructions, it really improves local manipulation success if you train a skill policy instead of just using a general task policy on complete demonstrations <ref:2610.12089#pg3>.

Taro: And the real power comes when you combine that with a planner that decides exactly when and how to use those skills while the robot is navigating around <ref:2610.12089#pg3>. This coordination raises the full-task success rate up from four point zero percent to eighteen point zero percent on activities like turning on a microwave or picking up trash, which is a substantial gain for complex embodied AI systems <ref:2610.12089#pg3>.

Rosa: It means we can train these skills in isolation and then rely on a planner to stitch them together for much better overall performance in those multi-step scenarios.

Dev: The paper also highlights how sensitive the execution is to starting conditions, showing that joint perturbations can reduce success rates by approximately fifty-six percent compared to those from demonstrated starting states <ref:2610.12089#pg3>.

Taro: The authors suggest future work should focus on improving navigation by training a task policy and refining the planner's ability to select which subtask instruction to use next <ref:2610.12089#pg3>.

Rosa: So, in simple terms, ManiUnit gives us the data and the framework to train better individual actions, and then shows how combining those actions intelligently with planning leads to much better results for long sequences.

Conclusion: Rosa: We've been digging into "ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks," which is this dataset and benchmark built from those fifty BEHAVIOR activities, and now we’re getting to the end of what it all means for mobile robotics <ref:2610.12089#pg1>.

Dev: Yeah, I remember thinking about how they structured this whole thing, treating each individual operation as a skill unit specifically for training and evaluation <ref:2610.12089#pg3>.

Taro: It really is neat how they tackled those three core problems: skill ambiguity, where similar observations make choosing the right action hard; state mismatch, where the next step doesn't start from where the last one left off; and task metrics that just don't help diagnose which specific skill failed <ref:2610.12089#pg1>.

Rosa: And what ManiUnit does to solve that is attaching an explicit subtask instruction to every single segment, which keeps the intended operation clear during learning and testing <ref:2610.12089#pg2>.

Dev: The construction of this data—the one hundred thirty-seven thousand eight hundred ninety-nine segments across twenty-one skill types—is really detailed; they show how they extracted those actions and then refined the temporal boundaries using joint departure from a reference posture <ref:2610.12089#pg2>.

Taro: Their benchmark itself is interesting too, with those test instances where they manually define a local goal, like a BDDL goal, so you can evaluate each skill independently without running the whole long task <ref:2610.12089#pg3>.

Rosa: And when we look at the results on those two long-horizon activities—turning on a microwave or picking up trash—the comparison between training just a task policy and training a skill policy is pretty stark <ref:2610.12089#pg3>.

Dev: We saw local manipulation success jump from about forty-nine point three percent for the task policy to nearly seventy-eight point seven percent when trained specifically on these segments <ref:2610.12089#pg3>.

Taro: And that whole combined system, where you have the task planner deciding when to invoke that more reliable skill policy, actually boosted full-task success up to eighteen point zero percent <ref:2610.12089#pg3>.

Rosa: It really shows that skill-level training helps with the fine motor skills, but coordinating those skills through a planner is what gets you doing the whole sequence successfully <ref:2610.12089#pg3>.

Dev: The authors are suggesting future work should focus on improving navigation when the robot's starting position or joint configuration is off—that sensitivity to perturbation was a big finding <ref:2610.12089#pg3>.

Taro: Exactly. They’ve shown we need to keep looking at how the system handles things when the world doesn't behave exactly as expected during those long sequences <ref:2610.12089#pg3>.

Rosa: So, ManiUnit gives us a solid foundation for training skills and composing them better, but the next step is figuring out how to make that coordination more robust when things go wrong.

Nanjing University of Science and Technology

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The work introduces ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities, to address challenges in learning and evaluating long-horizon mobile

Key concepts

Skill Ambiguity Challenges: When a robot performs a sequence of actions, similar observations under one instruction can make it hard to know which specific skill to use next. Also, the state inherited from one action might not match what the next action expects, and focusing only on overall task success hides problems in individual skills.
ManiUnit Dataset Construction: This dataset organizes manipulation segments into skill types and subtasks, each paired with an explicit instruction detailing the operation, objects, and context. Segments are constructed by combining supporting actions with the main operation and refining temporal boundaries based on joint posture changes.
Policy Composition for Long-Horizon Tasks: The paper suggests a two-policy system: a task policy handles navigation and transitions, while a skill policy handles specific manipulation stages. A planner coordinates these policies to decide when to switch from navigation to manipulation, significantly boosting overall success rates.

Terminology

Benchmark — 33× in this paper · explained in 2 episodes
Robustness — 4× in this paper · explained in 5 episodes
Benchmarking — 3× in this paper · explained in 4 episodes
Foundation Model — 3× in this paper · explained in 3 episodes
Transformer — 3× in this paper · explained in 2 episodes
Aggregation — 2× in this paper
Consistency — 2× in this paper · explained in 3 episodes
Embodied AI — 2× in this paper · explained in 2 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Foundation Models — 2× in this paper · explained in 7 episodes
Generalization — 2× in this paper · explained in 13 episodes
Instruction Tuning — 2× in this paper · explained in 3 episodes
Adaptation — 1× in this paper · explained in 1 episode
Affordance — 1× in this paper · explained in 1 episode
Grounding — 1× in this paper · explained in 3 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
VLA Models — 1× in this paper · explained in 3 episodes
Vision-Language-Action (VLA) Models — 1× in this paper · explained in 4 episodes

Summary

The gist The work introduces ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities, to address challenges in learning and evaluating long-horizon mobile manipulation tasks by treating individual operations as units of training and evaluation

Skill Ambiguity Challenges

The paper identifies three key challenges in long-horizon mobile manipulation: similar observations under a fixed task instruction may make skill selection ambiguous, even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution, and task-level metrics hinder skill-specific diagnosis ManiUnit addresses these challenges by attaching an explicit subtask instruction to each segment to keep the intended operation explicit during learning and evaluation

ManiUnit Dataset Construction

ManiUnit comprises a dataset of manipulation segments paired with explicit subtask instructions, organized into skill types and subtasks, containing 137,899 segments across 21 skill types and 417 subtasks The dataset distinguishes three levels of organization: a skill type identifies an operation category, a subtask is defined by an instruction specifying the operation, objects, and spatial context, and a segment records one demonstrated execution of that subtask Segment construction involves rule-based extraction and merging to combine supporting actions with the main operations they serve when source annotations record them separately Temporal boundary refinement locates the manipulation phase within each extracted interval using the armposture departure defined in Section 2.2, retaining continuous intervals based on joint departure from a reference posture

ManiUnit Benchmark Construction

The benchmark consists of 1,260 test instances drawn from 20 operation starts per skill type under three starting-state conditions: Original, Base Perturbation, and Joint Perturbation For each selected subtask, a local BDDL goal is manually defined to specify the object states and relations required to complete the manipulation operation Test instances are constructed by restoring intermediate simulator states from the source demonstration and then perturbing the robot’s base position or joint configuration while keeping the subtask instruction and local goal fixed

Policy Composition for Long-Horizon Tasks

The paper suggests a simple composition where a task policy acts as the default controller for navigation and transitions, while a skill policy delegates selected manipulation stages to perform operations more reliably than the task policy A planner coordinates this system by deciding when to invoke the skill policy and which subtask instruction to supply, using robot-state histories to estimate whether the robot is navigating or manipulating This combined system increases overall full-task success from 4.0% to 18.0% on two long-horizon activities like Turning On Radio and Picking Up Trash

Evaluation of Policy Performance

Evaluations show that similar aggregate scores can hide substantial differences in per-skill performance, and the tested starting-state perturbations degrade execution, with joint perturbations reducing success rates by approximately 56% relative to those from demonstrated starting states Skill policy training on ManiUnit segments achieves 78.7% local manipulation success on two long-horizon activities compared to 49.3% for a task policy trained on complete demonstrations Furthermore, the combined system improves full-task success by coordinating the task and skill policies through a planner, raising it from 4.0% to 18.0%

Conclusion

ManiUnit supports skill-level training that improves local manipulation success and policy composition that improves complete-task execution on two long-horizon activities Future work can focus on improving navigation through task-policy training and coordination through planner design, including task-progress estimation, subtask selection, and policy switching The paper concludes by demonstrating that ManiUnit allows for independent skill evaluation and the combination of skill-level training with policy composition leads to better overall performance in long-horizon tasks.

--- Page 1 ---

Preprint.

MANIUNIT: A MANIPULATION SKILL DATASET AND BENCHMARK FOR LONG-HORIZON TASKS Guoting Wei1,3 Dawei Yan2 Xia Yuan1 Gengming Zhang3 Yelin He3 Guodong Du3 Jiaquan Ye3 Heng Zhang1 Xinming Wei3 Xianbiao Qi3 Chunxia Zhao1 Haokui Zhang2,3† Rong Xiao3 1 Nanjing University of Science and Technology 2 Northwestern Polytechnical University 3 Intellifusion ①Skill Ambiguity Challenges ②Starting-state Mismatch ③Evaluation Opacity Task: Make microwave popcorn VS Next: Open Door or Turn On? Similar views, different next skills ManiUnit Data Statistics Pick Up Place Pour Open Door Open Lid Spray Chop Turn On Switch Skill Organization Pick Up subtasks Pour subtasks Inherited states affect next skill ①Explicit Subtask ②Controlled Starting-State Tests ③Independent Skill Evaluation Original Base Joint No preceding skills required Explicit guidance Fail Open Pick Place Close Turn on Task Observation Robot state Planner Task policy Task instruction Skill policy Subtask instruction Local manipulation success (%) Task policy Skill policy Overall 49.3% → 78.7% 60% 84% 56% 76% 32% 76% Original Base Joint Full-task success (%) Long-task execution Fail Suc... radio from table.. cap from counter.. game from bed.. bacon into pan.. cheese onto dough. Open Door Turn On Switch Turn on the microwave switch. Measure starting-state sensitivity Early failures leave later skills untested 4% 18% Task policy Combined system Close Door start Outcome 50 Source activities 21 Skill types 417 Subtasks 137,899 Training segments 1,260 Test instances??? Open Pick Place Close Turn on Instructions Close Door (illustrative) Full-task rollout Untested skills Restore state Restored start Local goal (BDDL) Close the microwave door Door closed Open the microwave door. Figure 1: ManiUnit overview. Left: Long-horizon execution poses three challenges for skill learning and evaluation. Center: Correspondingly, ManiUnit provides explicit subtask instructions to guide manipulation. Controlled tests measure starting-state sensitivity, while restored states and local BDDL goals enable independent skill evaluation. Right: ManiUnit organizes manipulation segments by skill type and subtask. Bottom: In a focused study on two long-horizon activities, skill-level training improves local manipulation success, and combining the task and skill policies through a planner improves full-task success. ABSTRACT Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of interdependent manipulation skills under a single natural language instruction. Learning and evaluating these skills present three challenges: similar observations under a fixed task instruction may make skill selection ambiguous; even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution; and task-level metrics hinder skill-specific diagnosis, while early failures leave later skills untested Equal contribution. This work was done during Guoting Wei’s internship at IntelliFusion 1 arXiv:2610.12089v1 [cs.RO] 8 Oct 2026

--- Page 2 ---

skills untested. We therefore introduce ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities. Its dataset contains 137,899 segments across 21 skill types and 417 subtasks, and its benchmark contains 1,260 test instances. Correspondingly, ManiUnit pairs each segment with an explicit subtask instruction; measures sensitivity to perturbations of the robot’s starting base position or joint configuration; and restores intermediate simulator states and defines local success conditions so that each skill can be evaluated without executing preceding stages Evaluations of representative vision-language-action (VLA) policies show that similar aggregate scores can hide substantial per-skill differences. The tested starting-state perturbations also degrade execution: on the full benchmark, joint perturbations reduce success rates by approximately 56% relative to those from demonstrated starting states. On two long-horizon activities, a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations The Trained skills further support complete-task execution on these activities, as coordinating the task and skill policies through a planner raises full-task success from 4.0% to 18.0%.

--- Page 3 ---

1 INTRODUCTION Enabling robots to complete long-horizon mobile manipulation tasks in everyday environments is a central challenge for embodied AI. Unlike tabletop manipulation, which operates in a fixed workspace, this setting requires robots to navigate across multi-room environments and execute a sequence of interdependent manipulation skills guided by a single natural language instruction. Recent vision-language-action (VLA) models have achieved strong results on tabletop benchmarks: StarVLA-π reaches 95.7% success on LIBERO and 88.8% under the randomized setting of RoboTwin 2.0 Despite this progress, reliable execution of long-horizon tasks remains challenging: the winning entry in the 2025 BEHAVIOR Challenge achieved a full-task success rate of only 12.4% on the held-out test set Consider one such task, “make microwave popcorn,” which requires navigation interleaved with opening the microwave door, picking up the popcorn bag, placing it inside, closing the door, and turning on the microwave.

Improvements for AI systems

Improved skill learning via subtask instructions: The ManiUnit dataset attaches an explicit subtask instruction to each segment, naming the operation, the objects it acts on, and the spatial context, which supports mixed-skill training and targeted data selection. This allows AI systems to learn specific operations more reliably than task policies alone.
Enhanced starting-state robustness: The benchmark constructs tests by measuring sensitivity to perturbations of the robot’s starting base position or joint configuration, such as Base Perturbation and Joint Perturbation, enabling evaluation of how preceding skills affect subsequent ones. This directly addresses the challenge that even a successfully completed skill may leave the robot in a base position or joint configuration that differs from its demonstrated starting states.
Improved policy composition via task-level coordination: The system utilizes a planner to coordinate policies, where the planner decides when to invoke the skill policy and which subtask instruction to supply, leading to an overall success increase from 4.0% to 18.0% on long-horizon tasks like Turning On Radio. This allows the AI system to execute complex sequences by delegating manipulation stages while maintaining control over navigation and transitions via a task policy.
Skill-level performance gains: Training a skill policy on ManiUnit segments improves local execution, as shown by the result that a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations. This enables AI systems to specialize in complex manipulation skills with high precision.
Targeted diagnostic evaluation: The benchmark construction allows researchers to evaluate each skill independently without first executing the stages that precede it, which addresses Evaluation opacity by enabling skill-level diagnosis. This allows for identifying precisely which operations limit performance, rather than relying solely on overall task success metrics.

Abstract

Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natural language instruction. Learning and evaluating these skills present three challenges: similar observations under a fixed task instruction may make skill selection ambiguous; even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution; and task-level metrics hinder skill-specific diagnosis, while early failures leave later skills untested. We therefore introduce ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities. Its dataset contains 137,899 segments across 21 skill types and 417 subtasks, and its benchmark contains 1,260 test instances. Correspondingly, ManiUnit pairs each segment with an explicit subtask instruction; measures sensitivity to perturbations of the robot's starting base position or joint configuration; and restores intermediate simulator states and defines local success conditions so that each skill can be evaluated without executing preceding stages. Evaluations of representative vision-language-action (VLA) policies show that similar aggregate scores can hide substantial per-skill differences. The tested starting-state perturbations also degrade execution: on the full benchmark, joint perturbations reduce success rates by approximately 56% relative to those from demonstrated starting states. On two long-horizon activities, a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations. The trained skills further support complete-task execution on these activities, as coordinating the task and skill policies through a planner raises full-task success from 4.0% to 18.0%.