Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations

summary

The gist

Successfully automating dexterous, long-horizon robotic manipulation requires frameworks capable of both highlevel reasoning and fine-grained execution<ref:2607.25397#pg2> DR-LfD (Decomposed and…

In short

DR-LfD is a framework that combines high-level reasoning (TAMP) with fine-grained execution (visuomotor policies) by decomposing human demonstrations into atomic skills based on contact relationships. It learns these skills as primitives or policies and then uses a TAMP solver to reassemble them into complex, long-horizon plans. This approach makes learning data-efficient and enables skill reuse.

Key concepts

TAMP
Task and Motion Planning is a system that combines symbolic planning (reasoning about goals) with motion planning (calculating physical movements). It helps robots figure out *what* to do symbolically and *how* to move physically, making it suitable for complex, multi-step robotic tasks.
Atomic Skills Decomposition
This process breaks down a complex human demonstration into the smallest meaningful actions or skills. The framework uses visual models (VLMs) and contact information to segment demonstrations into distinct categories like object manipulation or specific movements, allowing the system to learn and reuse these basic building blocks.
Visuomotor Policies
These are learned control strategies that directly map visual input (what the robot sees) to motor outputs (how the robot moves). In DR-LfD, these policies are learned for contact-rich segments of a demonstration, providing the fine-grained execution needed for physical interaction.
Skill Reorganization
This is the high-level planning step where individual atomic skills learned from demonstrations are combined by a TAMP solver to achieve a larger goal. Instead of learning one massive policy, the system learns many small skills and then intelligently organizes them into a complete plan.

Terminology used across episodes

This episode discusses

The paper

Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations · Read on arXiv

Yizhou Chen, Hang Xu, Dongjie Yu, Yupu Lu, Tengye Xu, Zeqing Zhang, Wei Zhang, Yi Ren, Ben M. Chen

University of Hong Kong University of Hong Kong Department of Computer Science JD.com Southern University of Science and Technology Huawei Technologies Chinese University of Hong Kong

Successfully automating dexterous, long-horizon bimanual manipulation requires frameworks capable of both high-level reasoning and fine-grained execution. Traditional task and motion planning (TAMP), while excellent at symbolic planning, is often brittle in contact-rich operations. Simultaneously, imitation learning (IL), while effective in manipulation tasks with visual feedback, is limited by its low capability in spatial generalization and multi-stage operation. To reconcile their complementary strengths and limitations, we propose DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations), a framework that integrates visuomotor policies into a TAMP-gated decision-making system. Based on contact relationships, DR-LfD decomposes human demonstrations into atomic skills, which are reproduced as visuomotor policies or object-centric primitives. The initiation, termination, and constraints of the visuomotor policies are carefully modeled and implemented in a TAMP-compatible form, enabling reorganization of skills learned from different sources. DR-LfD transforms the learning problem from one requiring exponential demonstration data over possible skill sequences to one whose demonstration burden scales with the number of distinct skill types, with limited data for each skill. Real-world and simulation benchmarks demonstrate data-efficient skill composition under unseen setups and physical constraints. On two DexMimicGen tasks, DR-LfD with 100 demonstrations outperforms diffusion-based imitation policies with 1000.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Decompose and Reorganize".

Dev: Here's the gist of the paper. Successfully automating dexterous, long-horizon robotic manipulation requires frameworks capable of both highlevel reasoning and fine-grained execution<ref:2607.25397#pg2> DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations), a framework that seamlessly integrates visuomotor policies into a TAMPgated decision-making system<ref:2607.25397#pg2> Based on contact relationships, DR-LfD decomposes human demonstrations into atomic skills, which are reproduced as visuomotor policies or object-centric primitives<ref:2607.25397#pg2> The core idea is to decompose the problem: we learn individual manipulation skills from a small number of demonstrations and reorganize them using a TAMP solver to achieve combinatorial goals<ref:2607.25397#pg2> The Core Framework and Philosophy DR-LfD adopts a graph-like representation of demonstrated skills, segmenting and categorizing skills according to changes in contact relationships<ref:2607.25397#pg2> The symbolic skill repertoire wraps visuomotor policies, object-centric primitives, and planned motions, all of which are used to compose plans by TAMP<ref:2607.25397#pg2> This framework combines the strength of imitation policies and TAMP: with a small amount of demonstrations for each task, DR-LfD exhibits strong performance on tasks with multiple steps, unseen setups, and physical constraints<ref:2607.25397#pg2> The contribution is to devise a novel TAMP-gated system that abstracts atomic skills from demonstrations and reorganizes them to achieve combinatorial goals<ref:2607.25397#pg2> Skill Acquisition and Decomposition The decomposition stage involves a VLM-assisted, contact-aware approach to segment and categorize demonstrations into skills<ref:2607.25397#pg2> Skills are divided into three distinct categories: learned object-centric primitives, learned visuomotor policies, and planned trajectories<ref:2607.25397#pg2> For lower complexity segments, an equivariant network is used to learn open-loop object-centric primitives<ref:2607.25397#pg2> For contact-rich segments, a visuomotor policy is learned, and keyposes are predicted at both ends of each skill for planning<ref:2607.25397#pg2> The VLM serves as a semantic grounding layer by extracting task-relevant objects and relations, which are transformed into contact graphs for skill abstraction<ref:2607.25397#pg2> Learning Atomic Primitives and Policies Object-centric primitives are learned using networks designed to reproduce the trajectory or keypose of each skill<ref:2607.25397#pg2> The primitive prediction network Φτ is designed for learning a skill in the form of an open-loop object-centric primitive τ, which comprises three trajectory types: a trajectory of end-effector pose in SE(3), a trajectory of the d-joint fingers in R d, and a trajectory of in-hand status<ref:2607.25397#pg2> Keypose predictors Φkp are designed to produce admissible initiation and termination conditions for each segmented skill<ref:2607.25397#pg2> These keyposes are assembled by TAMP, marking the initiation and termination of a visuomotor policy<ref:2607.25397#pg2> Online Reorganization and Constraint Handling The online phase involves converting current observations and skill signatures into a PDDL problem and domain<ref:2607.25397#pg2> The planner connects stream outputs to execution, contact verification, perception, and local replanning<ref:2607.25397#pg2> Physical constraints are enforced through test streams like IsReachable(h, p0) and IsSafePolicy(a, κ)<ref:2607.25397#pg2> The safety constraint uses a sweptvolume strategy to compute a minimum distance field (MDF) to detect potential collisions during policy rollout<ref:2607.25397#pg2> The system can correct errors after individual skill executions and recover from unmodeled dynamics through the online planning and adaptation workflow<ref:2607.25397#pg2> Empirical Performance and Generalization DR-LfD consistently outperforms state-of-the-art imitation learning methods across diverse scenarios<ref:2607.25397#pg2> In simulation, DR-LfD maintains 100% success in XY-OOD settings and achieves 88% success in XYH-OOD settings, over 7× higher than the stronger baseline<ref:2607.25397#pg2> In real-world tasks, DR-LfD outperforms both DP and π0.5 under OOD placements<ref:2607.25397#pg2> The success rate in the Cup-sleeve Insertion task under the Leaky condition is low for all methods<ref:2607.25397#pg2> Implications and Limitations Foundation-model semantics are most useful when turned into verifiable action structure, as DR-LfD makes skill semantics extracted from foundation models more reliable for robot deployment<ref:2607.25397#pg2> Long-horizon behavior benefits from compositional skill reuse, relieving a monolithic policy from having to learn both local-execution and skill-transition logic simultaneously<ref:2607.25397#pg2> A limitation inherited from TAMP is the exponential complexity inherent to TAMP search, which aligns with the PSPACE-hard complexity in most modern TAMP methods<ref:2607.25397#pg2> The reliance on metric depth images imposes stricter sensor requirements in comparison with those imageonly visuomotor policies<ref:2607.25397#pg2> DR-LfD demonstrates its ability to combine previously acquired skills to solve long-horizon tasks while complying with novel hard constraints<ref:2607.25397#pg2> It shows resilience to random initial conditions that are unseen in training data and even unattainable through teleoperation alone<ref:2607.25397#pg2> Nevertheless, the limitations inherited from its building blocks remain critical hurdles for better performance and easier deployment<ref:2607.

Paper summary: Rosa: We've talked about how this paper proposes DR-LfD, Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations. Basically, the thesis is that we can combine the strengths of imitation learning and TAMP by integrating visuomotor policies into a TAMP-gated decision-making system.

Dev: The core claim is that by basing skill decomposition on contact relationships found in human demonstrations, you break those long sequences down into atomic skills. These atomic skills are then reproduced either as visuomotor policies or as object-centric primitives, which are then fed into the symbolic skill repertoire for planning.

Taro: So to put it simply, they decompose the problem: you learn individual manipulation skills from a small number of demonstrations and reorganize them using a TAMP solver to achieve combinatorial goals. It’s about learning the parts first, not the whole sequence at once.

Rosa: Right, and why is this important? Because traditional end-to-end policies need exponential amounts of data to learn those skill transitions correctly for novel tasks. DR-LfD aims to make the demonstration burden scale with the number of distinct skill types, not the length of every possible skill sequence.

Dev: The paper is concerned that if you have a robot with N manipulation skills, an end-to-end policy has to learn all those sequences implicitly, demanding exponentially large datasets for generalization. This is what makes it hard to make these policies work outside the training data.

Taro: And they address that by saying symbolic planning and visuomotor policy learning are two parallel streams with complementary strengths, which supports this idea of separating the skill recognition from the execution control.

Rosa: The methodology involves using a VLM to help segment and categorize demonstrations based on task-relevant objects and relations, transforming those into contact graphs for skill abstraction. This is how they get the initial segmentation done before learning any specific policy.

Dev: Then, for lower complexity segments, they use an equivariant network to learn open-loop object-centric primitives, but for contact-rich parts, they learn a visuomotor policy and predict keyposes at both ends of each skill.

Taro: So the primitives are learned to represent the trajectory or keypose of each skill—it’s about learning a specific way an object moves that doesn't necessarily require full control during execution.

Rosa: And these keyposes, which mark the initiation and termination conditions, are then assembled by TAMP, which is crucial because it gives the planner the necessary structure to organize those learned skills into a coherent plan.

Dev: So they are not just learning policies; they are learning components that have defined start and stop points that can be explicitly used by the symbolic planner. That’s what makes them compatible with TAMP.

Taro: It's about making sure the learned components have clear boundaries so the planner knows when to switch from one skill to another. That addresses the sequence problem directly.

Rosa: So in short, they are moving from a problem that requires exponential demonstration data over possible skill sequences to one where the demonstration burden scales with how many distinct skill types you need data for.

Conclusion: Dev: So looking at this paper "Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations", the authors are proposing this systematic framework to solve long-horizon manipulation tasks. It’s about breaking down demonstrations into skills and reorganizing them for novel goals.

Rosa: They are trying to show that by integrating these decomposed skills into a TAMP-gated decision-making system, you can achieve strong performance on tasks with unseen setups and physical constraints using much less demonstration data than previous methods.

Taro: The implication is that for complex robotic manipulation, we don't need one single massive policy trying to learn everything at once; we need a library of reusable skills that the planner can intelligently compose.

Dev: They are showing this is a way to achieve compositionality, where you reuse learned skills across different tasks instead of having to relearn the transition logic every time.

Rosa: It’s about making skill semantics extracted from foundation models more reliable for actual robot deployment by turning that abstract knowledge into verifiable action structures. That’s a practical goal for getting this out of research and into real hardware loops.

Taro: The main hurdle is still the inherent complexity of the TAMP search itself, which means even with these primitives, finding the optimal sequence can still be computationally hard in the worst case scenario.

Dev: And they also noted that their reliance on metric depth images imposes stricter sensor requirements for their primitives than something that only uses image-only policies. That’s a hardware constraint to consider for deployment feasibility.

Rosa: So overall, it's a framework that combines the ability to learn specific skills with the planning power of TAMP, which is what they’re aiming for in this work.

Taro: It shows resilience to random initial conditions that are unseen in training data and even unattainable through teleoperation alone, which is significant for real-world reliability.

Dev: It’s a solid method because it provides a structured approach to handling the uncertainty of long-horizon tasks by breaking them down into something manageable.

More episodes

← Home