BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly

arXiv:2605.07605 · cs.RO · Submitted 2026-05-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly".

Rosa: The gist The BRICKCRAFT framework presents a compositional method for long-horizon interlocking brick assembly that integrates skill-oriented reasoning, situated manual guidance, and compositional visuomotor execution to achieve robust generalization.

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called "BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly," and the core idea here is tackling the whole long-horizon assembly problem by breaking it down into reusable skills.

Dev: Yeah, it claims they've got this compositional framework that uses relative formulation where every step is anchored to a reference brick inside the structure, which lets them decompose these big tasks into a finite set of primitive skills.

Taro: I'm thinking about what matters here is how they handle the complexity of those interlocking interfaces, because they mention two tricky things: visual ambiguity from repetitive features and the spatial multi-modality where there are multiple valid poses for an interface, which is tough for any robot to figure out <ref:2605.07605#pg2>.

Rosa: Exactly, so how does this framework bridge that gap between the high-level plan and the actual physical movement?

Dev: Well, they introduce something called assembly intent grounding, which generates what they call situated manuals. These manuals project the abstract assembly intents onto what the robot is seeing in real-time observations to give it explicit spatial guidance <ref:2605.07605#pg2>.

Taro: That sounds like a big step because it's not just relying on vision for general guidance; they are actively highlighting task-relevant entities right in the robot's view, which should help when the world misbehaves and things get messy <ref:2605.07605#pg3>.

Rosa: And then they chain all these grounded skills together through a modular execution pipeline to finish the assembly task, turning that plan into physical action <ref:2605.07605#pg1>.

Dev: The paper says this approach allows them to acquire proficient assembly skills from just a limited set of demonstrations and then effectively compose those for different long-horizon tasks <ref:2605.07605#pg1>.

Taro: I'm curious about the execution part; they mention pre- and post-conditions for skill transitions, which is supposed to make sure the whole chain stays robust by verifying what needs to happen before and after each move <ref:2605.07605#pg2>.

Rosa: So when you look at the numbers, they validated this against baselines like Goal Image-Conditioned Diffusion Policy and Goal Image-Conditioned pi zero point five <ref:2605.07605#pg1>, and BrickCraft achieved an overall success rate of eighty-six point two five percent across two hundred forty trials <ref:2605.07605#pg1>.

Dev: That's a solid number, especially when you compare it to the GI-DP baseline which suffered a performance degradation of about twelve point five percent to fifteen point zero percent on unseen tasks <ref:2605.07605#pg1>.

Taro: What I find interesting from the EigenCAM visualizations is how this situated manual isolates those geometric invariants across different assembly tasks, directing the robot's visual attention toward that relative geometric relationship between the target and reference bricks <ref:2605.07605#pg3>.

Rosa: So, for someone listening who isn't a robotics expert, what does this actually change for them in terms of what we expect from an autonomous system?

Dev: It means that instead of needing a massive dataset for every single new assembly design, the system learns how to combine small pieces and use those explicit spatial guides to handle structures it hasn't seen before <ref:2605.07605#pg1>.

Taro: It’s about moving away from brittle planning towards something that can generalize its physical execution by relying on these relative anchors instead of just trying to solve the whole maze at once <ref:2605.07605#pg2>.

Rosa: Thinking about the title, "BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly," it really captures how they combine those three major elements—composition, guidance, and skill learning—to solve a very specific physical problem.

Dev: And the authors are from Tsinghua University and Carnegie Mellon University, which suggests a strong foundation in both theory and practical control engineering for this kind of embodied AI <ref:2605.07605#pg1>.

Taro: I wonder what the practical limitations are; they mentioned that one thing they have to deal with is the visual ambiguity arising from periodic geometric units, which means even this framework still struggles a bit with repetitive patterns <ref:2605.07605#pg2>.

Rosa: So, to wrap up for now, this paper shows a way to use relative formulation and situated manuals to make long-horizon assembly more robust and compositional than previous methods <ref:2605.07605#pg1>.

Dev: And the implication is that we're getting closer to agents that can handle complex physical manipulation in real-world scenarios by making sure the execution plan stays grounded spatially <ref:2605.07605#pg3>.

Conclusion: Rosa: So, BrickCraft is all about taking this long chain of assembly steps and making them work by breaking them down into smaller, reusable skills anchored to a reference brick inside the structure.

Dev: Right, so it’s not just one big plan that gets executed; it's like building with Lego where each piece you grab is defined by where it connects to something already there.

Taro: The authors are from Tsinghua and Carnegie Mellon, which suggests they’ve got a solid grasp on both the theory of autonomy and the actual control engineering side of things.

Rosa: And what this whole thing means is that instead of needing a massive library for every single new type of brick arrangement, you can learn how to combine those basic moves to build something completely new.

Dev: It’s about moving away from systems that try to solve the entire maze at once and instead focusing on making sure the robot executes each small, grounded move correctly in sequence.

Taro: The big picture here is that we’re getting closer to agents that can handle complex physical manipulation in real-world settings by relying on these relative anchors instead of just trying to solve the whole maze at once.

Rosa: Exactly, so it shifts the focus from grand planning to robust, grounded execution for those long-horizon tasks.

Dev: We saw them get a solid success rate of eighty-six percent across all those tests, which is good when you compare it to some other methods that struggle with unseen designs.

Taro: That performance boost comes from the way they’ve managed to isolate the important geometric relationships between the parts using those situated manuals they talked about earlier.

Rosa: It really shows how important it is to provide that explicit spatial guidance, projecting what the robot *should* be doing onto what it’s currently seeing.

Dev: So, while this framework is very strong for these specific assembly tasks, we still need to figure out how much context-aware failure recovery we can bake into this pipeline for when things inevitably go wrong in a messy real environment.

Tsinghua University · Carnegie Mellon University

cs.RO

Submitted: 2026-05-08

Updated: 2026-10-09

Project page: https://intelligent-control-lab.github.io/BrickCraft

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The BRICKCRAFT framework presents a compositional method for long-horizon interlocking brick assembly that integrates skill-oriented reasoning, situated manual guidance, and compositional

Key concepts

Relative Formulation
This approach models assembly by anchoring each step to a specific reference brick already placed in the structure. Instead of planning absolute positions, the system plans actions relative to this known point, which simplifies task decomposition into smaller, manageable primitive skills.
Situated Manuals
These are visual aids that translate high-level assembly goals into explicit spatial guidance for the robot. They work by projecting the intended assembly relationship onto what the robot sees in real-time, helping it understand where to move and how to manipulate bricks correctly during physical execution.
Compositional Execution
This is the process of chaining together individual, spatially grounded skills sequentially to complete a long-horizon task. Each step is a primitive skill guided by its situated manual, and these skills are ordered to build the final complex assembly from start to finish.
Skill Chain
The framework realizes long-horizon tasks as a sequence of paired skills, such as picking an object and then assembling it onto a reference. These skills are governed by pre-conditions (what must be true before starting) and post-conditions (what must be true after finishing), ensuring the assembly process is robust.

Terminology

Summary

The gist The BRICKCRAFT framework presents a compositional method for long-horizon interlocking brick assembly that integrates skill-oriented reasoning, situated manual guidance, and compositional visuomotor execution to achieve robust generalization.

How it works

BRICKCRAFT models the assembly process through three phases: (i) Skill-Oriented Assembly Reasoning, which decomposes the task into steps anchored to reference bricks and maps them to reusable primitive skills. (ii) Assembly Intent Grounding generates situated manuals that provide spatial guidance by projecting assembly intent onto real-time robot observations. (iii) Compositional Visuomotor Execution chains these spatially grounded skills to complete the assembly task.

The framework utilizes a relative formulation where each step is anchored to a reference brick within the partial structure, thereby decomposing complex tasks into a finite set of reusable primitive skills. The assembly task is formulated as an ordered sequence P = (η(1), η(2),..., η(N)), where each step describes the action of attaching a target brick onto a reference brick. This action is parameterized by a geometric task encoding τ = [τx, τy, τz, τθ] ∈ T. A removal action is considered valid only if it satisfies C = Ccon ∧ Cobs ∧ Cop ∧ Cstab = 1.

Assembly Intent Grounding as Situated Manuals

Situated manuals are introduced to bridge the gap between high-level assembly plans and low-level visuomotor execution by projecting abstract assembly intents onto robot observations. This process involves four primary steps: 1) Visual Reference Generation using the Blender simulation engine to render a visual reference Ir. Expert demonstrations are collected using a teleoperation-replay strategy to minimize human variance by filtering out inconsistent pacing and misaligned contacts. Long-horizon assembly is executed as a skill chain where each assembly step η is realized through a skill pair consisting of S(i)pick and S(i)assembly. Pre- and post-conditions govern skill transitions to ensure robustness by verifying prerequisites for initiation and intended outcomes after execution.

Experimental Validation

BRICKCRAFT was validated against baselines like Goal Image-Conditioned Diffusion Policy (GI-DP) and Goal Image-Conditioned π0.5 (GI-π0.5). The results show that BRICKCRAFT achieves an overall success rate of 86.25% across 240 trials. This performance demonstrates robust structural generalization to unseen structures, contrasting with GI-DP which suffers a 12.5% to 15.0% performance degradation on unseen tasks. EigenCAM visualizations reveal that the situated manual isolates geometric invariants across diverse assembly tasks, directing visual attention toward the relative geometric relationship between target and reference bricks. The framework successfully accomplishes long-horizon assembly tasks autonomously across 4 structural designs.

Conclusion

BRICKCRAFT successfully bridges symbolic reasoning with spatially grounded atomic skills by modeling the process through a relative formulation and introducing situated manuals to provide explicit spatial guidance. Extensive real-world experiments confirm that this approach effectively suppresses task-irrelevant visual distractors and enables strong compositional generalization to unseen structures. The framework offers a promising pathway toward deploying robust embodied agents in broader real-world manipulation scenarios.

How it works

To further elevate the autonomy of BRICKCRAFT in future work, the plan is to integrate context-aware failure recovery skills into the compositional pipeline.

Improvements for AI systems

  1. Improve long-horizon assembly planning by replacing monolithic end-to-end control paradigms with a skill-centric framework that decomposes complex tasks into a finite set of reusable primitive skills. This allows for enhanced data efficiency and facilitates seamless adaptation to novel assembly tasks compared to existing approaches.

  2. Introduce situated manuals as a mechanism for spatial grounding by using them to project the assembly intent onto real-time robot observations. This addresses the challenge where inferring explicit poses is highly vulnerable to perception noise in tight-clearance assembly, effectively providing explicit visual guidance for learned visuomotor skills.

  3. Enhance skill learning robustness by incorporating external force observations into the observation space, as this has been shown to enhance contact awareness, thereby improving the compliance and robustness of the assembly process. This specifically addresses issues like Misalignment and helps in handling complex interactions during insertion.

  4. Improve generalization to unseen structures by utilizing a compositional execution pipeline that chains skills based on a plan where each step is anchored to a reference brick within the partial structure. This strategy ensures that learned skills are composed reliably, as demonstrated by BRICKCRAFT's ability to achieve an 86.25% success rate across 240 assembly trials on 48 distinct structures.

  5. Develop context-aware failure recovery skills to address potential execution errors by incorporating real-time anomaly detection and assembly state tracking, enabling the system to autonomously plan and execute adaptive correction actions such as re-grasping and deformation correction. This moves beyond static planning by allowing the robot to handle failures like Collision or Structural Deformation dynamically.

Sources

Related papers