ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation

arXiv:2611.07658 · cs.CV · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation".

Jane: Here's the gist of the paper. ArtCrafter presents a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in its rest state and the other in its open state.

Tom: This method is essential for robotic simulation and training because it recovers part geometry and kinematics that remain mutually consistent.

Jane: Core Architecture At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass.

Tom: Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot layout comprising one shared base slot and N paired rest/open slots.

Paper summary: Tom: So, looking at ArtCrafter, the authors are really pushing for a structured slot design—that base slot and those paired rest and open slots—to ensure that parts bind correctly across states. The mechanism they put in place, like Triplet Attention and the repulsion loss Lrep, is designed to actively prevent those geometry inconsistencies we discussed earlier.

Jane: It's about taking a generative model, specifically a DiT-based one, and adding architectural constraints that force it to respect the physical reality of how articulated objects work. What does this mean for the broader field of three dee asset creation <ref:2611.07658#pg1>?

Lu: It means they are focusing on representation learning that naturally accommodates multiple part-level slots without forcing a shared spatial grid, which is what their vecset representation is designed for. They're using the structure of the model to enforce kinematic constraints rather than relying solely on external data or retrieval methods for parts.

Meng: For deployment, if this analytic derivation of joint parameters holds up in real-world scenarios, it means we could deploy systems that generate complex articulated models much faster and more reliably than current methods that might require extensive iterative refinement. It changes the pipeline from a generation step to a direct simulation input.

Tom: The title, ArtCrafter: Feed-Forward Generation of Articulated three dee Object with Analytic Joint Derivation, points out that they didn't just generate geometry; they derived the kinematic parameters analytically from the rigid transformations between rest and open states <ref:2611.07658#pg1,ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation>. That’s a very specific technical achievement.

Jane: So in simple terms, it means you input two pictures of an object and get not just a three dee model, but you also get all the mathematical instructions on how those parts should move relative to each other for simulation purposes <ref:2611.07658#pg1>. That makes it much more useful than just generating a static shape.

Conclusion: Tom: So, ArtCrafter is presenting a new way to generate these articulated three dee objects using just two images—one of the object still and one open—and they’re deriving the actual joint parameters analytically from that.

Jane: It sounds like they are trying to solve the problem of making sure that when you generate a piece, it looks geometrically consistent whether it's in the closed state or the open state.

Lu: Exactly. They’re using this structured slot layout, with a shared base slot and paired rest and open slots, and they use triplet attention to keep those paired parts geometrically consistent across both states.

Meng: I’m thinking about deployment right now—if this works for simulation, what kind of data do we need to feed it? Does it handle complex rigs well?

Lalam: Based on the structure described, their core idea is that constraining the model to explain two images in different articulation states forces it to bind those parts together kinematically.

Tom: So they’re not just making pretty shapes anymore; they’re figuring out how the object actually moves, like its joint type and axis direction, directly from those images.

Jane: That analytic derivation part is key because it means you don't need a separate step to calculate the mechanics; it comes out as part of the generation process itself.

Lu: They use a coarse-to-fine pipeline involving RANSAC and ICP to get that rigid transformation first, and then they use that to determine the joint type, axis, origin, and motion range.

Meng: If those analytic derivations hold up when we try to run this on a massive set of varied objects, I’m worried about how much computational time it takes for that derivation step alone.

Tom: That’s fair; the complexity of that derivation is where the engineering challenge lies, but if it's faster than iterative refinement, it could speed up training significantly.

Jane: It definitely shifts the focus from just generating a mesh to generating a functional physical model ready for robotics or simulation.

Lu: The paper shows how this structured slot design directly addresses failure modes like independent part generation or unstructured joint generation that cause geometry to break down between states.

Meng: So the main thing is that they’ve engineered the slots so they naturally enforce the kinematic constraints we need for physical objects, rather than just hoping a standard generative model figures it out.

Tom: That’s what makes me interested; it sounds like they’ve put a lot of work into fixing those structural issues in how the model learns to represent parts.

Jane: And if this analytic derivation works as described, it opens up possibilities for training robotic systems where you need perfect part-to-part alignment across different poses.

Lu: It moves the problem from purely learning visual features to learning a representation that inherently understands physical constraints and kinematics.

Qualcomm AI Research

cs.CV

Submitted: 2026-11-05

Updated: 2026-11-07

Comments: Accepted to NeurIPS 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: ArtCrafter presents a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images…

Key concepts

DiT-based Generative Model
This is the core AI structure used to generate 3D shapes. DiT stands for Diffusion Transformer, which is a powerful type of model similar to large language models but adapted for generating complex data like 3D meshes. It learns to predict the object's structure step-by-step during the generation process.
Slot Attention
The model organizes its generation process using 'slots'—specific attention mechanisms that handle different parts of the object. By using one shared base slot and N paired rest/open slots, it forces the model to treat related parts together, improving consistency between the two articulation states.
Triplet Attention
This mechanism is used to enforce geometric consistency between paired slots (a rest slot and its corresponding open slot). It ensures that when generating a part in one state, the model maintains a consistent relationship with the part in the other state, preventing parts from becoming geometrically inconsistent.
Analytic Kinematic Derivation
After generating the parts, this step mathematically calculates how those parts move. By analyzing rigid transformations between rest and open states using RANSAC-ICP, it derives precise parameters like joint type, axis direction, and motion range for robotic simulation.

Terminology

Summary

ArtCrafter presents a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in its rest state and the other in its open state. p. 0

Where the paper says it

ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation Minh Canh Tu∗ Quang-Binh Nguyen∗† Khoi Nguyen Qualcomm AI Research‡ Abstract Articulated 3D assets are essential for robotic simulation and training, yet their generation from casual images poses a key challenge: recovering part geometry and kinematics that remain mutually consistent. We present ArtCrafter, a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in the rest state and the other in the open state.

This method is essential for robotic simulation and training because it recovers part geometry and kinematics that remain mutually consistent. p. 0
Where the paper says it

ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation Minh Canh Tu∗ Quang-Binh Nguyen∗† Khoi Nguyen Qualcomm AI Research‡ Abstract Articulated 3D assets are essential for robotic simulation and training, yet their generation from casual images poses a key challenge: recovering part geometry and kinematics that remain mutually consistent. We present ArtCrafter, a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in the rest state and the other in the open state.

Core Architecture

At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass. Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot layout comprising one shared base slot and N paired rest/open slots. p. 0

Where the paper says it

At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass. Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot layout comprising one shared base slot and N paired rest/open slots, encouraging each pair to bind to the same kinematic part.

To ensure geometrically consistent and disjoint part generation, the model introduces Triplet Attention to enforce cross-state geometric consistency between paired rest/open slots anchored through the shared base slot, and a repulsion loss Lrep on decoded part point clouds to prevent interpart penetration throughout denoising. p. 0
Where the paper says it

Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot layout comprising one shared base slot and N paired rest/open slots, encouraging each pair to bind to the same kinematic part. To ensure geometrically consistent and disjoint part generation, we introduce Triplet Attention to enforce cross-state geometric consistency between paired rest/open slots anchored through the shared base slot, and a repulsion loss Lrep on decoded part point clouds to prevent interpart penetration throughout denoising.

Failure Modes and Slot Design

The authors analyze two failure modes of naive multi-part generation that motivate their structured slot design. Failure Mode I is independent per-state generation, where the i-th rest slot is unrelated to the i-th open slot, leading to inconsistent part binding and geometry across states. p. 3

Where the paper says it

, PartCrafter [21]) independently on each image yields no cross-state slot identity: the i-th rest slot is unrelated to the i-th open slot, leading to inconsistent part binding and geometry across states (row 1, Fig. This motivates joint generation with a 1 + 2N slot layout that pairs rest/open slots by index from the outset.

Failure Mode II is unstructured joint generation, which lacks a mechanism to pair rest/open slots or distinguish the static base from moving parts, resulting in diffuse and overlapping slot clouds at t = 0. p. 4
Where the paper says it

Jointly generating 2N unstructured slots without state-specific routing or disjointness constraints provides no mechanism to pair rest/open slots or distinguish the static base from moving parts. Consequently, slot clouds remain diffuse and overlapping at t = 0, without clear part separation in either state (row 2, Fig.

The authors hypothesize that slots constrained to jointly explain two images of the same object in different articulation states bind to the object’s kinematic parts. p. 4
Where the paper says it

We extend this principle to articulated objects with the following hypothesis: Hypothesis (Slot-binding). Slots constrained to jointly explain two images of the same object in different articulation states bind to the object’s kinematic parts.

Training Objective and Regularization

ArtCrafter is trained with rectified flow matching [7], which transports a Gaussian distribution to the data distribution along a linear trajectory. The total objective is L = Lflow + λrep Lrep, where λrep = 0. p. 6

Where the paper says it

where the noise level t is shared across all 2N + 1 slots. The total objective is: L = Lflow + λrep Lrep, (9) with λrep = 0.

5. The repulsion loss Lrep penalizes inter-part overlap directly in 3D point space via a margin-based Chamfer Distance loss (Equation 7). p. 5
Where the paper says it

For each pair of distinct parts (i, j) at the same state, we penalize spatial overlap via a margin-based Chamfer Distance loss: d(Pi , Pj ) = 1 |Pi | X p∈Pi max 0, γ − min q∈Pj ∥p − q∥ 2 + 1 |Pj | X q∈Pj max 0, γ − min p∈Pi ∥q − p∥ 2 , (6) Lrep = 1 N 2 X (i,j)∈( [N] 2 ) d(Pi , Pj ), (7) where γ > 0 is the margin threshold. Gradients flow back through the frozen ϕ to the part latents, coupling the abstract latent space to geometric disjointness.

This regularization biases the denoising trajectory without modifying the pretrained backbone. p. 6
Where the paper says it

The total objective is: L = Lflow + λrep Lrep, (9) with λrep = 0. The point decoder ϕ remains frozen throughout training; the regularization biases the denoising trajectory without modifying the pretrained backbone.

Analytic Kinematic Derivation

In the second stage, joint parameters—type, axis, origin, and range—are derived analytically from the rigid transformation between each rest/open part pair. This derivation involves a coarse-to-fine RANSAC-ICP pipeline to estimate the rigid transformation T = [R t0 1]. p. 6

Where the paper says it

RANSAC over the feature correspondences yields a coarse transformation T′ i ∈ R 4×4 . For local registration, starting from T′ i , Iterative Closest Point (ICP) refines the alignment to produce the final rigid transformation Ti .

The joint type is determined by thresholding the rotation angle extracted from Ri ∈ R3×3, and the axis direction, origin, and motion range follow directly from the decomposition. p. 6
Where the paper says it

Finally, from Ti = [Ri | ti ], we determine joint type by thresholding the rotation angle extracted from Ri ∈ R 3×3 : large rotation angle indicates a revolute joint, otherwise it is a prismatic joint. Joint axis direction, origin, and motion range follow directly from the decomposition.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved geometry generation by enforcing cross-state geometric consistency through triplet attention to ensure consistent part binding across articulation states by t = 0. This prevents failures like Failure Mode I where independent per-state generation leads to inconsistent binding and incompatible part scale across the two independent runs.

  2. Enhanced kinematic prediction accuracy by deriving joint parameters analytically from rigid transformations, which allows for a method that is grounding kinematics directly in geometry without learned joint-parameter regression or per-instance optimization. This contrasts with methods like DIPO which struggle to match input geometry while still predicting joints.

  3. Increased robustness against structural prediction errors by employing penalized metrics like Dd, which penalize the lack of or redundant predicted parts to assess the geometric and kinematic quality of generated articulated objects on challenging out-of-distribution test sets.

  4. Improved generalization to novel object categories by demonstrating that ArtCrafter can recognize and generate faithful rest and open meshes with reasonable and kinematically meaningful part decomposition even for unseen objects like a kettle or trash can, showing robustness against completely new objects.

  5. Reduced inference time while maintaining high quality by achieving an average inference time of 22.4 seconds, which is approximately 17 times faster than PhysX-Anything, allowing for more practical deployment in simulation pipelines.

Abstract

Articulated 3D assets are essential for robotic simulation and training, yet their generation from casual images poses a key challenge: recovering part geometry and kinematics that remain mutually consistent. We present ArtCrafter, a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in the rest state and the other in the open state. At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass. Inspired by Slot Attention, its denoising process operates on a 1+2N slot layout comprising one shared base slot and N paired rest/open slots, encouraging each pair to bind to the same kinematic part. To ensure geometrically consistent and disjoint part generation, we introduce Triplet Attention to enforce cross-state geometric consistency between paired rest/open slots anchored through the shared base slot, and a repulsion loss L rep on decoded part point clouds to prevent inter-part penetration throughout denoising. Given the resulting paired meshes, URDF joint parameters---type, axis, origin, and range---are derived analytically from the rigid transformation between each rest/open pair, providing an interpretable alternative to learned joint-parameter regression. Extensive experiments on both in-distribution and out-of-distribution datasets demonstrate that ArtCrafter outperforms existing method under the same dual-state setting in geometry quality and articulation precision, while remaining competitive with methods operating under different input settings and paradigms.

Sources

Related papers