ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation".
Jane: Here's the gist of the paper. ArtCrafter presents a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in its rest state and the other in its open state.
Tom: This method is essential for robotic simulation and training because it recovers part geometry and kinematics that remain mutually consistent.
Jane: Core Architecture At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass.
Tom: Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot layout comprising one shared base slot and N paired rest/open slots.
Paper summary: Tom: So, looking at ArtCrafter, the authors are really pushing for a structured slot design—that base slot and those paired rest and open slots—to ensure that parts bind correctly across states. The mechanism they put in place, like Triplet Attention and the repulsion loss Lrep, is designed to actively prevent those geometry inconsistencies we discussed earlier.
Jane: It's about taking a generative model, specifically a DiT-based one, and adding architectural constraints that force it to respect the physical reality of how articulated objects work. What does this mean for the broader field of three dee asset creation <ref:2611.07658#pg1>?
Lu: It means they are focusing on representation learning that naturally accommodates multiple part-level slots without forcing a shared spatial grid, which is what their vecset representation is designed for. They're using the structure of the model to enforce kinematic constraints rather than relying solely on external data or retrieval methods for parts.
Meng: For deployment, if this analytic derivation of joint parameters holds up in real-world scenarios, it means we could deploy systems that generate complex articulated models much faster and more reliably than current methods that might require extensive iterative refinement. It changes the pipeline from a generation step to a direct simulation input.
Tom: The title, ArtCrafter: Feed-Forward Generation of Articulated three dee Object with Analytic Joint Derivation, points out that they didn't just generate geometry; they derived the kinematic parameters analytically from the rigid transformations between rest and open states <ref:2611.07658#pg1,ArtCrafter: Feed-Forward Generation of Articulated 3D Object with Analytic Joint Derivation>. That’s a very specific technical achievement.
Jane: So in simple terms, it means you input two pictures of an object and get not just a three dee model, but you also get all the mathematical instructions on how those parts should move relative to each other for simulation purposes <ref:2611.07658#pg1>. That makes it much more useful than just generating a static shape.
Conclusion: Tom: So, ArtCrafter is presenting a new way to generate these articulated three dee objects using just two images—one of the object still and one open—and they’re deriving the actual joint parameters analytically from that.
Jane: It sounds like they are trying to solve the problem of making sure that when you generate a piece, it looks geometrically consistent whether it's in the closed state or the open state.
Lu: Exactly. They’re using this structured slot layout, with a shared base slot and paired rest and open slots, and they use triplet attention to keep those paired parts geometrically consistent across both states.
Meng: I’m thinking about deployment right now—if this works for simulation, what kind of data do we need to feed it? Does it handle complex rigs well?
Lalam: Based on the structure described, their core idea is that constraining the model to explain two images in different articulation states forces it to bind those parts together kinematically.
Tom: So they’re not just making pretty shapes anymore; they’re figuring out how the object actually moves, like its joint type and axis direction, directly from those images.
Jane: That analytic derivation part is key because it means you don't need a separate step to calculate the mechanics; it comes out as part of the generation process itself.
Lu: They use a coarse-to-fine pipeline involving RANSAC and ICP to get that rigid transformation first, and then they use that to determine the joint type, axis, origin, and motion range.
Meng: If those analytic derivations hold up when we try to run this on a massive set of varied objects, I’m worried about how much computational time it takes for that derivation step alone.
Tom: That’s fair; the complexity of that derivation is where the engineering challenge lies, but if it's faster than iterative refinement, it could speed up training significantly.
Jane: It definitely shifts the focus from just generating a mesh to generating a functional physical model ready for robotics or simulation.
Lu: The paper shows how this structured slot design directly addresses failure modes like independent part generation or unstructured joint generation that cause geometry to break down between states.
Meng: So the main thing is that they’ve engineered the slots so they naturally enforce the kinematic constraints we need for physical objects, rather than just hoping a standard generative model figures it out.
Tom: That’s what makes me interested; it sounds like they’ve put a lot of work into fixing those structural issues in how the model learns to represent parts.
Jane: And if this analytic derivation works as described, it opens up possibilities for training robotic systems where you need perfect part-to-part alignment across different poses.
Lu: It moves the problem from purely learning visual features to learning a representation that inherently understands physical constraints and kinematics.
Qualcomm AI Research
cs.CV
Submitted: 2026-11-05
Updated: 2026-11-07
Comments: Accepted to NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: ArtCrafter presents a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images…
Key concepts
- DiT-based Generative Model
- This is the core AI structure used to generate 3D shapes. DiT stands for Diffusion Transformer, which is a powerful type of model similar to large language models but adapted for generating complex data like 3D meshes. It learns to predict the object's structure step-by-step during the generation process.
- Slot Attention
- The model organizes its generation process using 'slots'—specific attention mechanisms that handle different parts of the object. By using one shared base slot and N paired rest/open slots, it forces the model to treat related parts together, improving consistency between the two articulation states.
- Triplet Attention
- This mechanism is used to enforce geometric consistency between paired slots (a rest slot and its corresponding open slot). It ensures that when generating a part in one state, the model maintains a consistent relationship with the part in the other state, preventing parts from becoming geometrically inconsistent.
- Analytic Kinematic Derivation
- After generating the parts, this step mathematically calculates how those parts move. By analyzing rigid transformations between rest and open states using RANSAC-ICP, it derives precise parameters like joint type, axis direction, and motion range for robotic simulation.
Terminology
Summary
ArtCrafter presents a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in its rest state and the other in its open state. p. 0 ArtCrafter: Feed-Forward Generation of Articulated
3D Object with Analytic Joint Derivation
Minh Canh Tu∗ Quang-Binh Nguyen∗† Khoi Nguyen
Qualcomm AI Research‡
Abstract
Articulated 3D assets are essential for robotic simulation and training, yet their
generation from casual images poses a key challenge: recovering part geometry
and kinematics that remain mutually consistent. We present ArtCrafter, a novel
feed-forward generative model for controllable generation of articulated 3D objects
from a pair of images, one depicting the object in the rest state and the other in
the open state. ArtCrafter: Feed-Forward Generation of Articulated
3D Object with Analytic Joint Derivation
Minh Canh Tu∗ Quang-Binh Nguyen∗† Khoi Nguyen
Qualcomm AI Research‡
Abstract
Articulated 3D assets are essential for robotic simulation and training, yet their
generation from casual images poses a key challenge: recovering part geometry
and kinematics that remain mutually consistent. We present ArtCrafter, a novel
feed-forward generative model for controllable generation of articulated 3D objects
from a pair of images, one depicting the object in the rest state and the other in
the open state.Where the paper says it
Where the paper says it
Core Architecture
At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass. Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot layout comprising one shared base slot and N paired rest/open slots. p. 0 At the core of ArtCrafter is a DiT-based generative model that
jointly produces part-level meshes for both articulation states in a single forward
pass. Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot
layout comprising one shared base slot and N paired rest/open slots, encouraging
each pair to bind to the same kinematic part. Inspired by Slot Attention, its denoising process operates on a 1 + 2N slot
layout comprising one shared base slot and N paired rest/open slots, encouraging
each pair to bind to the same kinematic part. To ensure geometrically consistent
and disjoint part generation, we introduce Triplet Attention to enforce cross-state
geometric consistency between paired rest/open slots anchored through the shared
base slot, and a repulsion loss Lrep on decoded part point clouds to prevent interpart penetration throughout denoising.Where the paper says it
Where the paper says it
Failure Modes and Slot Design
The authors analyze two failure modes of naive multi-part generation that motivate their structured slot design. Failure Mode I is independent per-state generation, where the i-th rest slot is unrelated to the i-th open slot, leading to inconsistent part binding and geometry across states. p. 3 ,
PartCrafter [21]) independently on each image yields no cross-state slot identity: the i-th rest
slot is unrelated to the i-th open slot, leading to inconsistent part binding and geometry across states
(row 1, Fig. This motivates joint generation with a 1 + 2N slot layout that pairs rest/open slots by
index from the outset. Jointly generating 2N unstructured slots without
state-specific routing or disjointness constraints provides no mechanism to pair rest/open slots
or distinguish the static base from moving parts. Consequently, slot clouds remain diffuse and
overlapping at t = 0, without clear part separation in either state (row 2, Fig. We extend this principle to articulated objects with the following
hypothesis:
Hypothesis (Slot-binding). Slots constrained to jointly explain two images of the same object in
different articulation states bind to the object’s kinematic parts.Where the paper says it
Where the paper says it
Where the paper says it
Training Objective and Regularization
ArtCrafter is trained with rectified flow matching [7], which transports a Gaussian distribution to the data distribution along a linear trajectory. The total objective is L = Lflow + λrep Lrep, where λrep = 0. p. 6 where the noise level t is shared across all 2N + 1 slots. The total objective is:
L = Lflow + λrep Lrep, (9)
with λrep = 0. For each pair of distinct parts
(i, j) at the same state, we penalize spatial overlap via a margin-based Chamfer Distance loss:
d(Pi
, Pj ) = 1
|Pi
|
X
p∈Pi
max
0, γ − min
q∈Pj
∥p − q∥
2
+
1
|Pj |
X
q∈Pj
max
0, γ − min
p∈Pi
∥q − p∥
2
,
(6)
Lrep =
1
N
2
X
(i,j)∈(
[N]
2 )
d(Pi
, Pj ), (7)
where γ > 0 is the margin threshold. Gradients flow back through the frozen ϕ to the part latents,
coupling the abstract latent space to geometric disjointness. The total objective is:
L = Lflow + λrep Lrep, (9)
with λrep = 0. The point decoder ϕ remains frozen throughout training; the regularization biases
the denoising trajectory without modifying the pretrained backbone.Where the paper says it
Where the paper says it
Where the paper says it
Analytic Kinematic Derivation
In the second stage, joint parameters—type, axis, origin, and range—are derived analytically from the rigid transformation between each rest/open part pair. This derivation involves a coarse-to-fine RANSAC-ICP pipeline to estimate the rigid transformation T = [R t0 1]. p. 6 RANSAC over the feature correspondences yields a coarse
transformation T′
i ∈ R
4×4
. For local registration, starting from T′
i
, Iterative Closest Point (ICP)
refines the alignment to produce the final rigid transformation Ti
. Finally, from Ti = [Ri
| ti
], we
determine joint type by thresholding the rotation angle extracted from Ri ∈ R
3×3
: large rotation
angle indicates a revolute joint, otherwise it is a prismatic joint. Joint axis direction, origin, and
motion range follow directly from the decomposition.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved geometry generation by enforcing cross-state geometric consistency through triplet attention to ensure
consistent part binding across articulation states by t = 0.
This prevents failures like Failure Mode I where independent per-state generation leads toinconsistent binding and incompatible part scale across the two independent runs.
-
Enhanced kinematic prediction accuracy by deriving joint parameters analytically from rigid transformations, which allows for a method that is
grounding kinematics directly in geometry without learned joint-parameter regression or per-instance optimization.
This contrasts with methods like DIPO which struggle to match input geometry while still predicting joints. -
Increased robustness against structural prediction errors by employing penalized metrics like Dd, which
penalize the lack of or redundant predicted parts
to assess thegeometric and kinematic quality of generated articulated objects
on challenging out-of-distribution test sets. -
Improved generalization to novel object categories by demonstrating that ArtCrafter can recognize and generate
faithful rest and open meshes with reasonable and kinematically meaningful part decomposition
even for unseen objects like a kettle or trash can, showing robustness againstcompletely new objects.
-
Reduced inference time while maintaining high quality by achieving an average inference time of 22.4 seconds, which is
approximately 17 times faster than PhysX-Anything,
allowing for more practical deployment in simulation pipelines.
Abstract
Articulated 3D assets are essential for robotic simulation and training, yet their generation from casual images poses a key challenge: recovering part geometry and kinematics that remain mutually consistent. We present ArtCrafter, a novel feed-forward generative model for controllable generation of articulated 3D objects from a pair of images, one depicting the object in the rest state and the other in the open state. At the core of ArtCrafter is a DiT-based generative model that jointly produces part-level meshes for both articulation states in a single forward pass. Inspired by Slot Attention, its denoising process operates on a 1+2N slot layout comprising one shared base slot and N paired rest/open slots, encouraging each pair to bind to the same kinematic part. To ensure geometrically consistent and disjoint part generation, we introduce Triplet Attention to enforce cross-state geometric consistency between paired rest/open slots anchored through the shared base slot, and a repulsion loss L rep on decoded part point clouds to prevent inter-part penetration throughout denoising. Given the resulting paired meshes, URDF joint parameters---type, axis, origin, and range---are derived analytically from the rigid transformation between each rest/open pair, providing an interpretable alternative to learned joint-parameter regression. Extensive experiments on both in-distribution and out-of-distribution datasets demonstrate that ArtCrafter outperforms existing method under the same dual-state setting in geometry quality and articulation precision, while remaining competitive with methods operating under different input settings and paradigms.
Sources
- PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image
- SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge
- MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstruction
- Particulate: Feed-Forward 3D Object Articulation
- Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling
- ART: Articulated Reconstruction Transformer
- PAct: Part-Decomposed Single-View Articulated Object Generation
- Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
- URDF-Anything+: End-to-End Generation for Simulation-Ready Articulated Assets
- MotionAnymesh: Physics-Grounded Articulation for Simulation-Ready Digital Twins
- HoloPart: Generative 3D Part Amodal Segmentation
- Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs