Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation

summary

Video file (mp4)

The gist

The gist: STRUCTPARTS is a framework that allows part-decomposed generation models to inherit both geometric detail and structural coherence from holistic 3D priors through dual-frame coordinate

In short

STRUCTPARTS is a framework for part-decomposed generation that allows models to combine geometric detail and structural coherence from a holistic 3D prior. It achieves this using dual-frame coordinate interleaving, where each part is processed locally as a complete shape and globally as part of an object. This enables superior performance in decomposition quality and supports training-free refinement and editing.

Key concepts

Dual-Frame Coordinate Interleaving
This mechanism interprets each shape latent using two coordinate systems simultaneously: one local to the specific part and one global for the entire object. This forces the model to reason at both scales at once, treating a single part as both a complete shape locally and a component within a larger forest globally.
Segmented Mesh Scaffold
This is an input structure created by combining a coarse mesh from an existing holistic 3D model with segmentation masks derived entirely from the prior's native representation. This scaffold serves as the structural guide that the holistic prior reads without any objective shift, providing necessary information for part-level processing.
Global-Local Interleaved Transformer
This transformer alternates between two types of attention: intra-part attention (where each part is treated as a standalone shape to reuse geometric detail) and inter-part attention (where parts are coordinated as an assembly to preserve structural coherence across the whole object.
Part-Mesh Refinement
The framework can recover fine geometric details for individual parts even when the initial scaffold input is coarse. This is achieved because the dual-frame embeddings retain image conditions in every block, allowing the model to synthesize high-fidelity geometry from lower-resolution structural guidance.

Terminology used across episodes

This episode discusses

The paper

Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation · Read on arXiv

Yiqun Zhao, Binbin Huang, Haobin Duan, Zibo Zhao, Shenghua Gao

University of Hong Kong · Transengram · Tencent · Shenzhen Loop Area Institute

Generating 3D assets with explicit parts requires two properties at once: geometric detail inside each part and structural coherence across the whole object. Recent image-conditioned 3D generation models adopt a structured 3D representation that anchors tokens to explicit spatial positions, capturing both properties, but their output remains a single fused mesh. We ask how a part-decomposed generation model can inherit both properties of these holistic priors, and answer with STRUCTPARTS, a framework whose core technical contribution is a dual-frame coordinate interleaving mechanism: each shape latent is interpreted under both a part-local and an object-global reference frame, and processed through a global-local inter-leaved transformer that simultaneously reuses the prior's geometric detail at the part scale and its structural coherence at the whole-object scale. As its structural input, STRUCTPARTS uses a segmented mesh scaffold, a coarse mesh paired with a face-level segmentation mask kept entirely within the prior's native 3D representation. With only lightweight finetuning, both intra-part detail and inter-part coherence emerge from the same pretrained weights. On PartObjaverse-Tiny, STRUCTPARTS matches or surpasses prior part-decomposed models at a fraction of their training cost, while naturally supporting part-mesh refinement and part-level editing.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Seeing Both the Forest and the Trees".

Jane: The gist: STRUCTPARTS is a framework that allows part-decomposed generation models to inherit both geometric detail and structural coherence from holistic 3D priors through dual-frame coordinate interleaving.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We're diving into "Seeing Both the Forest and the Trees: Reusing Holistic three dee Priors for Part-Decomposed Generation <ref:2610.21850#pg1,Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for>." The authors are Zhao, Huang, Duan, and Zhao. They’re proposing a way to combine geometric detail with structural coherence in part-decomposed generation.

Jane: It's about taking those two properties—the fine detail inside each part and the overall structure of the whole object—and making a model capable of producing both simultaneously using this new mechanism they call STRUCTPARTS.

Lu: The implication is that we don't need to train a part-specific model from scratch; we can adapt an existing holistic prior by just reinterpreting how it’s processed locally and globally, which saves a ton of training time.

Meng: I’m still focused on the structural coherence. If the interleaving isn't right, we risk getting parts that look good individually but don't assemble into a coherent object when we try to put them together later. What exactly is this mechanism doing differently?

Jane: The core technical contribution is the dual-frame coordinate interleaving mechanism, which lets each shape latent be interpreted under both a part-local and an object-global reference frame at the same time.

Tom: So, it’s not just one frame; it’s two frames for every shape latent. And this dual interpretation happens through a global-local interleaved transformer that alternates between intra-part and inter-part attention.

The paper's summary: Jane: So, to summarize the paper, they are addressing the fact that recent image-conditioned three dee generation models already have structured representations that capture both high fidelity global structure and local geometric detail <ref:2610.21850#pg1,recent image-conditioned 3D generation models>.

Tom: But the challenge is extending those priors to part-decomposed generation because those priors were never trained for parts. The authors show how STRUCTPARTS solves this by using this dual-frame mechanism to inherit both required properties from the holistic prior.

Lu: It’s essentially treating each part as a complete shape locally, while also coordinating them as an anchored assembly globally, which is what they call reasoning at both scales simultaneously.

Meng: I see that connecting the geometric detail on a local level with the structural coherence on a global level is exactly what we need for any good three dee reconstruction task <ref:2610.21850#pg1>. It sounds like they’ve found a way to keep those two things balanced during generation.

Jane: They set up their input using this segmented mesh scaffold, which combines a coarse mesh from an off-the-shelf holistic model with a segmentation mask that stays within the prior’s native three dee representation <ref:2610.21850#pg1>.

Tom: So, the scaffold is essentially a bridge; it’s something the holistic prior can read without any objective shift, and it's also a byproduct of those upstream generation stages.

The paper's improvements: Jane: The improvements they suggest focus on how this method handles input and output. They use this dual-frame approach to ensure that in the intra-part attention layers, where F equals L and Zkv equals Z, each part is processed as a standalone shape, reusing the prior’s geometric detail exactly as it was during pretraining.

Tom: And then in the inter-part attention layers, where F equals G and Zkv equals Z, the parts are coordinated as one anchored assembly, which preserves the prior’s structural coherence across the whole object.

Lu: That alternating attention structure is smart because it lets them leverage both strengths of the prior at different stages of processing. It ensures they get the local detail when they need it and maintain global structure when coordinating.

Meng: So what does this mean practically for deployment? Does this complex interleaving add too much computational overhead to run on, say, a production server?

Tom: They show that on PartObjaverse-Tiny, STRUCTPARTS matches or surpasses prior part-decomposed models with substantially lower training costs. That’s a big win when you think about the efficiency gains for running these things.

Conclusion: Jane: To wrap up, the paper with this title, "Seeing Both the Forest and the Trees: Reusing Holistic three dee Priors for Part-Decomposed Generation," shows that we can inherit both geometric detail and structural coherence from a holistic prior using only lightweight finetuning <ref:2610.21850#pg1,Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for>.

Tom: The main implication is that they’ve created a framework that naturally supports part-mesh refinement when the scaffold is coarse, and also allows for training-free merge and split operations through segmentation regrouping.

Lu: This lowers the barrier for nonexperts because you can iterate on three dee content without needing to retrain a part-specific model from scratch; you just have to reinterpret the prior <ref:2610.21850#pg1>.

Meng: I’m still thinking about the robustness. They mentioned that whole-shape performance stays close to clean at one percent geometry noise, which is important for real-world data where things aren't perfect.

Lalam: From my perspective, I see this as a way to make three dee assets much more flexible and controllable through these training-free editing operations, which is a really practical application for generative models <ref:2610.21850#pg1>.

Jane: It’s definitely a significant step in making part-decomposed generation more accessible by cleverly leveraging existing knowledge from holistic models. That’s where we'll leave it for this one.

More episodes

← Home