3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models".
Jane: Here's the gist of the paper. 3D-PLOT-LLM introduces vocabulary-level part addressing on PartVerse-QA, leads on 3DCoMPaT-GrIn PaPGD, and lifts Objaverse captioning above PointLLM, all with under 1M additional parameters on a frozen point encoder.
Tom: Core Problem and Solution 3D multimodal large language models (3D MLLMs) describe a 3D object as a whole but cannot address, name, or reason about its parts.
Jane: Prior part-aware attempts add segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost.
Tom: The model takes a fundamentally different path by reorganizing the input token stream so that parts become directly addressable through the LLM’s own vocabulary.
Title and authors: Tom: So we're talking about this paper today, "three dee-PLOT-LLM: Part Level Object Tokens for three dee Large Language Models." It sounds like they're trying to fix a problem where these big three dimensional models can describe an object as a whole but can't actually talk about its individual pieces.
Jane: That sounds like it. Like if you show the model a chair, it can describe the chair, but if you ask it to find the legs of that chair and name them separately, it struggles because there’s no direct way for the language model to look at just those specific parts.
Lu: Exactly. Prior methods tried to add segmentation decoders or heavier three dimensional encoders, which just added a lot of extra parameter cost to the whole setup. This paper takes a different route by reorganizing the input stream so that these parts become directly addressable through the language model’s own vocabulary <ref:2606.19828#pg1>.
Meng: I'm worried about that parameter cost if we have to add heavy decoders. We need something lightweight if we want this thing to actually run in production, you know?
Lalam: The core idea here is that they partition the frozen point encoder’s patches into K locally coherent regions and insert a learnable per region marker and a reserved vocabulary token called part k before each region's patch tokens <ref:2606.19828#pg1>.
Tom: That’s interesting because it sounds like they are turning geometry into something the language model already understands, instead of forcing it to learn a whole new way to see the shapes.
Jane: It’s that idea, and then they introduce this Marker Space Refinement module, which conditions those markers on the region's spatial statistics and its neighbors <ref:2606.19828#pg3>.
Lu: That MSR module is key because it supplies that structural context that binds those part k embeddings to what's actually in the region, like knowing if a region is a leg or a seat cushion.
Meng: So they are refining these markers based on where they are spatially and what their neighbors look like. Does that mean we can get better results without having to retrain the whole massive point cloud encoder?
Tom: That’s the selling point, Meng. They say it only adds under one million new trainable parameters on a frozen twenty-two million parameter point encoder, which is way less than what prior part aware three dimensional models needed.
Jane: And they show consistent gains across three different benchmarks: PartVerse-QA, three deeCoMPaT-GrIn, and Objaverse captioning <ref:2606.19828#pg3>. That suggests it’s not just a fluke on one dataset.
Lu: The training protocol involves two stages: first, aligning the projector and markers on six hundred sixty thousand Capthree dee captions for three epochs with the LLM and point encoder frozen <ref:2606.19828#pg3>. Then, they do instruction tuning with seven thousand complex instructions and seven thousand seven hundred complex PartVerse-QA pairs for another three epochs <ref:2606.19828#pg3>.
Title and authors: Meng: Training on those specific pairs sounds like a necessary step to make sure the model learns that part k mapping correctly instead of just memorizing labels from the captions.
Lalam: They specifically include two directions: caption to slots, which is C2S, and slot to caption, which is S2C <ref:2606.19828#pg3>. This bidirectional interface is what they call a major contribution because it lets the LLM both read and write parts through that vocabulary token.
Tom: Reading and writing implies a real understanding of the parts as distinct entities within the language model’s structure, not just treating them like cosmetic labels that you just append to a sentence.
Jane: They even have a probe where they permute those part k identifiers at inference time, and they show the Jaccard score drops significantly, from zero point four five nine down to zero point two zero nine for the PartVerse-QA C2S task <ref:2606.19828#pg3>.
Lu: That drop confirms that the LLM has learned a genuine token region binding instead of just treating those tokens as cosmetic labels, which is pretty important because it validates the interface itself.
Meng: So if we look this from a deployment angle, what does this mean for running these models on real traffic? Does this new structure make inference faster or slower than PointLLM?
Tom: The paper shows that adding the MSR module leads to consistent gains across all six metrics when compared to the no-MSR baseline, and they only use about zero point seven million extra parameters for that refinement <ref:2606.19828#pg3>.
Jane: They also have a really interesting finding on the component ablation where they tested the Marker Space Refinement in two ways—using just statistics and using just adjacency—and combining them gives a gain of plus zero point one nine six Jaccard over the no-MSR baseline <ref:2606.19828#pg3>.
Lu: And they found that marker refinement dominates across metrics, even showing that combining the stats-only and adjacency-only branches exceeds the sum of their individual improvements by plus zero point one five four Jaccard over no MSR <ref:2606.19828#pg3>.
Meng: Okay, so if I’m deploying this to handle complex scenes, I need to know what breaks at scale. The paper says they partition into K regions, and they fit ninety-four point five percent of objects in their aligned PartVerse pool using K equals sixteen slots <ref:2606.19828#pg3>.
Tom: That fixed slot budget of fifteen is a limitation mentioned in the paper because they are leaving the variable K or hierarchical partition up for future work. It’s a constraint on how many distinct parts they can explicitly address at once, right?
Jane: Right. And they also note that for caption to slots, even when you don't perfectly align the partition with the ground truth annotations, like in Figure seven and Figure eight token Jaccard measures slot token agreement anyway <ref:2606.19828#pg3>.
Lu: They also point out that dense decoder methods aren't directly comparable on mesh mIoU because their structural ceiling is higher than what they have achieved with this K equals sixteen based approach <ref:2606.19828#pg3>.
Title and authors: Meng: So for now, it seems the value here is in getting a part aware interface that works without needing a massive segmentation decoder, and it does that with relatively few extra parameters compared to older methods.
Tom: It confirms that turning every geometric region into a first class addressable token refined by Marker Space Refinement is a viable scaling axis complementary to just augmenting whole object captioning.
Jane: It really shows how you can make an object level three dimensional large language model part aware without needing those extra heavy components we used to think were mandatory <ref:2606.19828#pg3>.
Lu: The final result from the paper, "three dee-PLOT-LLM: Part Level Object Tokens for three dee Large Language Models," confirms that this part level interface yields consistent gains across PartVerse-QA, three deeCoMPaT-GrIn, and Objaverse captioning <ref:2606.19828#pg1,3D-PLOT-LLM: Part Level Object Tokens for 3D Large Language Models>.
Meng: Before we wrap up, Lu and Lalam have some final thoughts on where this goes next?
Lu: I think the main direction for future work is exploring a learned variable K or hierarchical partition to handle objects with more complex parts that might exceed the fixed sixteen slot budget.
Lalam: From my perspective, the bidirectional interface means we can really start thinking about how these models can be used to generate parts based on language instructions, not just describe them.
Meng: I'm still focused on deployment cost. If we want to use this in a commercial system, we need to see if that one million parameter overhead is worth the performance lift over the simpler PointLLM setup.
Tom: So, in summary, three dee-PLOT-LLM shows that you can make an object level three dimensional model part aware by using vocabulary tokens tied to spatial regions and refining them with Marker Space Refinement, all while keeping the parameter increase under one million <ref:2606.19828#pg1>.
Jane: It’s a really cool demonstration of how we can get deep structural information into language models in a very lightweight way, and it validates that turning geometric regions into refined tokens is a solid strategy for scaling.
Lu: The future work they point toward is exploring that learned variable K or hierarchical partition to handle objects with more complex parts than the fixed sixteen slot budget allows.
Meng: I'm still focused on deployment cost, though; if we want to use this in a commercial system, we need to see if that one million parameter overhead is worth the performance lift over the simpler PointLLM setup.
Tom: So, three dee-PLOT-LLM confirms that you can make an object level three dimensional model part aware by using vocabulary tokens tied to spatial regions and refining them with Marker Space Refinement, all while keeping the parameter increase under one million <ref:2606.19828#pg1>.
Jane: It’s a really cool demonstration of how we can get deep structural information into language models in a very lightweight way, and it validates that turning geometric regions into refined tokens is a solid strategy for scaling.
The paper's summary: Tom: So we’ve talked about how three dee-PLOT-LLM tackles the problem of making three dimensional models aware of their parts by using vocabulary tokens tied to spatial regions and refining them with a Marker Space Refinement module, all while keeping the new parameter increase under one million.
Jane: That’s right. Basically, this paper shows that instead of just describing an object as one big thing, the model can now say "the chair has a seat, four legs, and a backrest," with those parts being actual tokens you can talk to directly.
Lu: What makes this specific approach different is that they don't just slap on some segmentation decoder or anything heavy. They reorganize the input stream itself so that the parts become these direct addressable tokens within the existing token stream of the large language model.
Meng: So, it’s about reorganizing how we feed the data into the AI, rather than adding a huge new part to its brain structure. That sounds much more manageable for us when we’re trying to ship things to production.
Tom: Exactly. They use this idea of partitioning the frozen point encoder patches into K locally coherent regions, and they insert these learnable per-region markers and reserved vocabulary tokens before each region's patch tokens.
Jane: And then they have this Marker Space Refinement module, which conditions those markers on the spatial statistics and the neighbors of that specific region to give it context.
Lu: That MSR module is where the real magic happens for grounding the parts. It supplies that structural context so the model knows what kind of part it’s looking at based on where it is in space.
Meng: The paper mentions they trained this in two stages, first aligning things on some existing three dimensional object descriptions, and then doing instruction tuning with a specific set of examples called PartVerse-QA. That sounds like a solid way to teach the model the part-to-token relationship without starting from scratch.
Tom: And the results show they reach a Jaccard score of about zero point four five nine on that test set, and they show significant gains when adding this refinement module, with those gains being over six times better than just using a vocabulary approach alone.
Jane: That’s the hook for me. It means we can get this level of part awareness without needing a segmentation decoder or bounding box grammar, which are usually the biggest hurdles in these kinds of models.
Lu: What's really interesting is how they confirmed this wasn't just a cosmetic label thing—they did a label-permutation probe where they showed the language model actually learned a genuine token region binding.
Meng: That’s good, because if it’s not just a cosmetic label, it means the AI understands that those tokens represent real physical components of the object.
Tom: And for practical terms, they say this whole system adds about one million new trainable parameters on top of the frozen point encoder and fine-tuned large language model. That keeps it way smaller than prior part aware three dimensional models.
Jane: So, in short, three dee-PLOT-LLM shows a way to make object models part aware by turning every geometric region into a first class addressable token refined by that spatial context module, and the performance gains are there with only a relatively small parameter cost.
Lu: This approach is really about scaling without massive parameter inflation. The question now is how they can handle objects with more complex parts than their fixed sixteen slot budget allows.
The paper's improvements: Tom: So we’re talking about how they suggest making this system even better than what they built initially by focusing on those refinement modules and how to handle more complex objects.
Jane: Right, so the core improvements center around two main ideas: using that Marker Space Refinement module more deeply and thinking about how to make the partition itself more flexible.
Lu: The first big suggestion is making sure you use both branches of the Marker Space Refinement module—the one based on just spatial statistics and the one based just on adjacency—because combining them lifts the performance even further.
Meng: So, it’s not just about having one way to refine things; it’s about using both ways and seeing that they actually give you a bigger bump in performance, plus zero point one nine six Jaccard over the baseline.
Tom: And on the second front, Lu mentioned thinking about a learned variable K or hierarchical partition. That addresses the limitation we saw—the fixed sixteen slot budget—by letting the model decide how many parts it needs to track for a specific object.
Jane: That means instead of being stuck with exactly fifteen slots, you could have a system that adapts based on how complicated the geometry is, which sounds like it would help with much more detailed scenes.
Lu: Exactly. If you have an object with twenty distinct parts, having a fixed K equals sixteen means you’re losing information about those extra four parts completely. A variable K solves that by letting the model dynamically allocate tokens to the regions that matter most.
Meng: That would be good for deployment because it means the model doesn't have to guess how many slots an object needs; it learns that on the fly. But what’s behind a variable K?
Tom: It requires changing how you set up the initial geometric region partition, so you’re not just doing a fixed grouping anymore. It moves from a static setup to something more adaptive during training or inference.
Jane: So, we’re moving from this fixed K system to something that grows and shrinks based on the object being processed. That really changes how we think about scaling this kind of part aware three dee language model in the real world.
Conclusion: Tom: So we’re wrapping up on "three dee-PLOT-LLM: Part Level Object Tokens for three dee Large Language Models," which basically shows that you can make object models part aware by using vocabulary tokens tied to spatial regions and refining them with Marker Space Refinement, all while keeping the parameter increase under one million.
Jane: It's a really neat demonstration of how we can get deep structural information into language models in a very lightweight way, without needing those extra heavy components we used to think were mandatory.
Lu: The main implication is that part-grounded supervision is now a viable scaling axis alongside just augmenting whole object captioning for building better three dee language models.
Meng: It's good for production because the parameter overhead is low, which means we can actually deploy this kind of part awareness without totally blowing up our memory budget or running into massive training costs.
Lalam: I’m confident in my answer: the work confirms that an object level three dee language model can be made part aware without needing a segmentation decoder or bounding box grammar by turning every geometric region into a first class addressable token refined by Marker Space Refinement.
Tom: And we saw that this system, specifically the MSR module, leads on every PartVerse-QA metric, with the combined gain being over zero point one nine six Jaccard over the no-MSR baseline.
Jane: That structural refinement is what really sets it apart for grounding descriptions in cases where PointLLM was drifting to wrong classes or confabulating parts.
Lu: And they also confirmed that permuting those part identifiers at inference time causes the accuracy to drop significantly, which proves the interface is functional and not just cosmetic.
Meng: It’s interesting because it shows that this vocabulary-level addressing works well even when you use a relatively small slice of PartVerse-QA supervision; matching the baseline performance with only sixteen percent of that data was enough.
Tom: So, the conclusion is pretty clear: reframing parts as a vocabulary problem opens up a lightweight way to get part aware three dee language models without adding massive parameter costs.
Jane: It really shows how you can get deep structural information into language models in a very lightweight way, and it validates that turning geometric regions into refined tokens is a solid strategy for scaling.
Lu: The future work they point toward is exploring that learned variable K or hierarchical partition to handle objects with more complex parts than the fixed sixteen slot budget allows.
Meng: I'm still focused on deployment cost, though; if we want to use this in a commercial system, we need to see if that one million parameter overhead is worth the performance lift over the simpler PointLLM setup.
Tom: So, three dee-PLOT-LLM confirms that you can make an object level three dee language model part aware by using vocabulary tokens tied to spatial regions and refining them with Marker Space Refinement, all while keeping the parameter increase under one million.
Jane: It’s a really cool demonstration of how we can get deep structural information into language models in a very lightweight way, and it validates that turning geometric regions into refined tokens is a solid strategy for scaling.
Jintang Xue, Xinyu Wang, Yixing Wu, Jingwen Chen, C.-C. Jay Kuo
University of Southern California · Ohio State University
cs.CV
Submitted: 2026-06-18
Updated: 2026-11-04
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: 3D-PLOT-LLM introduces vocabulary-level part addressing on PartVerse-QA, leads on 3DCoMPaT-GrIn PaPGD, and lifts Objaverse captioning above PointLLM…
Key concepts
- PartVerse-QA
- This is a new benchmark designed to test if a 3D language model can understand and reason about individual parts of an object. It uses caption-to-slots and slot-to-caption tasks, probing the model's ability to handle part information directly within the LLM's token stream, which is a capability prior models lacked.
- Marker Space Refinement (MSR)
- MSR is a lightweight module that conditions a learnable marker for each spatial region based on its surrounding statistics and neighboring regions. This provides the structural context necessary to bind the region's embeddings to the specific part content, ensuring that the model understands how different parts relate spatially.
- Vocabulary-Level Addressing
- Instead of using heavy segmentation decoders, 3D-PLOT-LLM addresses parts by creating a reserved vocabulary token for each spatial region. The LLM learns to treat these tokens as direct handles for specific object components, allowing it to name and reason about them explicitly.
- Geometric Region Partition
- This unsupervised process groups the patches of the 3D point cloud into K spatially coherent regions. This step is crucial because it establishes a deterministic spatial structure that forms the foundation for inserting region markers and enables the subsequent refinement module to understand local context.
Terminology
Summary
3D-PLOT-LLM introduces vocabulary-level part addressing on PartVerse-QA, leads on 3DCoMPaT-GrIn PaPGD, and lifts Objaverse captioning above PointLLM, all with under 1M additional parameters on a frozen point encoder. p. 9
Core Problem and Solution
3D multimodal large language models (3D MLLMs) describe a 3D object as a whole but cannot address, name, or reason about its parts. Prior part-aware attempts add segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost. p. 0 The model takes a fundamentally different path by reorganizing the input token stream so that parts become directly addressable through the LLM’s own vocabulary. p. 0 3D-PLOT-LLM partitions the frozen point encoder’s patches into K locally coherent regions and inserts, before each region’s patch tokens, a learnable per-region marker and a reserved vocabulary token; a Marker-Space Refinement (MSR) module then conditions each marker on its region’s spatial statistics and adjacency neighbors. p. 0
Architectural Components
The system operates through three main components before the LLM: (i) an unsupervised geometric region partition that groups patches into K spatially coherent regions and exposes per-region statistics and inter-region adjacency; (ii) Marker-Space Refinement (MSR) which conditions each region’s learnable marker on those statistics and adjacency; and (iii) per-region token assembly that interleaves [CLS], reserved tokens, mˆ k, and Gk before the LLM. The total LLM-input length is 545 tokens at K=16. p. 4
Training and Supervision
Training involves a two-stage protocol: Stage 1 (alignment) trains the projector, per-region markers, and MSR on 660K Cap3D captions of Objaverse objects for 3 epochs, with the LLM and point encoder frozen. Stage 2 (instruction tuning) adds the tokens to the tokenizer and trains the projector, per-region markers, MSR, and the LLM on PointLLM’s 70K complex instructions and 77K PartVerse-QA pairs for 3 epochs. p. 5 The PartVerse-QA pairs cover two directions of the interface: caption-to-slots (C2S) and slot-to-caption (S2C). p. 5
Empirical Results and Contributions
3D-PLOT-LLM reaches Jaccard 0.459 and Exact-match 13. p. 078% on the held-out PartVerse-QA split, with MSR lifting both metrics (+64% relative over vocab-only). p. 6 On Objaverse whole-object captioning, adding PartVerseQA at Stage 2 yields +0. p. 065 SBERT and +1. p. 085 GPT-4o over PointLLM, leading on all six metrics with the full model. p. 6 The label-permutation probe confirms that the LLM has learned a genuine token-region binding rather than treating as a cosmetic label. p. 9 MSR leads on every PartVerse-QA metric, with the combined gain being +0. p. 8196 over the no-MSR baseline. p. 6 The model demonstrates structural refinement by recovering correct class and material/color attributes in 3DCoMPaT-GrIn PaPGD cases where PointLLM drifts to wrong classes. p. 8
Limitations and Future Directions
3D-PLOT-LLM addresses parts at K-slot granularity rather than point-wise, and the fixed slot budget K=16 accommodates 94.5% of objects in the aligned PartVerse pool by part count. p. 8 A learned variable-K or hierarchical partition is left to future work, alongside swapping in a stronger backbone without architectural change and extending to scene-level part understanding. p. 8 The C2S direction is in principle retrofittable by projecting a method’s predicted masks or boxes onto our K=16 slot grid, but this is not run in this paper due to engineering layers and quality conflation. p. 6
Conclusion
Reframing addressable parts as a vocabulary problem opens a lightweight, decoder-free route to part-aware 3D language models, with part-grounded supervision a viable scaling axis complementary to whole-object caption augmentation. The model adds ≈ 0. p. 28M architectural parameters on top of the shared frozen 22M encoder and fine-tuned LLM, an order of magnitude below prior part-aware 3D MLLMs. p. 7 The work confirms that an object-level 3D MLLM can be made part-aware without a segmentation decoder or bounding-box grammar, by turning every geometric region into a first-class addressable token refined by Marker-Space Refinement (MSR). p. 9 The paper releases the code, checkpoints, the aligned PartVerse-QA benchmark, and the partition cache upon publication. p. 8
References
Ahmed Abdelreheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. Satr: Zero-shot semantic segmentation of 3d shapes. p. 9 In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15166–15179, 2023. p. 9 Mahmoud Ahmed, Junjie Fei, Jian Ding, Eslam Mohamed Bakr, and Mohamed Elhoseiny. p. 9 Kestrel: 3d multimodal llm for part-aware grounded description. p. 9 In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8973–8983, 2025. p. 9 Satanjeev Banerjee and Alon Lavie. p. 9 Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. p. 9 In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. p. 9 Omid Bonakdar and Nasser Mozayani. p. 9 Mmpart: Harnessing multi-modal large language models for part-aware 3d generation. p. 9 arXiv preprint arXiv:2509. p. 916768, 2025. p. 9 Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. p. 9 Ll3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. p. 9 In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26428–26438, 2024a. p. 9 Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. p. 9 Grounded 3d-llm with referent tokens. p. 9 arXiv preprint arXiv:2405. p. 910370, 2024b. p. 9 Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. p. 9 Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. p. 9 See https://vicuna. p. 9 lmsys. org (accessed 14 April 2023), 2(3):6, 2023. p. 9 Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. p. 9 Objaverse: A universe of annotated 3d objects. p. 9 In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142–13153, 2023. p. 9 Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub, and Ian Reid. p. 9 3d-llava: Towards generalist 3d lmms with omni superpoint transformer. p. 9 In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3772–3782, 2025. p. 9
Appendix A: Geometric region partition
This appendix expands §3.1 by detailing the partition procedure which takes the N=512 patch centers and patch features from a frozen Point-BERT and returns a grouping into K=16 regions. p. 3 The algorithm has four stages: (S0) Patch connectivity graph, (S1) Superpixel growth, (S2) Balanced merge to K, and (S3) Boundary smoothing and connectivity cleanup. p. 3
Appendix B: Hyperparameters and compute
All runs use 2×A100 80GB GPUs with bf16 mixed precision. Stage 1 trains for 3 epochs on 660K Cap3D captions, and Stage 2 trains for 3 epochs on the union of PointLLM’s complex instructions and PartVerse-QA pairs. p. 5
Appendix C: Per-run variance and significance tests
This appendix backs the main-paper Tables 1 (Objaverse) and 2 (PartVerse-QA) with 5-run mean ± std and pairwise Welch’s two-sample t-tests.
Appendix D: Evaluation prompt templates
This appendix lists the prompt strings used for training supervision and evaluation, including specific templates for caption-to-slots and slot-to-caption tasks.
Appendix E: PartVerse-QA benchmark construction and statistics
PartVerse-QA is constructed on top of the part annotations of the PartVerse mesh dataset, where semantic part annotations are re-projected onto the 8192-point Objaverse sample. The construction pipeline includes point re-projection, semantic-to-slot mapping, IoU filtering, and query instantiation.
Appendix F: Model-level partition-agnostic mesh mIoU
This appendix extends the partition statistic to a model-level mIoU computed against mesh-derived per-point masks rather than slot-set ground truth, providing a partition-agnostic view of the architectural progression.
Appendix G: MSR branch ablation
This appendix ablates the two branches of the Marker-Space Refinement module—stats only and adjacency only—to show they are complementary, with combining them lifting PartVerse-QA Jaccard by +0.154 over adjacency-only. p. 0
Appendix H: Probing token-region binding
This appendix confirms that the tokens function as a real addressing handle rather than a cosmetic label by permuting the identifiers at inference while keeping markers and patches unchanged.
Figure 3: Qualitative examples of part-aware tokenization
Figure 3 illustrates two regimes, showing how Top shows a single vocabulary carries both C2S and S2C across granularities, and Bottom shows PointLLM drifting to wrong classes while 3D-PLOT-LLM stays on-class and recovers multi-part materials/colors.
Table 4: Component ablation
This table isolates the four architectural pieces, showing that MSR (row 6, ours) leads on every PartVerse-QA metric and on Objaverse SBERT. The final result shows MSR full achieves Jaccard 0. p. 7459 and S2C GPT-4o to 44. p. 768 at only 0.7M extra parameters compared to the no-MSR baseline. p. 7
Table 5: Per-row mean ± std
This table unifies the variance panels for Objaverse and PartVerse-QA S2C metrics, showing that 3D-PLOT-LLM (full) leads on all six metrics under matched data. The pairwise significance tests show that the architectural contribution is significant across several metrics, such as METEOR for Objaverse (p ≈ 0. p. 604).
Table 6: Mesh mIoU on PartVerse-QA C2S
This table shows that the architectural ranking is preserved under the partition-agnostic GT, with MSR full achieving a mean mesh mIoU of 0.399. The final finding confirms that dense-decoder methods are not directly comparable on this metric as their structural ceiling exceeds any K=16-based ceiling.
Table 7: MSR branch ablation
This table shows that the stats-only and adjacency-only branches are complementary, with combining them exceeding the sum of individual-branch improvements over the no-MSR baseline by +0.196 Jaccard. p. 5 The asymmetry across metrics is interpretable, as stats-only leads on Objaverse captioning while adjacency-only leads on PartVerse-QA Jaccard. p. 6
Table 8: Label-permutation probe on PartVerse-QA C2S
This table confirms that the LLM has learned a genuine token-region binding because permuting the identifiers at inference causes Jaccard to drop from 0.459 to 0.209. This evidence suggests the interface is functional, and the model learns a genuine token-region binding instead of treating as a cosmetic label. p. 6
Table 9: Design space of refinement module
This table shows that Marker refinement (MSR) dominates, lifting C2S Jaccard from 0.263 to 0.459 (+0.196), and MSR achieves the best result with only 0. p. 67M extra parameters compared to the vocab-side variant requiring 3. p. 72M parameters for vocab refinement. p. 7 The asymmetry is consistent across metrics, indicating that marker refinement is more parameter-efficient and structurally informative than vocabulary refinement in this setup. p. 2
Table 10: PartVerse-QA supervision scaling
This table shows that 3D-PLOT-LLM with only 16% of the PartVerse-QA pool already matches the no-refinement Markers baseline, and it rises to Jaccard 0.459 at full data usage. p. 5 This indicates that a small slice of part-grounded supervision is enough to reach a baseline-equivalent grounding signal, and the model's performance on Objaverse captioning remains flat across different fractions of PartVerse supervision.
Figure 7: PartVerse-QA C2S dialogues (extended)
Figure 7 extends the Caption-to-Slots (C2S) interface beyond the single Jaccard = 1.00 panel in the main figure, showing that token-Jaccard measures slot-token agreement and is independent of how the partition aligns with the PartVerse annotation. p. 1
Figure 8: PartVerse-QA S2C dialogues (extended)
Figure 8 extends the Slots-to-Caption (S2C) direction in the same way, illustrating that token-level set agreement is attainable even when the green partition highlight does not visually cover the full PartVerse part annotation.
Figure 9: 3DCoMPaT-GrIn full grounded descriptions
Figure 9 reproduces the three abridged 3DCoMPaT-GrIn panels at full length, showing that PointLLM drifts to wrong object class and confabulates parts, while 3D-PLOT-LLM recovers the correct class and multi-material attributes.
Figure 10: 3DCoMPaT-GrIn full grounded descriptions
Figure 10 shows that PointLLM drifts to “planter” and confabulates parts, whereas 3D-PLOT-LLM recovers the correct class and multi-material attributes in the barbecue grill case.
Figure 12: Objaverse whole-object captioning
Figure 12 shows that PointLLM fails with class drift and structural hallucination, while 3D-PLOT-LLM stays anchored to both class and attributes in each case.
Figure 13: Failure modes visualized across the three benchmarks
Figure 13 visualizes failure modes, showing that both models collapse the GT’s six distinct material/color combinations toward a single metal color in the shower case, but 3D-PLOT-LLM still recovers one non-metal element. The paper concludes that this part-aware interface is a viable scaling axis complementary to whole-object caption augmentation. p. 9
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Part-aware addressing without canonical decomposition: The model reserves per-object vocabulary tokens to tied to an unsupervised, deterministic spatial partition, allowing the LLM to learn part identity as
compositions of these region tokens.
-
Marker Space Refinement (MSR): This lightweight residual module conditions each region's learnable marker on its
region’s spatial statistics and adjacency neighbors,
supplying thestructural context that binds embeddings to region content.
-
Bidirectional part-level interface: The model enables the LLM to both read and write parts through a vocabulary token, as demonstrated by its capability to follow prompts that refer to parts by token.
-
PartVerse-QA benchmark: This bidirectional C2S and S2C benchmark probes a
capability no prior part-aware 3D MLLM natively supports,
allowing evaluation ofvocabulary-level part addressing in a single LLM token stream.
-
Architectural efficiency: The system achieves these gains with
under 1M new trainable parameters on a frozen point encoder, an order of magnitude below prior part-aware 3D MLLMs,
which is critical for scaling without massive parameter inflation.
Sources
- MMPart: Harnessing Multi-Modal Large Language Models for Part-Aware 3D Generation
- Grounded 3D-LLM with Referent Tokens
- Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
- SpatialLM: Training Large Language Models for Structured Indoor Modeling
- Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
- Part-X-MLLM: Part-aware 3D Multimodal Large Language Model
- Uni3D: Exploring Unified 3D Representation at Scale
- LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs