LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs
cs.CV, cs.LG
Submitted: 2025-06-11
Updated: 2026-11-11
Comments: Accepted and published in IEEE Congress on Evolutionary Computation (CEC) 2026. Copyright 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses
Journal ref: IEEE Congress on Evolutionary Computation (CEC) 2026
DOI: 10.1109/CEC69861.2026.11727362
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities.
Terminology
Abstract
The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.
Sources
- CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-Consistency from a Single Image
- Shap-E: Generating Conditional 3D Implicit Functions
- Advances in 3D Generation: A Survey
- Generative Physical AI in Vision: A Survey
- LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives
- LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
- Structured 3D Latents for Scalable and Versatile 3D Generation
- Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs