WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusion

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "WorldMesh: Generating Navigable Multi-Room 3D Scenes via Mesh-Conditioned Image Diffusion".

Jane: Here's the gist of the paper. WorldMesh presents a geometry-first approach that decouples large-scale 3D scene synthesis into constructing an explicit mesh scaffold for global spatial consistency and using image synthesis conditioned on this scaffold for local photorealistic appearance, enabling scalable, arbitrarily-sized multi-room 3D worlds How it works The core idea of WorldMesh is to decouple the complex task of large-scale 3D scene synthesis into its fundamental challenges: global spatial consistency and local photorealistic appearance From an input text prompt, the process begins by constructing a mesh scaffold that establishes the output scene’s layout and geometric structure This mesh scaffold is then used as a scaffold for conditioned image synthesis to encourage multi-view consistency across both local (e.g., around objects) and global (e.g., across rooms) scales Synthesized views are optimized into 3D gaussian splats representing a navigable 3D world through view synthesis Mesh Scaffold Construction The construction of the mesh scaffold involves several key steps to define the scene's geometry First, from the input text prompt T, a floor plan layout L is generated in JSON text format using Claude Opus 4.6

2: This layout L specifies structural scene metadata such as wall thickness and room definitions Based on L, the 3D structure Mstruct is instantiated by insetting floor polygons by the wall thickness and extruding them vertically to the ceiling height Matching edges between rooms are detected within a 0.01 m tolerance to ensure continuous walls, and door and window openings are carved using boolean subtraction Finally, thin planar extrusions are added to form floors and ceilings, completing the structural mesh Mstruct Object Instantiation and Mesh Texturing To populate the scaffold with objects in realistic layouts, image synthesis is leveraged based on the constructed floor plan scaffold Mstruct For each room, Mstruct is rendered from an initial coverage camera pose c0 to produce a depth map D0 This depth map D0 conditions a photorealistic image generation model (Flux2-Klein

6: ) guided by the text prompt T and architectural context L Individual 3D object instances are then extracted from these generated images using SAM3

10: and reconstructed into 3D representations using SAM-3D-Objects

12: The structural elements of the scaffold are textured through projective texture accumulation, where each generated view's pixel colors are projected back onto Mgeo Mesh-Conditioned Appearance Synthesis The constructed mesh scaffold M now serves as a guide for synthesizing photorealistic appearance anchored to M An image synthesis model Φ is employed, conditioned on depth and color renderings of M to enforce structural fidelity while allowing the model to generate realistic materials, lighting, and fine-grained visual detail A viewpoint generation strategy is used where cameras are sampled iteratively via greedy nearest-neighbor rotational similarity to encourage consistent appearance synthesis Each output image Ii is synthesized as Ii = Φ(Xi, Ik(i), T), where Ik(i) is the style reference from the previously generated image in the sequence Image verification is performed by comparing implied geometry to M by estimating a depth map D'i and comparing its Canny-detected edges E'i against edges EM i extracted from M to evaluate structural fidelity Geometry-Regularized 3DGS Scene Reconstruction For all successfully validated views, a 3DGS representation of the output scene S is optimized using the synthesized images Ii, their camera poses ci, and depth maps Di rendered from M The training loss combines photometric reconstruction with depth regularization: L = (1 - λs) L 1(I i, ˆI i) + λs L DSSIM(I i, ˆI i) + λd D i - D S i, where the depth term prevents geometric drift and ensures that S preserves the architectural structure of M Results and Evaluation The method demonstrates superior 3D consistency compared to baselines, outperforming methods like WorldExplorer

33: , SpatialGen

15: , and DreamScene360

50: across metrics such as cross-view depth reprojection error and mesh-depth/normal alignment The perceptual study confirms that WorldMesh outperforms all baselines across object consistency, scenestructure consistency, and overall visual fidelity Furthermore, the method shows graceful scaling with the number of rooms (up to 12 rooms) while maintaining stable 3D-consistency and quality The navigability analysis confirms near-full traversability, with the largest 4-connected free-space component covering an average of 96.

Title and authors: Tom: So we’ve talked about the title and authors, and now let’s get into what WorldMesh actually does, based on their summary. The paper says they present the first approach for scalable multi-room three dee synthesis from text prompts by building a mesh scaffold to represent scene geometry <ref:2603.22972#pg1>.

Jane: They explain that this scaffold is then used as a guide for mesh-conditioned appearance generation, which means the scaffold provides structural information like depth and object geometry to condition the image synthesis process, ensuring consistency across both local and global scales.

Lu: The main contribution they highlight is introducing this mesh-guided appearance synthesis where the rendered scaffold acts as a structural backbone signaling depth, object geometry, and textures to be used as a condition for consistent image synthesis.

Meng: Okay, so if I'm an engineer building this up in production, that means we have to build a reliable system that can accurately reconstruct three dee geometry from just text and then use that structure to guide the visual generation <ref:2603.22972#pg1>. What’s the cost of generating this mesh scaffold for a complex scene?

Lalam: The paper says they reconstruct objects as part of the mesh scaffold based on synthesized images, which is intended to preserve realistic layouts and high object richness, showing that mesh conditioning can scale photorealistic three dee scene generation to arbitrarily large, multi-room environments with robust three dee consistency <ref:2603.22972#pg1>.

The paper's summary: Tom: To summarize the summary they give us, WorldMesh starts by constructing a mesh scaffold from the text prompt—think of this as defining the blueprint of your entire scene's layout first. They then use that scaffold to anchor image synthesis so that every view generated has to respect that underlying structure, which is how they achieve multi-view consistency across rooms.

Jane: It’s about using an explicit geometric representation, the mesh scaffold, to condition diffusion models. This moves away from just letting the model guess the geometry every time, which is what often causes inconsistencies in larger scenes.

Lu: They specifically detail how they use this scaffold to populate it with objects by extracting individual three dee object instances from generated images and reconstructing them using SAM-three dee-Objects, which is supposed to keep the layouts realistic <ref:2603.22972#pg1>.

Meng: Reconstructing objects on top of a mesh scaffold sounds computationally expensive. How do you handle that when you’re trying to generate views iteratively? Does the fidelity of the object extraction depend heavily on the quality of that initial depth map D0?

Lalam: The paper says they use a viewpoint generation strategy where cameras are sampled iteratively via greedy nearest-neighbor rotational similarity, which is designed to encourage consistent appearance synthesis across those views.

The paper's improvements: Tom: Now let’s look at the specific improvements they propose beyond the initial concept. They introduce mesh-guided appearance synthesis where the scaffold provides a structural backbone signaling depth, object geometry, and textures to condition image synthesis.

Jane: They also use an iterative viewpoint generation strategy where cameras are sampled via greedy nearest-neighbor rotational similarity to encourage consistent appearance synthesis, which is designed to create a gradual outward spiral where each camera has a maximally similar style reference.

Lu: They integrate depth-based validation during image synthesis by comparing Canny edge detection between generated images and depth maps rendered from the mesh scaffold, measuring "depth edge recall" to ensure structural fidelity. This means an image only passes if its recall exceeds a threshold.

Meng: So, you’re adding a geometric check into the generation loop? That sounds like it adds latency to every single view synthesis step. How does that impact your deployment pipeline at scale?

Lalam: They also employ geometry-regularized three dee Gaussian Splatting optimization using a loss function that combines photometric reconstruction with depth regularization, specifically using the term "depth term prevents geometric drift, a common failure mode in three deeGS under sparse views <ref:2603.22972#pg1>."

Conclusion: Tom: So to wrap up, WorldMesh claims superior three dee consistency compared to baselines like WorldExplorer and SpatialGen across metrics like cross-view depth reprojection error and mesh-depth/normal alignment <ref:2603.22972#pg1>. They also show graceful scaling up to twelve rooms while maintaining stable three dee consistency and quality <ref:2603.22972#pg1>.

Jane: The perceptual study confirms that WorldMesh outperforms all baselines across object consistency, scenestructure consistency, and overall visual fidelity. It seems like a very strong result for environment-scale generation when you look at the navigation analysis too, showing near-full traversability.

Lu: From my view, the key is that they've addressed the problem of maintaining geometric structure by anchoring appearance generation to an explicit scaffold, which is what really lets them scale photorealistic three dee scene generation to arbitrarily large environments with robust three dee consistency <ref:2603.22972#pg1>.

Meng: I’m still thinking about the cost and complexity of building that entire pipeline—from floor plan generation by Claude Opus four point six to the final geometry-regularized optimization—but if this proves scalable, it changes how we think about putting complex virtual worlds into production environments.

Lalam: Lalam thinks the integration of depth-based validation during synthesis is a smart way to guarantee that what you’re seeing visually actually matches the underlying structure M, and their WorldMesh paper is a significant step toward generating truly immersive three dee worlds from text prompts <ref:2603.22972#pg1>.