COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing

The gist: This paper proposes a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently to achieve precise and disentangled editing of 3D human head avatars.

Introduction

Recent 3D Gaussian Splatting (3DGS) GANs for human heads synthesize and render photorealistic 3D models in real-time and offer a vast variety in identity and appearance However, controlling specific semantic attributes such as hair color or glasses remains challenging, as edits in the entangled latent space often induce unintended changes in identity or appearance Although there are several methods that aim to disentangle the latent space post training by estimating directions that only modify certain features, these methods cannot guarantee complete disentanglement and often require pre-trained classifiers In our approach, we propose a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently This allows for changing the latent vector for one region while keeping the remaining parts fixed Further, we achieve this separation using only sparse information such as the hair or skin color, eliminating the requirement of segmentation masks or geometric priors, often seen in prior work To ensure matching shape and lighting conditions during editing, we allow minimal shared information via context tokens between the independent generators These tokens even allow us to control the shape and light, without any prior annotation Compared to existing works on GAN-based generation and editing, our method shows better disentanglement, more precise editing control, and competitive visual quality.

Method

The proposed method consists of two main ideas: a compositional GAN architecture to produce disentangled 3D components and several techniques to improve the quality and consistency when mixing different components during inference.

3.1 Compositional GAN Architecture

To enable independent manipulation of semantic attributes, we construct the generator as a composition of multiple sub-generators. We allocate a dedicated 3DGS generator, similar to CGS-GAN [4], to each semantic region, i.e., hair, face, glasses and torso. The complete 3D head is then formed by composing the Gaussian primitives into a single coherent scene representation. The composed 3D scene is then rendered into a 2D image using a differentiable 3DGS renderer, which is forwarded to the discriminator. To enforce componentwise specialization, we supply region-specific attribute annotations only to the corresponding sub-generator.

Additive Components

For components that are not always present during generation, like glasses, we use a boolean flag that describes the presence or absence of this component. If the component should be present, we generate it and add it to the composition, while we hide it by subtracting a large value from the Gaussian opacity when it is absent. At the same time, we provide the boolean flag as 0 or 1 to the discriminator, which penalizes the generator if the respective component is incorrectly missing or appearing. As the boolean flag is exclusively provided to the respective sub-generator, it implicitly learns to become responsible for generating this component. Similarly, if the sub-generator starts to generate more than necessary, for example, if the glasses generator produces the eyes or the skin, those features will be missing from the composition when the output is hidden.

Permanent Components

To synthesize components always present during generation (like the skin), we have to define a description that exclusively relates to the respective region. For our purpose, we select the color as a suitable description to separate skin, hair, and the torso. Additionally, this gives us the ability to control the color next to the overall latent vector. Specifically, we use a sparse 10-bin RGB histogram aggregated over the respective head region This color histogram is then applied to modulate the color attribute of the Gaussian primitives inside the respective sub-generator. This is done in the color mod decoder layer, shown in Figure 3, which converts the features of the point generator into Gaussian color attributes. Similar to the additive component, we provide the same color information to the conditional discriminator, which then produces a loss if the target color is not met. This sparse information is sufficient to cleanly separate the components of our 3D human head into hair, skin and torso.

Discriminator

We employ the conditional discriminator architecture of EG3D [7] with minimal changes. In addition to the camera conditioning, we concatenate all color information and the boolean glasses flag into one single vector and multiply it to the discriminator features. In contrast to existing compositional GANs [15, 26], our discriminator receives the entire image instead of a masked region, as the semantic separation is entirely handled within the generator. We consider this design advantageous, as segmentation masks are often not precise, while additionally mask-based supervision is difficult to apply during early training stages where the generator produces less structured outputs.

Global Consistency

To enforce better compatibility during inference, we propose several techniques that improve global consistency in shape and light, while also preparing the generator for novel component combinations already during training.

Latent Mixing

The most simplistic way to enforce better compatibility during inference is by forwarding mixed components during training. Specifically, with a probability of 20%, we randomly switch the latent vector in one sub-generator.

Shape Context

To counteract inconsistencies in head size and overall shape when recombining novel head combinations already during training, we provide a shared low dimensional random shape context latent to all sub-generators, that is never mixed. This shared latent serves as a global conditioning signal, encouraging agreement on head scale and overall shape To ensure that this information will only be used for scaling and reshaping the 3D head, we input the vector at the final synthesis layer, where the features are decoded into Gaussian primitive positions (wShape ∈ R 512 in Fig. 3) By applying the shape context exclusively at this stage, it can only modify the spatial coordinates.

Light Context

Similarly to inconsistencies in shapes and sizes, all sub-generators have to agree on a shared global lighting condition. Therefore, we provide the sub-generators with a shared low dimensional lighting latent code (see wLight ∈ R 512 in Fig. 3) that is never mixed during latent mixing Intuitively, to achieve this, we would simply modulate the output colors in the decoder layers, similarly to the shape context vector. However, as the final color decoder layer provides very limited capacity, it cannot model complex shadows or reflections For this reason, we condition the input latent vector with the light context vector As this allows the generator to modify everything, and not just the lighting, we then apply a regularization that penalizes the generator if two renderings from the same latent, but with different light context, differ too much.

1. Bold Header: Semantic Component Decomposition for Disentangled Editing

The system can achieve precise and disentangled editing of 3D human head avatars by utilizing a generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently. This allows users to change the latent vector for one region while keeping the remaining parts fixed.

2. Bold Header: Implicit Attribute Control via Sparse Conditioning

The architecture enables control using only sparse information such as the hair or skin color instead of requiring explicit segmentation masks or geometric priors. This is achieved by modulating 3DGS color attributes, which allows for very good generalization for novel colors by learning a direct mapping from color to 3DGS features, unlike conditional GANs that break when provided with out-of-distribution conditions.

3. Bold Header: Global Consistency Enforcement through Shared Context Latents

To ensure components fit together during inference, the system provides shared learned features via shape context vectors and a shared low dimensional lighting latent code to all sub-generators. This allows users to control the shape and lighting without using any prior annotation, ensuring that changes in one component do not cause unintended modifications elsewhere.

4. Bold Header: Geometric Integrity Preservation via EMA Regularization

The system incorporates an Exponential Moving Average Regularization loss that penalizes deviations of generated face geometry from an EMA face geometry, which avoid[s] generating incomplete faces that stop just below the matching hairline. This ensures structural coherence when mixing novel component combinations during inference.

5. Bold Header: High-Fidelity, Real-Time Editing Performance

The proposed method demonstrates superior editing control, achieving a recall of about 95% for adding specific attributes like glasses and almost 100% recall for toggling components. This is quantified by showing that the LPIPS distance in unrelated regions after editing is minimal (e.g., Ours 0.016 vs. GANSpace [18] at 0.254).

Recent 3D Gaussian Splatting (3DGS) GANs for human heads synthesize and render photorealistic 3D models in real-time and offer a vast variety in identity and appearance. However, controlling specific semantic attributes such as hair color or glasses remains challenging, as edits in the entangled latent space often induce unintended changes in identity or appearance. Although there are several methods that aim to disentangle the latent space post training by estimating directions that only modify certain features, these methods cannot guarantee complete disentanglement and often require pre-trained classifiers. In our approach, we propose a new generator architecture that synthesizes components, such as hair, skin, glasses, and torso, completely independently. This allows for changing the latent vector for one region while keeping the remaining parts fixed. Further, we achieve this separation using only sparse information such as the hair or skin color, eliminating the requirement of segmentation masks or geometric priors, often seen in prior work. To ensure matching shape and lighting conditions during editing, we allow minimal shared information via context tokens between the independent generators. These tokens even allow us to control the shape and light, without any prior annotation. Compared to existing works on GAN-based generation and editing, our method shows better disentanglement, more precise editing control, and competitive visual quality.