Semantic Latent Decomposition with Normalizing Flows for Face Editing

The gist: SDFlow proposes a novel framework for face editing that utilizes semantic decomposition in the original latent space of StyleGAN through continuous conditional normalizing flows to eliminate attribute entanglement and enhance manipulation controls.

Introduction and Motivation

Face editing aims to change desired facial attributes while maintaining image quality and other undesired characteristics, which is possible by navigating the semantically abundant latent space of StyleGAN [1]. Current methods often struggle with complicated navigation due to the entanglement among different attributes in this latent space [1]. Existing approaches, such as supervised and unsupervised methods, fall short of producing precise user-desired editing without human annotations [1]. Specifically, linear interpolation methods may fail when the linear assumption does not hold, and nonlinear methods have not explicitly disentangled different facial attributes due to the use of binary attributes [1].

SDFlow Framework

SDFlow introduces a framework with two key components: (i) a semantic encoder and (ii) a flow-based transformation module [2]. The semantic encoder produces semantic variables for different attributes directly from input faces, and under these conditions, the flow-based transformation module maps the latent codes into semantic-irrelevant variable in Gaussian distribution conditioned on the learned semantic variables [1]. To achieve disentanglement between variables, SDFlow jointly optimizes these two components under a mutual information framework [18]. Moreover, a pre-trained attribute classifier is distilled to inject supervision signals of human annotations into the semantic variables [1].

Model Architecture

The flow-based transformation module employs conditional continuous normalizing flows (CNFs) [19] optimized by neural ODEs where the differential equation is defined as dz/dt = Φθ(z(t), s, t) [20]. The inverse inference of CNFs is defined as z(t1) = z(t0) + Z t1/t0 Φθ(z(t), s, t)dt [2]. The loss function to optimize the flow-based transformation module is Lnll = − log p(z(t0)) + Z t1/t0 Tr ∂Φ/∂z(t) [3]. Face editing can be achieved by only changing semantic variables s in the reverse order of Eq. 2: we = z(t1) − Z t1/t0 Φθ(z(t), sˆ, t)dt [4].

The Semantic Estimator (Es) employs a backbone of ResNet-34 [21] and is trained to estimate the semantic variables, which are jointly trained with the flow-based transformation module [3]. To inject supervision signals of human annotations into the semantic variables, a pre-trained attribute classifier is distilled to Es via Lkd = 1/N Σ i C(I) − Es(I) 2 2 [5].

Disentangled Learning and Training

To better regularize the components, SDFlow utilizes a disentangled learning strategy in [18] where semantic variables s are randomly manipulated to sˆ sampling from [0, 1], and then the faces are generated according to Eq. 4 [3]. If s are disentangled well, Es should be able to reconstruct sˆ [3]. The mutual information between variables is maximized through Lmi = 1/N Σ i Es(Iˆ) − Es(I) 2 2 [6], where Iˆ = G(we) is the edited faces. Additionally, to compress unnecessary changes, the Euclidean distance between original and edited latent codes is restricted by Lreg = w − we 2 2 [7]. The overall training objective minimizes the sum of all losses: L = Lnll + Lkd + Lmi + Lreg [8]. During inference, only desired semantic variables are changed while keeping z and unrelated variables fixed.

Experimental Results

The results demonstrate that Es can learn meaningful values for different attributes; for instance, in the case of Eyeglasses, there is a clear decision boundary since it is distinct to identify the faces with/without glasses [2]. However, Age exhibits overlapping, which conforms to the practical scenario of facial aging process [3]. Qualitative evaluation shows that SDFlow successfully disentangles highly correlated attributes and achieves best results in multi-attribute editing compared to methods like InterfaceGAN [8], Latent Transformer [9], and StyleFlow [7] [4]. Quantitative results show that SDFlow outperforms competitors across metrics including editing accuracy, attribute preservation accuracy, and identity preservation when performing single or multiple attribute editing towards the opposite attribute until the editing accuracy reaches 99% for identity preservation in some cases [4]. Ablation studies confirm that adding Lreg can achieve significant improvements over baseline, indicating that only inverse inference during optimization cannot preserve identity/attributes [5].

Conclusion

SDFlow introduces a semantic decomposition in the original latent space of StyleGAN using continuous conditional normalizing flows and a semantic encoder to estimate proper semantic variables [1]. This framework enables nonlinear editing by mapping latent codes into semantic-irrelevant variables under the conditions of these variables [2]. A disentangled learning strategy is adopted to eliminate the entanglement among attributes [1]. Extensive experiments demonstrate that SDFlow can generate photo-realistic results with better disentanglement.

References

[1] Tero Karras, Samuli Laine, and Timo Aila, “A stylebased generator architecture for generative adversarial networks,” in CVPR, 2019 [5].

[2] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, “Analyzing and improving the image quality of stylegan,” in CVPR, 2020 [1].

[3] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, “Generative adversarial nets,” in NeurlIPS, 2014 [3].

[4] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel CohenOr, “Encoding in style: A stylegan encoder for imageto-image translation,” in CVPR, 2021 [4].

[5] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or, “Designing an encoder for stylegan image manipulation,” TOG, vol. 40, no. 4, pp. 1–14, 2021 [5].

[6] Rameen Abdal, Peihao Zhu, John Femiani, Niloy Mitra, and Peter Wonka, “CLIP2StyleGAN: Unsupervised extraction of stylegan edit directions,” in SIGGRAPH, 2022 [6].

[7] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka, “Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows,” TOG, vol. 40, no. 3, pp. 1–21, 2021 [7].

[8] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou, “Interpreting the latent space of gans for semantic face editing,” in CVPR, 2020 [8].

[9] Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier, “A latent transformer for disentangled face editing in images and videos,” in ICCV, 2021 [9].

[10] Huiting Yang, Liangyu Chai, Qiang Wen, Shuang Zhao, Zixun Sun, and Shengfeng He, “Discovering interpretable latent space directions of GANs beyond binary attributes,” in CVPR, 2021 [10].

[11] Zongze Wu, Dani Lischinski, and Eli Shechtman, “Stylespace analysis: Disentangled controls for stylegan image generation,” in CVPR, 2021 [11].

[12] Yifan Yuan, Siteng Ma, Hongming Shan, and Junping Zhang, “DO-FAM: Disentangled non-linear latent navigation for facial attribute manipulation,” in ICASSP, 2023 [12].

[13] Zhizhong Huang, Siteng Ma, Junping Zhang, and Hongming Shan, “Adaptive nonlinear latent transformation for conditional face editing,” in ICCV, 2023 [13].

[14] Erik Hark ¨ onen, Aaron Hertzmann, Jaakko Lehtinen, and ¨ Sylvain Paris, “Ganspace: Discovering interpretable GAN controls,” in NeurlIPS, 2020 [14].

[15] Yujun Shen and Bolei Zhou, “Closed-form factorization of latent semantics in GANs,” in CVPR, 2021 [15].

[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark et al., “Learning transferable visual models from natural language supervision,” in ICML, 2021 [16].

[17] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski, “StyleCLIP: Textdriven manipulation of stylegan imagery,” in ICCV, 2021 [17].

1. A semantic decomposition framework for StyleGAN latent space allows users to only change semantic variables while keeping z and unrelated variables during inference, enabling precise manipulation of specific facial attributes without affecting others.

2. The system can perform multi-attribute editing by jointly optimizing losses, as shown in Figure 4, allowing for disentangled and accurate face editing where the user can select the best editing strength.

3. The proposed SDFlow achieves superior quantitative results over competitors, with performance metrics such as Editing Accuracy(%), Attribute Preservation Accuracy, and Identity Preservation reaching higher levels compared to existing methods.

4. The semantic encoder can be trained using a pre-trained attribute classifier via the loss function: Lkd = 1/N sum i C(I) - Es(I) squared, which injects supervision signals of human annotations into the semantic variables, improving editing control.