DivAvatar: Diverse 3D Avatar Generation with a Single Prompt
summary
The gist
Text-to-Avatar generation has recently made significant strides due to advancements in diffusion models and DivAvatar is a novel framework that generates diverse avatars from a single text prompt…
In short
DivAvatar introduces a framework to generate diverse 3D avatars from a single text prompt by finetuning a pretrained 3D generative model, EVA3D. It achieves diversity through novel techniques like noise sampling during training and a semantic-aware zoom mechanism. This method produces avatars with greater appearance variety and natural poses compared to existing methods.
Key concepts
- EVA3D
- A pretrained 3D generative model used as the foundation for DivAvatar. It is built using a compositional human NeRF representation, which provides a strong prior for realistic human appearances learned from its training data.
- Noise Sampling Technique
- A critical training strategy where the model alternates between sampling random noise and fixed noise during finetuning. This technique is essential for reviving the GAN's ability to produce a wide range of distinct avatar appearances, overcoming limitations inherent in standard SDS.
- Semantic-Aware Zoom Mechanism
- A design feature that ensures textual fidelity by using a comprehensive set of angles during generation. This mechanism helps prevent appearance discrepancies when the generated avatar's look does not match the descriptive text prompt.
Terminology used across episodes
This episode discusses
- DivAvatar: Diverse 3D Avatar Generation with a Single Prompt · Paper Radio
- DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models
- Classifier-Free Diffusion Guidance
- EVA3D: Compositional 3D Human Generation from 2D Image Collections
- AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars
- DreamWaltz: Make a Scene with Complex 3D Animatable Avatars
- AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control
- DreamHuman: Animatable 3D Avatars from Text
- DreamFusion: Text-to-3D using 2D Diffusion
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors
- Denoising Diffusion Implicit Models
- ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
- AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose
The paper
DivAvatar: Diverse 3D Avatar Generation with a Single Prompt · Read on arXiv
Weijing Tao, Biwen Lei, Kunhao Liu, Shijian Lu, Miaomiao Cui, Xuansong Xie, Chunyan Miao
Nanyang Technological University · Alibaba Group
DOI: 10.1109/WACV61041.2025.00255
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "DivAvatar: Diverse 3D Avatar Generation with a Single Prompt".
Paper summary: Tom: So we're looking at this paper now called "DivAvatar: Diverse three dee Avatar Generation with a Single Prompt" <ref:2402.17292#pg1,DivAvatar: Diverse 3D Avatar Generation with a Single Prompt>. Basically, the big claim is that they’ve made text-to-avatar generation much more diverse than what we see right now.
Jane: That sounds like a lot of diversity for just one text prompt. What exactly is the thesis here, Tom? What are they trying to fix in current text-to-avatar methods?
Tom: They’re saying existing methods are constrained by limited diversity, producing avatars with only subtle differences, whereas DivAvatar aims for unlimited number of diverse appearances through directly sampling noise from a normal distribution <ref:2402.17292#pg3>.
Lu: What's interesting about their approach is that they finetune a pretrained three dee generative model called EVAthree dee rather than using something scene-specific like NeRF, which means they are leveraging the intrinsic appearance prior of EVAthree dee <ref:2402.17292#pg6>.
Meng: So it’s not building a whole new scene representation from scratch; they’re using existing human body modeling and diffusion models but tweaking them for better variety. How does that help with diversity?
Tom: They claim this approach, by finetuning EVAthree dee, allows them to generate avatars of diverse appearances while also readily generating those avatars in different sampled poses at inference time <ref:2402.17292#pg1>.
Jane: But the paper says they tackle the diversity problem with two main designs during training: a noise sampling technique and some semantic-aware zoom mechanism <ref:2402.17292#pg4>.
Lu: The noise sampling technique is critical; they alternate between a random noise and a fixed noise throughout the finetuning process with probabilities of p and one minus p, which they say effectively resurrects the GAN’s capability for diverse avatar generation despite the nature of SDS <ref:2402.17292#pg5>.
Tom: So that means this sampling method is what allows them to produce a wide range of distinct avatars, even when dealing with the strong mode-seeking property of SDS <ref:2402.17292#pg9>.
Meng: If it works, how much does this translate to actual production costs? Because they’re doing this finetuning work. What about the geometry quality while they're doing all this noise sampling?
Jane: They also introduced a novel depth loss function to refine the geometry quality in the feature space <ref:2402.17292#pg4>. It's formulated as Ldepth equals one over N summed over i from one to N of the absolute difference between the feature of the predicted depth and the ground truth depth <ref:2402.17292#pg4>.
Paper summary: Tom: That loss function is used to smooth out unnatural dents in geometry that can sometimes come from watermarks artifacts in the texture <ref:2402.17292#pg9>. It’s a specific way to enforce better shape fidelity during training.
Lu: And they pair that with a comprehensive semantic aware zoom mechanism, which aims to ensure textual fidelity, especially when the text prompt is quite long <ref:2402.17292#pg9>. This zoom uses a more comprehensive set of angles than some existing works, and it even includes backviews which other methods often neglect <ref:2402.17292#pg6>.
Meng: So they are improving the connection between the words in the text prompt and what actually shows up in the final three dee model <ref:2402.17292#pg1>. That makes sense for user experience.
Jane: It does, because if you describe a character wearing ski clothes, you want to see that look accurately reflected, and this zoom mechanism seems designed to prevent that fidelity from being compromised <ref:2402.17292#pg4>.
Tom: The paper shows extensive experiments where they generate numerous avatars of different appearances in various poses by sampling the noise and SMPL parameters at inference time <ref:2402.17292#pg6>. The qualitative results show these avatars are more diverse in appearance and can be readily posed to natural poses compared to existing methods <ref:2402.17292#pg7>.
Lu: The ablation study confirms that the noise sampling strategy is critical for producing diverse outputs, showing that without it, the generator doesn't produce a wide range of distinct avatars <ref:2402.17292#pg9>. Also, the mesh optimization itself doesn't contribute to diversity; that true variation comes solely from their noise sampling method <ref:2402.17292#pg9>.
Meng: So if I’m looking at this for production, what’s the catch? Because they used a GAN model for diverse appearances during inference times, and the paper notes that the output texture lacks photorealistic details unless you do additional mesh optimization for each sample <ref:2402.17292#pg10>.
Tom: That's a real limitation they admit—the texture quality isn't perfect out of the box, so you still need that extra work on geometry for every single sample <ref:2402.17292#pg10>. Also, they found limited diversity in specific uniforms like nurses or firefighters because the model relies too much on appearance priors from EVAthree dee’s training data on the DeepFashion dataset <ref:2402.17292#pg10>.
Jane: So even with all these innovations, it still stops working well for certain specific types of clothing or roles, which is a real constraint they point out <ref:2402.17292#pg10>. What do you think this means for the people using these avatars?
Paper summary: Lu: It means the framework leverages a more comprehensive set of priors than existing avatar works, which enables efficient and rapid generation of a wide array of avatars <ref:2402.17292#pg4>. The idea is to create a much richer prior space from which to sample.
Meng: For deployment, that rapid generation aspect is key if you need high throughput for virtual try-ons or something similar, even if the texture isn't perfect yet. The speed of sampling the noise and parameters at inference time <ref:2402.17292#pg6> sounds efficient enough for real-time use.
Tom: And this whole DivAvatar system is built on finetuning EVAthree dee, which uses a compositional human NeRF representation as its foundation <ref:2402.17292#pg6>. That strong prior is what gives them the starting point to generate something coherent from just text <ref:2402.17292#pg6>.
Jane: So it’s an attempt to bridge the gap between high-quality human geometry and the need for expressive, diverse appearances all driven by a single text description <ref:2402.17292#pg1>. It tackles the limitation that most existing work only generates avatars with subtle differences.
Lu: The overall implication is that if we can get this noise sampling and semantic zoom working perfectly, we could create a system where generating a character in any desired pose and appearance just requires typing a description <ref:2402.17292#pg4>.
Meng: I guess the real world change here is reducing the manual effort needed to generate unique assets for digital environments. If you can generate fifty different looks from one prompt quickly, that’s a productivity win.
Tom: Exactly, it’s about streamlining the creative process and markedly decreasing production times <ref:2402.17292#pg2>. But we have to remember those limitations on texture detail and specific clothing types for now <ref:2402.17292#pg10>.
Jane: So, to wrap up this look at DivAvatar: Diverse three dee Avatar Generation with a Single Prompt, it’s a framework that uses noise sampling and semantic awareness to create avatars with much greater appearance diversity from one text prompt <ref:2402.17292#pg1>.
Lu: It builds on EVAthree dee by using finetuning techniques to unlock the full potential of its compositional representation <ref:2402.17292#pg6>.
Meng: It’s a solid step toward more flexible digital human assets, even if we still have texture fidelity issues and specific clothing limitations to work around <ref:2402.17292#pg10>.
Tom: We'll keep an eye on how they address those texture details in future iterations of the model <ref:2402.17292#pg10>.
Conclusion: Tom: So we've been looking at DivAvatar, this paper called "Diverse three dee Avatar Generation with a Single Prompt." Basically, they’re taking one text description and generating many different looks for that person.
Jane: That means instead of getting one static image from a prompt, you get a whole family of avatars. What’s the core idea behind how they achieve this variety?
Tom: They use this new framework to finetune an existing three dee model called EVAthree dee, which is good at understanding human shapes. They aren't starting from scratch with a brand new representation.
Lu: The key here is that they exploit the existing appearance of EVAthree dee, treating it like a strong starting point for generating different people based on text.
Meng: So it’s not inventing a whole new way to model human bodies, but just tweaking how you use what’s already there. What are the main tricks they used to make sure they get variety?
Jane: They introduced two specific techniques during training: one is this noise sampling method, which they say is crucial for getting different looks.
Tom: Yeah, that noise sampling alternates between random noise and a fixed noise during training. That's what lets the generator produce those distinct appearances instead of just one result.
Lalam: I can confirm that the authors show in their experiments that without this noise sampling, the generator really struggles to produce a wide range of different avatars. It’s critical for diversity here.
Jane: And they also have this semantic-aware zoom mechanism, which helps make sure the avatar actually matches what you asked for in the text prompt, especially if the description is long.
Tom: That zoom helps with textual fidelity, which is important because sometimes if you describe a person wearing a specific outfit, you want that to look exactly right.
Meng: From an engineering standpoint, I’m curious about deployment—does this whole finetuning process make it fast enough to actually put into a real-time application?
Jane: The paper shows they can sample the noise and the body parameters at inference time quite quickly. It suggests rapid generation is possible.
Tom: So the authors are saying this approach allows for much more flexible asset creation from just text, provided you handle those texture details later.
Lu: The implication is that we can move toward generating a much wider variety of human assets efficiently, rather than having to model every single person individually.
Jane: It’s a big step in making digital environments feel more populated and realistic based on simple descriptions. But they also admit some things didn't work perfectly yet, like certain uniforms.
Tom: Right, so it's a powerful tool for diversity, but we still have to deal with the texture quality and those specific clothing limitations they mentioned. (Sound of gentle radio transition sound)
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI