EmbedTalk: Talking Head Synthesis using Gaussian Embeddings

summary

The gist

Deformable 3D Gaussian Splatting (3DGS) has emerged as a popular method for real-time talking head synthesis, offering high-quality renderings at low latency.

In short

EmbedTalk is a method for real-time talking head synthesis using Deformable 3D Gaussian Splatting (3DGS). It improves rendering quality and lip synchronization by replacing tri-plane features with learnable per-Gaussian embeddings. This results in more compact models that run at 60+ FPS on laptops, producing highly consistent and realistic facial movements.

Key concepts

Deformable 3D Gaussian Splatting (3DGS)
A technique used to represent 3D scenes using a collection of small, volumetric shapes called Gaussians. These Gaussians are rendered by projecting them onto a 2D plane. In this method, the shape of these Gaussians can be dynamically changed or 'deformed' based on input data like speech and motion.
Per-Gaussian Embeddings (zg)
These are small, learnable numerical codes assigned to each individual Gaussian in the 3D scene. Instead of using fixed features, these embeddings allow the system to capture unique characteristics for every part of the head. These embeddings are crucial because they enable the model to understand and accurately deform specific facial attributes like position and opacity based on audio input.
Deformation Mechanism
This is how EmbedTalk changes a Gaussian's shape. It works by projecting the Gaussian primitives into continuous, lower-dimensional subspaces. These subspaces are then combined with audio signals and control signals, which are finally projected back into the 3D space to create an attribute-wise deformation that accurately reflects the desired facial movement.

Terminology used across episodes

This episode discusses

The paper

EmbedTalk: Talking Head Synthesis using Gaussian Embeddings · Read on arXiv

School of Computer Science, University of Leeds · Leeds Institute of Medical Education, School of Medicine, University of Leeds

Deformable 3D Gaussian Splatting (3DGS) has emerged as a popular method for real-time talking head synthesis, offering high-quality renderings at low latency. Tri-planes are a common choice for encoding Gaussians prior to deformation since they provide a compact and continuous representation. However, tri-plane encodings are limited by grid resolution and approximation errors introduced by projecting 3D volumetric fields onto 2D subspaces. Recent work has demonstrated the effectiveness of per-Gaussian embeddings for driving temporal deformations in 4D scene reconstruction. We introduce EmbedTalk, which leverages these embeddings to model speech-driven facial deformations for talking head synthesis. Comprehensive experiments show that EmbedTalk improves rendering quality, lip synchronisation, and motion consistency over previous 3DGS-based methods, while remaining competitive with state-of-the-art generative models. Replacing tri-plane features with embeddings also yields significantly more compact models that achieve 60+ FPS on a laptop GPU (RTX 2060 6 GB). Code and further details are available at https://arpita2512.github.io/EmbedTalk/

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "EmbedTalk: Talking Head Synthesis using Gaussian Embeddings".

Jane: Here's the gist of the paper. Deformable 3D Gaussian Splatting (3DGS) has emerged as a popular method for real-time talking head synthesis, offering high-quality renderings at low latency.

Tom: EmbedTalk improves rendering quality, lip synchronisation, and motion consistency over previous 3DGS-based methods while remaining competitive with state-of-the-art generative models.

Jane: Replacing tri-plane features with embeddings also yields significantly more compact models that achieve 60+ FPS on a laptop GPU.

Tom: Synthesising audio-driven talking heads in real-time is an important task in computer vision, with applicability in film production, teleconferencing, and virtual assistants.

Title and authors: Tom: So we’re looking at this paper called EmbedTalk: Talking Head Synthesis using Gaussian Embeddings. It’s written by Arpita Saggar, Jonathan Darling, Duygu Sarikaya, and David Hogg. Jane, what’s the main takeaway from just looking at that title?

Jane: Well, it tells you right away that they are taking three dee Gaussian Splatting and adding these learnable embeddings to make talking heads <ref:2603.07604#pg1>. It suggests they are trying to fix some problems with how we currently do this kind of synthesis. It’s about making the three dee scene reconstruction more robust for speech input <ref:2603.07604#pg1>.

Tom: Exactly, it sounds like they're focusing on using these embeddings to model the facial movements driven by audio. I always like when papers move away from just using simple features and introduce something learnable that adapts to the data.

Lu: What’s interesting here is that they are specifically targeting the deformation process itself. They aren't just using a standard tri-plane encoding, which they say has issues with mirroring artifacts because of feature entanglement between subspaces.

Meng: So, if I understand this right, they are replacing those fixed tri-plane features with these per-Gaussian embeddings. That sounds like a way to make the model much more compact and flexible for handling different identities.

Lalam: I’m seeing here that they are aiming to generate stable talking heads with high audio visual alignment by using those embeddings for deformation, as you see on page two of the paper > EmbedTalk: Talking Head Synthesis using Gaussian Embeddings.

The paper's summary: Tom: So, what’s the core idea behind this approach? Basically, they are taking a dense reconstruction from COLMAP to start their three dee Gaussians and then giving each of those Gaussians its own set of learnable embeddings <ref:2603.07604#pg1>.

Jane: Right, so instead of every Gaussian being just a fixed shape, it gets an embedding that learns how it should deform based on the audio signal and the upper-face movements we feed in. It’s like giving each part of the three dee model a personality that reacts to what you’re saying <ref:2603.07604#pg1>.

Lu: What really stands out is their method for handling those high-frequency displacements, especially around the mouth region. They introduce positional encodings on top of these Gaussian embeddings to predict the deformations, which should help reconstruct those fine details more accurately.

Meng: From a deployment standpoint, having per-Gaussian embeddings means they are using smaller models for the deformation MLP instead of something huge that tries to model everything at once. That sounds good for real-time performance on consumer hardware.

Lalam: They’re feeding the speech signal and upper-face movements into this deformation MLP along with a positional encoding of the Gaussian embedding to predict things like changes in position and opacity, as mentioned on page three of EmbedTalk: Talking Head Synthesis using Gaussian Embeddings > Initialisation Per-Gaussian Embeddings Speech Signal Audio Encoder Δμ, Δα three deeGS Rasteriser + Dense Reconstruction e (blink, brow raise) Canonical three dee Gaussians (μ, s, r, α, f) a Camera Input Video Rendered Frames (Monocular) <ref:2603.07604#pg1,EmbedTalk: Talking Head Synthesis using Gaussian Embeddings>.

The paper's improvements: Tom: Okay, so how does this actually improve the existing methods? Page two mentions they are extending the embedding-based deformation paradigm to talking head synthesis. They claim this yields stable talking heads with high audio visual alignment.

Jane: And it goes further by using a local smoothness constraint on those embeddings to encourage similar embeddings for neighboring Gaussians, which should lead to better motion consistency overall. It addresses the wobbling issue that happens when you rely on imprecise facial tracking for camera pose, which they point out as a problem with prior work > EmbedTalk: Talking Head Synthesis using Gaussian Embeddings.

Lu: They also tackled the head wobbling by starting their Gaussians with a stable, dense reconstruction obtained via COLMAP, which is a solid initialization step. But they also used positional encodings specifically to reconstruct high-frequency displacements in the mouth region, which is key for lip synchronisation.

Meng: I’m interested in how much better the lip movement fidelity is compared to just using tri-plane features. If this works well, it means we can get much more realistic mouth movements without needing massive computational resources for every single feature interaction.

Lalam: They state that by generating mouth movements consistent with an identity’s style, their method achieves higher realism than state-of-the-art methods > EmbedTalk: Talking Head Synthesis using Gaussian Embeddings.

Conclusion: Tom: So, to wrap up, the main implication here is that learnable embeddings can significantly improve the quality of talking head synthesis while keeping the computational overhead down. They achieved better lip synchronisation and motion consistency compared to previous three deeGS-based works <ref:2603.07604#pg1>.

Jane: It seems like this approach lets us get much more faithful lip movements in these syntheses, especially when other methods often default to a closed-mouth state. It’s about getting the mouth right when you use audio input.

Lu: The method is restricted to facial animation for now, but it opens up possibilities for integrating this with full-body motion modelling techniques to support fully interactive avatars later on.

Meng: For deployment, they also started adaptive densification and pruning from iteration five hundred onwards, which shows they have a structured way of managing the memory footprint as the model gets bigger.

Lalam: And just to be clear about what it doesn't do, the paper says this method is focused on facial animation and isn't necessarily aiming for full-body motion modeling yet > EmbedTalk: Talking Head Synthesis using Gaussian Embeddings.

More episodes

← Home