Listen to the Features: Voice Anonymization Driven by Content Embedding Matching over Signal Reconstruction

arXiv:2607.09767 · eess.SP, cs.AI · Submitted 2026-07-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Listen to the Features".

Dev: Here's the gist of the paper. The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech and relies on content embeddings extracted from a frozen pretrained wav2vec2 encoder.

Rosa: Model Core Concept The model is designed to generate an anonymized audio segment by decoding content embeddings into a signal using vector quantization and a HiFi-GAN vocoder.

Dev: The training objective enforces that the embeddings of the anonymized signal match those of the original one, without any reconstruction loss or speaker embedding mapping.

Rosa: Training Objectives and Architecture The architecture employs a twobranch approach where the first branch uses a frozen wav2vec2 model to extract latent content representations.

Paper summary: Rosa: So we’re looking at this paper called "Listen to the Features: Voice Anonymization Driven by Content Embedding Matching over Signal Reconstruction". What's the main idea here?

Dev: Basically, the whole thesis is about voice anonymization that focuses on keeping the content intact instead of trying to make it sound realistic. They’re using content embeddings pulled from a frozen wav2vec2 encoder to do that.

Taro: So they aren't focusing on making beautiful new audio, but just making sure what you say is still recognizable, even when the voice is changed.

Rosa: Exactly. The key thing they claim is that by matching the content embeddings of the original and anonymized signals using vector quantization and a HiFi-GAN vocoder, they achieve good performance without needing a reconstruction loss or any speaker embedding mapping during training.

Dev: That’s the core mechanism—it enforces that those embeddings stay aligned without worrying about reconstructing the exact waveform, which is smart for efficiency.

Taro: But they also have this second branch to actively discard speaker information using gradient reversal and an auxiliary speaker predictor during training, which adds a layer of control over that privacy aspect.

Rosa: And when you look at the results, it shows that this straightforward embedding-based approach gets a very low word error rate of two point five three while achieving an anonymization performance ranking within the first level for VPC <ref:2607.09767#pg1,that this straightforward embedding-based approach>.

Dev: That's impressive because they also show that emotions are partially preserved with a unweighted average recall of forty-three point nine one, even without needing an extra training objective specifically for emotion <ref:2607.09767#pg2>.

Taro: So it suggests that the model is actually picking up some of that emotional information naturally just by focusing on the content matching.

Rosa: It does, but we have to talk about quality. The qualitative automated mean opinion score analysis shows that even though the quality of the anonymized signals is clearly degraded with a mean opinion score of one point sixty-three they remain usable for automatic speech recognition and somewhat useful for emotion classification.

Dev: That's a big caveat—the signal sounds robotic in informal listening tests, but people can still recognize most words.

Taro: So the implication here is that for tasks like automatic speech recognition or speech emotion recognition, you don't necessarily need a perfectly natural-sounding output if the underlying content representation is stable and informative.

Rosa: That seems to be their main point for real world applications—the waveform doesn't need to be perfect, just faithful to the content embedding.

Dev: The computational efficiency is also something they highlight; the inference model size is less than ninety-nine million parameters, with only eighteen million of them being trained.

Taro: That size makes it feasible for recording directly anonymized speech during corpus collection, which cuts down on some overhead in the data pipeline.

Rosa: So to summarize this paper, "Listen to the Features: Voice Anonymization Driven by Content Embedding Matching over Signal Reconstruction" presents a lightweight voice anonymization model that uses frozen wav2vec2 content embeddings decoded via vector quantization and a HiFi-GAN vocoder, focusing on matching content representations without reconstruction loss.

Dev: The main takeaway from the conclusion is that this model achieves very low word error rates of two point five three while achieving an anonymization performance ranking within the first level for VPC <ref:2607.09767#pg1>.

Taro: It really boils down to showing that you can achieve privacy and good speech recognition metrics simultaneously by focusing on matching those content embeddings rather than trying to generate perfect audio reconstruction.

Rosa: And what this means for us is that for downstream tasks like ASR or SER, generating a high-quality natural signal isn't strictly necessary as long as the anonymized waveform carries a stable and informative content representation.

Dev: The authors also pointed out some limitations in their work; specifically, the architecture they proposed doesn't differentiate between linguistic content and other features like emotional content because it uses a single codebook.

Taro: That means it might not be as nuanced as you'd want for preserving subtle emotional context, which is something we should probably keep in mind for future work.

Rosa: Right, so the next step for this research is to explore more nuanced training objectives that can better balance privacy, intelligibility, and emotional content.

Conclusion: Rosa: So we’re looking at this paper called "Listen to the Features: Voice Anonymization Driven by Content Embedding Matching over Signal Reconstruction."

Dev: It’s about voice anonymization that focuses on keeping the content intact instead of trying to make it sound realistic.

Taro: They’re using embeddings from a frozen wav2vec two encoder to do that.

Rosa: Exactly, they're matching content embeddings between the original and anonymized audio signals using vector quantization and a HiFi GAN vocoder.

Dev: The key thing is that they don't use any waveform reconstruction loss or speaker embedding mapping during training.

Taro: And they have this second branch to actively discard speaker information using gradient reversal with an auxiliary speaker predictor while training.

Rosa: When you look at the results, this straightforward embedding-based approach gets a very low word error rate of two point five three while achieving an anonymization performance ranking within the first level for VPC.

Dev: That’s interesting because they also show that emotions are partially preserved with a unweighted average recall of forty-three point nine one, even without needing an extra training objective specifically for emotion.

Taro: It suggests the model is actually picking up some of that emotional information naturally just by focusing on the content matching.

Rosa: But we have to talk about quality; automated mean opinion score analysis shows that even though the quality of the anonymized signals is clearly degraded with a mean opinion score of one point six three, they remain usable for automatic speech recognition and somewhat useful for emotion classification.

Dev: That's a big caveat—the signal sounds robotic in informal listening tests, but people can still recognize most words.

Taro: So the implication here is that for tasks like automatic speech recognition or speech emotion recognition, you don't necessarily need a perfectly natural-sounding output if the underlying content representation is stable and informative.

Rosa: That seems to be their main point for real world applications—the waveform doesn't need to be perfect, just faithful to the content embedding.

Dev: The authors also pointed out some limitations; specifically, the architecture they proposed doesn't differentiate between linguistic content and other features like emotional content because it uses a single codebook.

Taro: That means it might not be as nuanced as you'd want for preserving subtle emotional context, which is something we should probably keep in mind for future work.

Rosa: So the next step for this research is to explore more nuanced training objectives that can better balance privacy, intelligibility and emotional content.

University Grenoble Alpes, CNRS, Grenoble INP, LIG

eess.SP, cs.AI

Submitted: 2026-07-07

Updated: 2026-11-10

Journal ref: 6th Symposium on Security and Privacy in Speech Communication joined with 4th Voice Privacy Challenge at Interspeech 2026, Sep 2026, Sidney, NSW, Australia

Code: https://github.com/tarepan/SpeechMOS

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech and relies on content embeddings extracted from a frozen pretrained wav2vec2…

Key concepts

Content Embeddings
These are numerical representations of the speech's core meaning, extracted from a pre-trained wav2vec2 model. Instead of trying to recreate the exact sound waves, these embeddings capture what is being said, allowing the system to focus on preserving information.
Vector Quantization (FVQ)
This module acts as a bottleneck during training. It forces the model to compress and retain only the most important features from the content embeddings while discarding less relevant noise or speaker-specific details. This ensures that only meaningful information is passed forward.
HiFi-GAN Vocoder
This component takes the compressed content embeddings and transforms them into actual audio waveforms. It's a high-quality synthesis tool, but in this context, it is used to generate an anonymized signal based strictly on the preserved content information.

Terminology

Summary

The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech and relies on content embeddings extracted from a frozen pretrained wav2vec2 encoder. p. 0

Model Core Concept

The model is designed to generate an anonymized audio segment by decoding content embeddings into a signal using vector quantization and a HiFi-GAN vocoder. The training objective enforces that the embeddings of the anonymized signal match those of the original one, without any reconstruction loss or speaker embedding mapping. p. 0

Training Objectives and Architecture

The architecture employs a twobranch approach where the first branch uses a frozen wav2vec2 model to extract latent content representations. These embeddings pass through a Factorized Vector Quantization (FVQ) module, which acts as a bottleneck to force the model to retain useful features while discarding others. p. 2 The second branch is used only during training to optimize network weights and discard speaker information via gradient reversal with an auxiliary speaker predictor. p. 2 The total loss is a weighted sum of the content loss Lc and the speaker loss Ls. p. 2

Experimental Results and Performance

Experiments show that the best hyperparameter set performs very well on Word Error Rate (WER) at 2.53%. The model achieves an anonymization performance (EER of 13. p. 039%) ranking within the first level for VPC. p. 0 Notably, emotions are partially preserved with a Unweighted Average Recall (UAR) of 43. p. 091, even without a supporting training objective. p. 0

Perceptual Quality Trade-offs

Qualitative automated Mean Opinion Score (MOS) analysis highlights that even if the quality of anonymized signals is clearly degraded (MOS of 1.63), the generated signals remain usable for automatic speech recognition and to some extent for emotion classification. p. 1 Informal listening tests highlighted that the anonymized utterances sounded “robotic”, but most words could still be recognized by a human ear. p. 3

Conclusion and Future Directions

The best configuration achieves WER close to original data while maintaining a fair level of emotion recognition performance and reaching privacy scores in the lowest ranking category. The findings suggest that for automated downstream tasks such as ASR and SER, generating a high-quality natural signal is not strictly necessary as long as the anonymized waveform carries a stable and informative content representation. p. 5 The paper calls for more nuanced training objectives that balance privacy, intelligibility, and emotional content. p. 5

Computational Efficiency

The total size of the inference model is less than 99M parameters, comprising 18.54M trained ones. p. 5 This size is competitive regarding WER for the target automatic speech recognition task and aligns with the goal of recording directly anonymized speech during corpus collection. p. 5 The trade-off between metric performance and computational cost must be carefully questioned in future research. p. 5

Key Findings Summary

The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech

It relies on content embeddings extracted from a frozen pretrained wav2vec2 encoder

These embeddings are decoded into an anonymized signal using vector quantization and a HiFi-GAN vocoder, both trained on LibriTTS without any waveform reconstruction loss or speaker embedding mapping

Results show that this straightforward embedding-based approach achieves very low WER (2.53) with an anonymization performance (EER 13. p. 039) ranking within first level for VPC

**"Notably, emotions are partially preserved (UAR 43.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Enhanced content preservation for ASR by leveraging factorized vector quantization to create a bottleneck that forces the model to retain features that are useful for the downstream task and to discard the rest. This allows speech recognition performance (WER) to reach 2.53% even without reconstruction loss or speaker embedding mapping.

  2. Emotion preservation during anonymization by observing that emotions are partially preserved (UAR 43.91), even without a supporting training objective, indicating that foundation models like wav2vec2 already encode relevant emotional information useful for downstream tasks like Speech Emotion Recognition (SER).

  3. Lightweight model deployment by achieving an inference model with less than 99M parameters, comprising 18.54M trained ones, making the anonymization system computationally inexpensive and suitable for direct recording during corpus collection.

  4. Improved privacy metric performance by achieving an Equal Error Rate (EER) ranking within first level for VPC (between 10 to 20%) using the best configuration (cb65536), demonstrating effective speaker information discarding without relying on explicit speaker embeddings or reconstruction losses.

Sources

Related papers