LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models

arXiv:2610.15379 · eess.AS, cs.LG, cs.SD · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models".

Jane: The gist The LiveAnon system is a voice-conversion-based streaming speaker anonymization framework that directly conditions an autoregressive codec language model on hidden features from a pretrained streaming ASR encoder,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper now, "LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models," and the main thing here is they’ve built a system that works directly on speech streaming.

Jane: That sounds important because speaker anonymization usually means you have to deal with these discrete content bottlenecks when you try to keep emotion in the audio.

Lu: Yeah, they tackle that by conditioning an autoregressive codec language model directly on hidden features from a pretrained streaming ASR encoder, which is a zipformer model.

Meng: So it’s using features from an existing ASR encoder as the content representation for their own acoustic generation model. What’s the core idea behind that?

Tom: The idea is that this setup preserves emotion without needing any emotion-specific supervision during training, which is a big deal because usually you need those extra labels to get good emotional results.

Jane: It sounds like they are using features extracted from the final encoder stage before downsampling, and they keep a fifty-Hz frame rate which matches the codec tokens they’re generating.

Lu: Exactly, and they linear project those content features to fit into the decoder conditioning dimension, and then use feature-wise linear modulation called FiLM to condition every Transformer layer.

Meng: How does that fiLM work in practice? Does it mean the content features are predicting parameters for how much each layer should modulate its input?

Tom: Right, they say that the content features predict per-layer affine modulation parameters, which is how they condition the Transformer layers to focus on the right stuff.

Jane: Then you have this neural audio codec, JHCodec, and acoustic prompting where the reference utterance generates discrete acoustic tokens that serve as a prompt for the decoder.

Lu: They adapt acoustic prompting by using source content features to condition codec continuation instead of using the reference as a separate conditioning prefix for the generation process.

Meng: So they are merging the reference and generated tokens into one autoregressive stream with shared codebook embeddings, which is an interesting way to link them together.

Tom: And they optimize their loss function using LCE, which is defined as X N n=one X D d=one − log pθ (an,d Hn,d, cn), where an and d are the codec token at frame n and RVQ depth d <ref:2610.15379#pg1>.

Title and authors: Jane: That means during training both regions—the reference and the generated part—are teacher-forced to learn how to continue the acoustic generation while following that source content.

Lu: The training involves teacher-forced codec sequences, so they are learning this joint continuation process under those conditions.

Meng: Now, I want to talk about latency because streaming systems can’t always afford long waits at scale. How do they handle that in the actual generation?

Tom: They introduced something called capped RVQ decoding to reduce the codec generation latency by limiting it to three codec frames and parallelizing the rest of the RVQ codebooks.

Jane: So they’re using this mechanism, d k = (k, delta max), where k goes from zero up to D minus one, and delta max is a maximum offset that controls the trade-off.

Lu: They found that setting delta max = three which corresponds to sixty milliseconds codec latency, gives them the best balance between latency, privacy, and utility <ref:2610.15379#pg1>.

Meng: So it’s a controllable trade-off where you can choose how much dependency across codebooks you want to keep versus how fast the generation needs to be.

Tom: It sounds like they’ve found a sweet spot for real-time applications, even though the latency is still tied to that chunk size, which they say is three hundred twenty milliseconds with a chunk size of sixteen kHz.

Jane: The results show improvements under the VPC24 setting, where LiveAnon improved intelligibility and emotion preservation over the fixedtarget Stream-Voice-Anon system.

Lu: Specifically, under the matched vctk-1fix setting, LiveAnon achieved a word error rate WER of four point three one compared to SVA's four point five four for intelligibility.

Meng: And they also got an emotion UAR of forty-four point zero nine, which is comparable to SVA’s emotion-prompted cremad-emo-4rnd setting at forty-four point five nine, even though LiveAnon only used a single fixed VCTK prompt during that comparison.

Tom: That’s interesting because it shows the content features they are using are effective for preserving emotion without needing explicit emotion supervision on top of the system.

Jane: And they also reported a lower speaker classification accuracy, twenty-eight point seven five compared to SVA's thirty-two point five zero under that same setting, which suggests it’s less speaker-discriminative than the prior method.

Lu: The paper points out that their ASR representation achieves a higher emotion UAR of fifty-five point one seven versus SVA's content representation score of thirty-nine point nine five, and interestingly, lower speaker classification accuracy, twenty-eight point seven five versus thirty-two point five zero >

Meng: So what does this mean for deployment? If I’m running this on a production server, what’s the practical cost of running it compared to the SVA baseline?

Title and authors: Tom: The algorithmic latency for LiveAnon with a three hundred twenty milliseconds chunk size is five hundred milliseconds, giving them a real-time factor of zero point one seven on an RTX four thousand ninety GPU.

Jane: It seems like they’ve really managed to keep the complexity down enough while still getting better results than what was available before.

Lu: The paper is clear that LiveAnon preserves emotion information without emotion-specific data or supervision, which is a key finding for using general features to do this.

Meng: I just want to know, what are the limitations they themselves pointed out? Where does this system stop working effectively?

Tom: They mention that the current latency largely depends on the encoder chunk size, so if you change that setting, the latency changes a lot.

Jane: That makes sense because it points to an area where future work could focus on finding lower-latency streaming ASR encoders.

Lu: And they also suggest exploring diverse per-utterance target prompts as a way to potentially improve performance further down the line.

Meng: So, if I were planning to deploy this tomorrow, what’s my first thought about whether it’s robust enough for real-world, noisy speech environments?

Tom: Well, the results are strong under VPC24 and two thousand twenty-six Track one protocols for word error rate and emotion preservation <ref:2610.15379#pg1>.

Jane: It seems like they've managed a competitive privacy–utility balance against offline baselines by using just a single fixed VCTK prompt.

Lu: The overall implication is that direct acoustic conditioning on streaming features can be a viable path for speaker anonymization when you want to avoid the heavy supervision requirements.

Meng: I think the main thing here is that they've shown this method works well without needing those specific emotion labels, which simplifies things significantly for deployment pipelines.

Tom: So, to wrap up on "LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models," it’s a voice-conversion based system that conditions a codec language model on ASR encoder features to preserve emotion without emotion-specific supervision.

Jane: And they did it while balancing latency using capped RVQ decoding, achieving strong results against SVA under VPC24 and two thousand twenty-six <ref:2610.15379#pg1>.

Lu: It shows that even with streaming constraints, you can extract rich emotional data from content features and use them for acoustic generation effectively.

Meng: It’s a solid paper showing a path forward for systems that need to be fast but still want to maintain quality in speaker anonymization tasks.

The paper's summary: Tom: So, we just talked about how LiveAnon uses features from an ASR encoder to condition an audio codec model, which keeps emotion without needing specific emotion training data or supervision.

Jane: Right, but what does that mean for a regular person listening? Basically, it’s taking the sound of someone talking and converting it into a different voice while keeping their emotional tone intact.

Lu: Exactly. They use these hidden features from a pretrained model, like Zipformer, which they keep running at fifty hertz frame rate to match the audio tokens they are generating.

Meng: So you have this source speech, you extract the content information from it using that encoder—that’s your acoustic prompt—and then you use that prompt to guide a decoder that spits out a new audio stream. That’s how the process flows.

Tom: And they do this by feeding those content features into every single layer of the Transformer decoder through something called feature-wise linear modulation, or FiLM, which lets the content information influence how each layer processes the data.

Jane: It sounds like a very direct way to link what we know about *what* is being said with *how* it's being said emotionally. It’s not guessing the emotion; the features are already there.

Lu: And they train this whole thing by making both the reference voice and their generated tokens learn to follow that source content simultaneously, using a loss function called LCE for acoustic continuation.

Meng: From a production standpoint, I’m thinking about how big these models are. They have to handle this streaming process efficiently when it's running live.

Tom: That’s where they introduced capped RVQ decoding, which limits the generation delay to three codec frames and parallelizes the rest of the codebooks so it can actually run with a real-time factor of zero point one seven on an RTX four thousand ninety.

Jane: So they manage to keep that quality while controlling the speed, which is crucial for anything meant to be used in a streaming application. It’s not just about making it work; it's about making it work fast enough for a real conversation.

Lu: The authors found that capping the delay at three frames, corresponding to sixty milliseconds codec latency, gives them the best trade-off between generation speed and maintaining privacy and utility.

Meng: Sixty milliseconds is still quite slow if you’re aiming for super low latency interaction. I wonder what happens if we try to go even faster than that?

Tom: The paper says the chunk size determines the latency, so they are currently using a three hundred twenty millisecond chunk size to get those results, and they suggest future work should look at lower-latency streaming ASR encoders.

Jane: So it’s a great balance right now—they get strong results on metrics like word error rate of four point three one compared to the baseline SVA system, while also showing good emotion preservation scores.

Lu: And they noted that their representation of content features retains richer emotion information than the prior method, even though it is less speaker-discriminative, which is a key finding in itself.

Meng: It’s interesting that they report lower speaker classification accuracy with LiveAnon compared to the SVA baseline, suggesting this method might be better for anonymization because it doesn't just erase the identity but also keeps the nuance.

Tom: So what we’re seeing here is a system that uses existing content knowledge to guide speech synthesis while actively managing latency through clever codebook scheduling.

Jane: It shows that you don't always need entirely new, massive datasets to get good speaker anonymization results if you can leverage existing streaming audio technology and content representations effectively.

Lu: The implication is that direct acoustic conditioning on these features is a solid path forward for systems where emotion preservation without explicit emotion supervision is the main goal.

Meng: For me, the real question I have is how much of this performance relies on that specific Zipformer encoder they used; if we swap it out, does it break?

Tom: Well, the paper focuses on showing that this specific setup works well under VPC24 and VPC26 protocols using their chosen encoder.

Jane: And the results are quite strong against those established baselines, which suggests this approach is competitive right now for privacy and utility.

Lu: The authors do flag that latency is still highly dependent on the choice of the streaming ASR encoder chunk size, so that’s a clear area they point toward for future research.

The paper's improvements: Tom: So we’re talking about what they suggest doing next with LiveAnon, and it looks like they aren't just stopping there—they're looking at making it even more flexible for different situations.

Jane: Right, so they want to move beyond just one fixed reference utterance and create a system that can handle different target speakers or contexts on the fly.

Lu: That’s the direction they are heading, exploring diverse per-utterance target prompts, which means instead of using one universal prompt for everyone, you could have a specific prompt for each new voice you want to anonymize.

Meng: From my side at the engineering level, having diverse prompts sounds complicated. Does that mean we’d need a whole new pipeline just to manage all those different reference inputs?

Tom: Well, they see it as an opportunity because it lets the system adapt its acoustic generation based on more specific information about the target speaker you’re trying to anonymize.

Jane: It means the core mechanism of conditioning the codec continuation on source features becomes even more powerful because you can tailor that prompt to get a better result for that specific voice.

Lu: Yes, and they are also looking into lower-latency streaming ASR encoders as a way to tackle the speed issue head-on, because right now the latency is still tied directly to how big the chunk size of those encoders is.

Meng: That’s a practical concern for me; if we use smaller chunk sizes to get lower latency, we might sacrifice some of that rich content representation they built into the system.

Tom: They are actively looking at models that can process audio in smaller chunks faster, which would directly help reduce the generation time and improve the real-time factor significantly.

Jane: So, it’s a two-pronged approach then: one side is making the prompts smarter for different speakers, and the other side is making the underlying content extraction itself much quicker.

Lu: It seems like they want to decouple those two challenges—prompt diversity and encoding speed—so that they can optimize each part independently.

Meng: If we could get a lower latency encoder, I think that would be a huge win for deployment because it makes the whole system usable in more demanding real-time environments without having to over-provision hardware just to compensate for slow processing.

Tom: Exactly, and the results show that even with the current setup, they have already established a strong privacy utility balance against offline baselines under VPC24 and VPC26.

Jane: So, what this suggests for people using this technology is that we can get high-quality anonymization while still preserving emotional nuance without having to collect special emotion-labeled datasets for every single voice we want to process.

Lu: That’s the practical implication—using general content features instead of emotion-specific supervision simplifies the training pipeline immensely and makes it accessible for more applications.

Conclusion: Tom: So to wrap up on LiveAnon, they’ve shown that you can get good speaker anonymization without needing emotion-specific training data or labels by conditioning the codec model directly on ASR encoder features.

Jane: It’s a system that works because it uses general content knowledge from speech to guide the audio conversion process, which is pretty neat for privacy because it keeps the emotional tone but not the identity.

Lu: The main technical achievement is using that content representation to condition every Transformer layer through FiLM modulation, which lets those features directly influence how much each part of the decoder processes information.

Meng: From a deployment view, they managed to balance that quality with latency by capping the RVQ delay at three codec frames and using parallel processing for the rest of the codebooks.

Tom: They achieved a real-time factor of zero point one seven on an RTX four thousand ninety when they used a three hundred twenty millisecond chunk size.

Jane: It really shows that even under streaming constraints, you can maintain performance against established benchmarks like SVA, achieving a word error rate of four point three one and decent emotion scores.

Lu: They also found that their ASR representation of content features holds more emotion-related information than the previous method, even though it’s less accurate at identifying the speaker itself.

Meng: That lower speaker classification accuracy is actually good for anonymization because it means the system isn't just erasing someone's identity; it’s also keeping some of their vocal character.

Tom: The overall implication is that this paper, LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models, provides a solid, balanced path forward by leveraging existing streaming ASR technology.

Jane: It means we can build systems that are both fast enough for real-time use and effective at preserving things like emotion without needing massive amounts of specific supervision data.

Lu: They did flag that the latency still depends heavily on the choice of the streaming ASR encoder chunk size, so future work should focus on finding faster encoders.

Meng: I agree with that; if we can make those chunk sizes smaller or find a fundamentally faster encoder, it would simplify deployment immensely because we wouldn't need to run models for such long durations.

Tom: So yeah, the core idea of using content features to condition the codec language model is sound and it’s already performing well against VPC24 and VPC26 baselines.

Jane: It’s a nice demonstration of how we can get strong results in privacy utility by focusing on the content itself rather than trying to predict complex emotional states directly.

Lu: The work confirms that direct acoustic conditioning is a viable strategy for speaker anonymization when you want to avoid the heavy requirement of emotion-specific supervision.

Meng: I’m still skeptical about running this at massive scale because I don't know how stable those content features are when we hit noisy, real-world traffic patterns.

Tom: That’s a fair concern; the benchmarks are good, but production reality is always trickier than a clean test set.

Jane: We’ll keep an eye on those future work directions to see if they can get that latency down further or explore those diverse target prompts we talked about earlier.

Yejin Kim, Kyungsu Kim, Seungu Han, Kyogu Lee

Music and Audio Research Group (MARG) · Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University

eess.AS, cs.LG, cs.SD

Submitted: 2026-10-11

Updated: 2026-10-11

Project page: https://jin1025.github.io/LiveAnon

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The LiveAnon system is a voice-conversion-based streaming speaker anonymization framework that directly conditions an autoregressive codec language model on hidden features from a pretrained

Key concepts

Streaming ASR Encoder: This component is a pretrained model that processes continuous speech in real-time to extract meaningful content features. It's frozen during training and provides the core source content representation for the anonymization process, ensuring the generated voice reflects the original speaker's words.
Neural Audio Codec (JHCodec): This is a neural network that converts speech into discrete acoustic tokens, acting like a language model for audio. In LiveAnon, it generates new acoustic tokens conditioned on both the source content features and a reference prompt to create the anonymized output.
Capped RVQ Decoding: To keep generation fast, this technique limits how far ahead the system can look in generating audio tokens (the delay). By capping this delay, it balances generating high-quality speech with maintaining low latency and ensuring privacy.

Summary

The gist The LiveAnon system is a voice-conversion-based streaming speaker anonymization framework that directly conditions an autoregressive codec language model on hidden features from a pretrained streaming ASR encoder, preserving emotion without emotion-specific supervision.

System Overview

LiveAnon consists of three main components: a pretrained streaming ASR encoder, a neural audio codec, and a causal Transformer decoder The system is given source speech and a reference utterance from the target speaker, where the ASR encoder extracts content features from the source while the codec converts the reference into discrete acoustic tokens that serve as the speaker prompt The decoder then autoregressively continues this prompt conditioned on the source content, and these generated tokens are reconstructed into a waveform by the codec decoder.

Pretrained ASR Content Representation

The system utilizes hidden representations from a frozen pretrained streaming Zipformer ASR encoder as the source content representation The features are extracted from the final encoder stage before the final downsampling, preserving a 50-Hz frame rate that matches the codec tokens These features are linearly projected to the decoder conditioning dimension, and a fixed temporal shift aligns the ASR and codec representations. The aligned content sequence conditions every Transformer layer through feature-wise linear modulation (FiLM), where the content features predict per-layer affine modulation parameters.

Codec Prompt Continuation and Acoustic Generation

The system uses JHCodec, a streaming neural audio codec that represents speech with discrete acoustic tokens Acoustic prompting is adapted to voice-to-voice anonymization by conditioning codec continuation on source content features Instead of using the reference as a separate conditioning prefix, the reference and generated codec tokens form a single autoregressive acoustic stream with shared codebook embeddings. The loss function optimized is LCE = X N n=1 X D d=1 − log pθ (an,d Hn,d, cn), where an,d is the codec token at frame n and RVQ depth d. During training, both regions are teacher-forced to learn acoustic continuation from the reference voice while following the source content. At inference, the reference codec tokens and their content features are pre-filled into the Transformer KV cache, after which autoregressive generation continues from the end of the prompt using source content.

Capped RVQ Decoding

To reduce codec generation latency, a capped delay scheme is introduced that limits the delay to three codec frames and parallelizes the remaining RVQ codebooks The capping mechanism is defined as dk = min(k, δmax), k = 0..., D − 1 (2) This allows for a controllable trade-off between inter-codebook dependency and generation latency. For codec frame duration ∆, the resulting codebook latency is δmax∆. The authors found that δmax = 3 (60 ms codec latency) provides the best balance between latency, privacy, and utility.

Experimental Results and Trade-offs

LiveAnon shows improvements across evaluation metrics under the VPC24 setting, where it improves intelligibility and emotion preservation over fixedtarget Stream-Voice-Anon (SVA) It achieves a strong privacy–utility balance, ranking second across all metrics under VPC26. Specifically, LiveAnon achieves a lower word error rate (WER↓) of 4.31 compared to SVA's 4.54 under the matched vctk-1fix setting. Furthermore, LiveAnon achieves an emotion UAR of 44.09, which is comparable to SVA’s emotion-prompted cremad-emo-4rnd setting (44.59) despite using only a single fixed VCTK prompt. The algorithmic latency for LiveAnon with a 320 ms chunk size is 500 ms, yielding a real-time factor (RTF) of 0.17 on an RTX 4090.

Conclusion and Future Work

LiveAnon preserves emotion-related information without emotion-specific data or supervision, while capped RVQ decoding balances codec latency and generation quality The system maintains a competitive and well-balanced privacy–utility performance against offline VPC26 baselines. Future work should explore diverse per-utterance target prompts and lower-latency streaming ASR encoders, as the current latency largely depends on the encoder chunk size. The demo is available on https://jin1025.github.io/LiveAnon/.

How it works

The system architecture involves a pretrained streaming ASR encoder, a neural audio codec, and a causal Transformer decoder The content features are extracted from the final encoder stage before the final downsampling, preserving a 50-Hz frame rate that matches the codec tokens These features are linearly projected to the decoder conditioning dimension, and they condition every Transformer layer through feature-wise linear modulation (FiLM). The model is trained for 400k optimization steps on a single RTX 3080 Ti GPU using AdamW with a peak learning rate of 10−4 and a cosine decay schedule.

Key Components

The system incorporates several key mechanisms to achieve its goals:

)(a) Training and Inference Flow:

Training uses teacher-forced codec sequences, whereas inference pre-fills the reference prompt into the Transformer KV cache and continues autoregressive generation with the capped RVQ delay pattern.

) Codec Prompt Continuation:

The reference and generated codec tokens form a single autoregressive acoustic stream with shared codebook embeddings. Each region is conditioned on its aligned content features.

) Capped Delay Scheme:

The capped delay dk = min(k, δmax), k = 0..., D − 1 (2) This scheme preserves staggered dependencies up to a maximum offset δmax. The authors found that δmax = 3 (60 ms codec latency) provides the best balance between latency, privacy, and utility.

) Content Representation Analysis:

LiveAnon’s ASR representation achieves higher emotion UAR than SVA’s content representation (55.17 vs. 39.95) while yielding lower speaker classification accuracy (28.75 vs. 32.50). The results suggest that LiveAnon’s ASR representation retains richer emotion-related information while being less speaker-discriminative than SVA’s content representation.

Evaluation Metrics

The evaluation protocol follows the VoicePrivacy Challenge 2024 (VPC24) and 2026 (VPC26) Track 1 protocols. The reported metrics include word error rate (WER↓) for intelligibility, unweighted average recall (UAR↑) for emotion preservation, and equal error rate (EER↑) for speaker privacy. EER is reported with lazy-informed (EER-L) attackers trained on original speech and semi-informed (EER-S) attackers trained on anonymized speech. The system demonstrates a substantial gain in EER-S (32.28 vs. 15.92) under the matched vctk-1fix setting.

Improvements for AI systems

Bold header: Direct Acoustic Conditioning on Streaming Features

LiveAnon directly conditions acoustic generation on hidden representations from a pretrained streaming Zipformer ASR encoder, allowing it to retain substantial emotion-related information while exhibiting less speaker discriminability. This enables the system to preserve emotion without requiring emotion-specific supervision or additional components for prosody modeling, addressing the challenge that existing systems often lose emotion information through discrete content bottlenecks.

Bold header: Capped RVQ Decoding for Latency Control

The introduction of capped RVQ decoding, which preserves dependencies among coarse codebooks while predicting the remaining codebooks in parallel, directly targets latency reduction. This mechanism allows LiveAnon to operate efficiently in real-time streaming applications by modifying the MusicGen delay pattern to limits the delay to three codec frames and parallelizes the remaining RVQ codebooks.

Bold header: Enhanced Privacy-Utility Trade-off Management

LiveAnon achieves a strong privacy–utility balance by maintaining competitive performance against prior systems, notably achieving an EER-S of 32.28 under VPC24 while using only a single fixed VCTK prompt. This demonstrates that the method successfully balances the need for anonymization with preserving linguistic and paralinguistic information such as emotion, showing that LiveAnon maintains a competitive and well-balanced privacy–utility performance against offline VPC26 baselines.