From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models

summary

The gist

Full-duplex spoken dialogue models aim to make voice agents more natural by allowing them to listen, speak, pause…

In short

Researchers developed French Full-Duplex Benchmarks (CALLFC-FDB and MEDIA-FDB) to evaluate how spoken dialogue models handle real conversations in different languages. The study found that timing metrics are similar across languages, but content evaluation, especially for user interruptions, degrades significantly when the model language mismatches the benchmark language. This suggests optimizing for naturalness requires considering both acoustic timing and language-specific interaction competence.

Key concepts

Full-Duplex Spoken Dialogue Models
These are AI agents that can listen, speak, pause, and respond all at once during a conversation. They aim to make voice interactions feel more natural by allowing for simultaneous listening and speaking rather than just one-way turns.
Timing-Based Metrics
These metrics measure the speed and flow of dialogue events like Takeover Rate (TOR) or latency. The study found these metrics tend to behave similarly regardless of the language being used, suggesting a general acoustic competence that is language-agnostic.
Content-Based Evaluation Degradation
This refers to how well a model handles the actual meaning and appropriateness of dialogue, such as responding correctly after an interruption. The paper shows this type of evaluation drops sharply when the model's language does not match the benchmark language, highlighting language dependency in interaction quality.

Terminology used across episodes

This episode discusses

The paper

From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models · Read on arXiv

Hamid Soltani, Gilles Boulianne

Luqia Technologies · Natural Sciences and Engineering Research Council of Canada · Ministry of Economy and Innovation of the Government of Québec

Full-duplex spoken dialogue models aim to make voice agents more natural by allowing them to listen, speak, pause, and respond during ongoing conversation. However, it is not clear whether full-duplex benchmarks behave the same way when models are evaluated in a different language. To investigate this, we introduce a French full-duplex benchmark (FDB) with two variants, CALLFC-FDB for Canadian French and MEDIA-FDB for European French, and compare them with an English FDB. Built from real spoken resources, these benchmarks evaluate key full-duplex skills, including pause handling, turn-taking, backchannels, and interruptions. Beyond introducing French FDBs, we evaluate human--human conversations with FDB metrics to better understand the values these metrics take in real-world dialogue. Our analysis reveals that most timing-based metrics behave similarly across languages, while content-based evaluation degrades under language mismatch. We also find a trade-off between optimizing benchmark metrics and preserving conversational naturalness.

DOI: 10.18653/2026.emnlp-industry.137

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "From Metrics to Natural Dialogue".

Dev: Here's the gist of the paper. Full-duplex spoken dialogue models aim to make voice agents more natural by allowing them to listen, speak, pause, and respond during ongoing conversation The analysis reveals that most timing-based metrics behave similarly across languages, while content-based evaluation degrades under language mismatch Full-duplex spoken dialogue models reduce the gap between half-duplex systems by listening and speaking at the same time, supporting turn-taking (Raux and Eskenazi, 2012), backchannels (Schegloff, 1982), and overlapping speech (Schegloff, 2000) These systems generally follow two architectural families: cascaded and speech-to-speech systems Cascaded systems involve converting user speech into text, generating a response with an LLM, and converting it back to speech Speech-to-speech systems process and generate speech directly, making them more suitable for full-duplex interaction where models must listen while speaking and react to interruptions Benchmark Development for French Dialogue The researchers introduce two variants of the French Full-Duplex-Bench: CALLFC-FDB for Canadian French and MEDIA-FDB for European French These benchmarks are built from real spoken resources, specifically CALLFRIEND and MEDIA corpora The tasks adapted from FDB v1.0 include Pause Handling, Backchanneling, Smooth Turn Taking, and User Interruption For MEDIA-FDB, the backchanneling task was omitted due to a lack of valid examples for that dataset The transcription process differs between datasets, with CALLFC-FDB using manual transcription with automatic channel alignment and MEDIA-FDB relying on manually transcribed text with MFA-derived word-level timing Metric Behavior Across Languages The evaluation metrics are categorized into timing-based and content-based measures Most timing-based metrics, such as Takeover Rate (TOR) and latency, behave similarly across languages However, content-based evaluation degrades under language mismatch The rating metric for user interruption is noted as the main exception because it depends on response content, such as an English response to a French interruption receiving a low rating Comparative Model Performance The evaluation compares English and French SDMs using reference models like Freeze-Omni and PersonaPlex Results show that for pause handling, English FDB generally gives lower or comparable TOR values across both English and French models For smooth turn-taking, cascaded models achieve higher TOR when the benchmark language matches the model language The user-interruption rating is the most language-dependent metric, dropping sharply when the model language does not match the benchmark language The study concludes that evaluation language, system architecture, and latency should be considered together when selecting and evaluating full-duplex systems Human–human conversations provide a natural reference point, showing that optimizing FDB metrics does not always align with conversational naturalness The findings highlight the need to distinguish between acoustic-timing competence and languagedependent interaction competence in multilingual full-duplex evaluation Future work will investigate whether this language dependency extends to other FDB v1.5 and v2.0 tasks The paper also notes that benchmark scores may not fully reflect human-like interaction Human Reference Value Beyond agent-generated interactions, the researchers evaluate human–human conversations with FDB metrics to establish reference values for how these metrics behave in real dialogue These conversations provide a natural reference point for interpreting metric-based evaluation The analysis shows a trade-off between optimizing FDB metrics and preserving conversational naturalness Human–human baselines show that humans exhibit low pause-handling TOR but strong turn-taking and interruption TOR On MEDIA-FDB, the French cascade received a higher rating than humans, indicating that quality and naturalness may differ The paper emphasizes that benchmark scores should be interpreted together with qualitative analysis and human-centered evaluation Limitations of the Study The work adapts the French Full-Duplex-Bench only for FDB v1 tasks, meaning future work should extend these benchmarks to FDB v1.5 and FDB v2 The latency comparison is limited by the lack of a fast French speech-to-speech model The language comparison varies in corpus, domain, and conversational formality across the English FDB, CALLFC-FDB, and MEDIA-FDB The study only evaluates language mismatch for FDB v1; it remains important to examine how mismatched evaluation languages affect FDB v1.

Title and authors: Rosa: So we’re looking at this paper today, "From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models," and it’s about how we measure these full-duplex systems.

Dev: Yeah, Hamid Soltani and Gilles Boulianne are testing whether the way we score these models changes when you switch languages.

Rosa: Exactly. They introduce two French benchmarks, CALLFC-FDB for Canadian French and MEDIA-FDB for European French, to compare against an English FDB setup.

Taro: So they’re building this from real spoken resources, like CALLFRIEND and MEDIA corpora, to test things like pause handling and backchannels.

Rosa: Right. And the main point they find is that most timing-based metrics are pretty similar across languages, but content-based evaluation really suffers when there's a language mismatch.

Dev: That’s a big deal because it means if you tune your model just to be fast, you might not actually get good conversational flow if the languages don't match perfectly.

Rosa: Right. They also found that user interruption rating is the main exception because it depends on the actual content of the response, like responding in English when someone interrupts in French gets a low score.

Taro: That makes sense; handling an interruption requires understanding not just timing, but what's actually being interrupted and how to pivot correctly.

Dev: From an engineering standpoint, this suggests we need to focus our optimization efforts more on those content-based metrics when dealing with multilingual setups.

Rosa: They also point out that the architecture matters too; for instance, the English cascade seems to have lower latency compared to the French cascade across all their FDB datasets.

Taro: So even if we get good language matching, a slow system architecture can kill the real-time feel of a full-duplex interaction.

Dev: It does. And they also look at how different system architectures handle things like pause handling and turn taking, finding that for smooth turn taking, cascaded models do better when the benchmark language matches the model language.

Rosa: So it’s not just about building a fast pipeline; it’s about matching the architecture to what the specific task demands across different languages.

Taro: It sounds like they want us to stop treating all these FDB tasks as interchangeable, especially when you introduce another language into the mix.

Dev: Exactly. The paper suggests that evaluation language, system architecture, and latency have to be considered all at once when picking and testing these systems.

Rosa: So for the future, they’re looking at whether this language dependency extends to newer versions of these benchmarks, like FDB v1 point five and v2 point zero.

Taro: That’s important because if it keeps happening across different versions, we need to build in more robust handling for those specific linguistic nuances.

Dev: And they also brought in human-human conversations to set a baseline, showing that optimizing these benchmark metrics doesn't always align with how natural a conversation actually feels to people.

Rosa: They found that humans show low pause handling scores but strong turn taking and interruption scores, and they noted that for MEDIA-FDB, the French cascade got higher ratings than humans, suggesting quality and naturalness can be different things.

Taro: So we have a conflict between optimizing the benchmark numbers and actually sounding like a natural human conversation when we use these AI systems.

Dev: It really boils down to distinguishing between acoustic timing competence and interaction competence that depends on the language.

Rosa: So, as we wrap up this discussion on "From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models," it seems like the key isn't just hitting high numbers but understanding what those numbers mean in context.

Taro: I think they’re giving us a clearer map of where the system is strong and where the language mismatch causes real problems.

Dev: For me, it means we have to be careful not to over-optimize for timing metrics when deploying these systems in multilingual settings, especially regarding interruptions.

Rosa: It’s a reminder that we need that qualitative analysis alongside the quantitative data when we evaluate these full-duplex spoken dialogue models.

Taro: Good stuff for understanding where the real interaction competence lies versus just acoustic performance.

Dev: Let's keep an eye on how they handle those language-specific dependencies in the next set of tasks.

The paper's summary: Rosa: So, we're looking at how these researchers set up their French benchmarks for full-duplex spoken dialogue systems and what they actually found in that summary.

Dev: The main thing is that they introduced two specific French benchmarks, CALLFC-FDB for Canadian French and MEDIA-FDB for European French to compare against English models.

Rosa: Right. And the core finding there is a distinction between timing metrics—like how fast the AI responds—and content-based evaluation, which they found degrades when you mix languages.

Dev: That's because most timing metrics behave similarly across different languages, but the way an AI handles a language mismatch totally messes up its ability to be natural in conversation.

Rosa: They also pointed out that the user interruption rating is the biggest exception to this rule because it depends on what the response actually says, like getting a low score if you answer someone interrupting in French with an English sentence.

Dev: That makes sense for real interaction; timing is one thing, but being socially appropriate when there's a language clash in the flow of talk is another layer.

Rosa: They also compared different models, like English and French full-duplex systems, and they saw that cascaded models performed better on smooth turn taking when the benchmark language matched what the model was built for.

Dev: So it’s not just about which system is technically faster; it’s about whether the architecture aligns with the specific conversational demands of that language pairing.

Rosa: And they brought in human-human conversations too, using those French benchmarks to set a baseline, and they found that optimizing these AI metrics doesn't always match how natural a real human conversation actually feels.

Dev: They showed humans have low pause handling scores but strong turn taking and interruption scores, which tells you that just hitting high numbers on the benchmark doesn't guarantee quality or naturalness in dialogue.

Rosa: The researchers concluded that we need to look at the evaluation language, the system architecture, and latency all together when picking these systems for real-world use.

Dev: It suggests we need to separate acoustic timing competence from interaction competence that depends on language so we don't get fooled by just looking at fast response times.

Rosa: So it’s a reminder that when we evaluate these full-duplex systems, we have to be careful not to just optimize for the numbers and forget how the conversation actually flows.

Dev: And they also noted that this work was mostly focused on older full-duplex versions of these benchmarks, so future researchers need to check if this language dependency holds true for newer versions like v1 point five and v2 point zero.

Rosa: It sounds like the next step is figuring out if this language sensitivity carries over into more advanced systems, which is where we need to pay close attention next.

The paper's improvements: Tom: So we're looking at how these researchers set up their French benchmarks for full-duplex spoken dialogue systems and what they actually found in that summary, now focusing on what they propose to fix things.

Rosa: They suggest a few ways to improve this setup, starting with making the benchmarks even more language-aware.

Dev: They want us to use language-specific benchmarks like CALLFC-FDB for Canadian French and MEDIA-FDB for European French, which lets us better assess how timing metrics behave across those different languages.

Rosa: That makes sense because it helps us separate general acoustic timing competence from the actual interaction competence that depends on the language.

Dev: I agree. And they also push to focus our optimization efforts more on those content-based metrics, specifically the user interruption rating, because that's where things get sensitive to language mismatch.

Rosa: They want model tuning to prioritize natural conversational flow and appropriateness over just chasing timing numbers in the first place.

Dev: Plus, they suggest designing full-duplex systems with architectures that naturally have lower latency when comparing them against non-target language benchmarks, since we see the English cascade consistently having lower latency than the French cascade across all their datasets.

Rosa: That’s a practical engineering point; minimizing that LLM generation delay is crucial for real-time performance in multilingual situations.

Dev: And they also talk about improving backchannel handling by fine-tuning models using metrics like Backchannel Frequency and Jensen-Shannon Divergence, because they found that backchannel behavior depends on the model architecture itself, not just the languages involved.

Rosa: So it’s not just about changing the benchmark data; it’s about adjusting how we build and tune the AI systems themselves to handle those specific linguistic cues better.

Dev: They also point out that enhancing user interruption capabilities involves training on data that mimics real-world conversational cues, like French interjections such as "euh" or "ben," because those don't map directly onto English translations.

Rosa: So the implication is we need to integrate linguistic markers into the training data so the AI understands those small, specific interactional nuances.

Dev: It means we’re moving beyond just making sure the AI responds quickly to making sure it responds correctly within a specific cultural or linguistic context.

Rosa: They also flag a limitation: this work only adapts these benchmarks for older full-duplex versions, so future research needs to extend these improvements to FDB v1 point five and v2 point zero tasks.

Dev: That means we need more work on those newer tasks, otherwise, the improvements we find here might not apply to the more advanced systems coming out later.

Rosa: So it seems like the path forward involves a combination of better benchmarks, smarter model tuning based on content metrics, and architectural choices that minimize latency.

Dev: It’s a lot to juggle when you're trying to get real-time performance right while also aiming for high-quality conversational competence across different languages.

Conclusion: Tom: So we're wrapping up our discussion on "From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models." Basically, we looked at how these researchers structured their benchmarks and what they found about language differences and model performance.

Rosa: They really highlighted that the biggest issue isn't just getting a fast response; it’s how much the AI fails when the language of its input doesn't match the language of its output.

Dev: Right. The numbers show that timing metrics are pretty stable across languages, but once you get to content evaluation, like user interruptions, things really start degrading when there's a mismatch.

Taro: It changes how we think about failure modes in autonomy; it’s not just a hardware glitch or a slow loop rate issue, it’s a linguistic breakdown in understanding the interaction.

Rosa: Exactly. And they showed that even human conversations don't perfectly match the benchmark scores, which means we can't just chase those numbers without checking for actual conversational naturalness.

Dev: So for us working on real-time loops, this means we need to prioritize tuning based on those content metrics—the interruption rating—instead of just chasing a low latency number.

Taro: That makes sense for graceful recovery; if the AI handles the language context poorly during an interruption, it won't recover gracefully, no matter how fast its processing is.

Rosa: It’s a reminder that we need that qualitative analysis alongside the quantitative data when evaluating these full-duplex spoken dialogue models.

Dev: I think this paper gives us a clearer picture of where the system is strong and where the language mismatch causes real problems in French interactions specifically.

Taro: Yeah, it pushes us to design systems that are robust not just against technical errors, but against those kinds of linguistic surprises during real-world deployment.

Rosa: So we’re leaving this benchmark analysis there for now, but keep an eye on how these findings translate into practical safety cases for the next generation of full-duplex agents.

More episodes

← Home