Beyond Speaker Independence: Evaluating Cross-Lingual Acoustic-to-Articulatory Inversion Across Finnish and Russian

arXiv:2606.20478 · eess.AS · Submitted 2026-06-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Beyond Speaker Independence".

Dev: Here's the gist of the paper. Acoustic-to-articulatory inversion (AAI) remains challenging under domain shifts where changes in speaker attributes and cross-language conditions often degrade performance.

Rosa: The results indicate that cross-gender mismatch introduces moderate Pearson correlation declines (approx.

Dev: 0.05–0.10) relative to the in-domain baseline, whereas cross-language mismatch causes larger drops (approx 0.10- 0.20).

Rosa: Systematic Evaluation Framework The authors conduct a systematic evaluation under domain shifts and establish baseline benchmarks on FROST-EMA, a Finnish–Russian bilingual EMA corpus.

Paper summary: Rosa: So we're looking at this paper called "Beyond Speaker Independence: Evaluating Cross-Lingual Acoustic-to-Articulatory Inversion Across Finnish and Russian." Basically, the authors are testing how well systems that try to turn sound waves back into physical mouth movements work when you switch between different speakers and different languages.

Dev: Right. The main idea is that acoustic to articulatory inversion, or AAI, gets really tricky when you move across those domain shifts where speaker traits and language differences come into play, and this paper sets up a way to systematically check that performance using a specific corpus called FROST-EMA.

Taro: What's the bigger picture here? It sounds like they're trying to see if the acoustic signal alone is enough to reliably reconstruct the actual physical movements of the mouth when the language or gender changes.

Rosa: Exactly. They use this Finnish and Russian bilingual corpus, FROST-EMA, which is designed to be better than existing resources because it addresses that English bias and has a much wider range of speakers. So they set up benchmarks for three things: what we call articulatory targets, which are the raw coordinates versus something called tract variables; acoustic front-ends, like using standard MFCC features versus modern self-supervised features; and inversion back-ends, comparing a basic two layer BiLSTM model against a lighter attention based sequence model.

Dev: The setup is pretty thorough. They're not just looking at one way to do the inversion; they are checking different ways to get there, which is important for building something that actually works in the real world, you know, with latency and failure modes in mind.

Taro: And then they define specific ways to look at things too. They set up protocols for cross gender transfer within a language and cross language transfer within a gender. That means they are testing those specific mismatches directly to see how much the performance drops when that specific condition is met.

Rosa: The key finding here, which is what makes me interested, is how different those drops are. They found that when you switch between genders within the same language, say Finnish speakers moving from female to male, it causes a moderate drop in Pearson correlation of about zero point zero five to zero point one.

Paper summary: Dev: And cross-language mismatch—when you switch languages, like Finnish to Russian—that causes a much bigger drop. They report drops in the range of about zero point one to two tenths. That suggests the language barrier is a tougher problem for this inversion method than just the speaker's gender difference.

Taro: So what does that mean for real deployment? If we're building something that needs to work reliably across different groups, we have to be way more careful about how we handle language switching than just making sure our model is robust to different genders within one language.

Rosa: Right. And they also looked at the diagnostic power of these tract variables versus the raw coordinates. They found that both raw coordinates and tract variables give comparable overall accuracy, but the tract variables offer a diagnostic advantage because things break down differently depending on what part of the mouth you're looking at.

Dev: Specifically, they noted that lip tract variables stay pretty robust across different languages, but the tongue constriction location variables degrade substantially when moving between Finnish and Russian. That points to how much more specific those place contrasts and palatalization differences are in Russian versus Finnish.

Taro: So it's saying that the system learns something about where things are going that is tied directly to the specific articulation of a language, not just a general mouth shape concept. That’s a concrete detail we can use when designing failure modes for autonomy.

Rosa: And under cross language transfer, they saw the tongue constriction predictions visibly diverge from what the ground truth is during fast articulatory transitions, which are movements happening over about zero point zero four to zero point seven seconds. That's a measurable difference in timing that they flagged as a larger degradation than for just a language switch.

Dev: The front-end and back-end parts also gave some specific feedback on what works best. They found that the self supervised representations, like Wav2Vec two point zero or XLSR fifty three, consistently beat the traditional MFCC features under both matched and mismatched conditions <ref:2606.20478#pg2>.

Taro: So the acoustic input quality matters a lot here. The paper suggests that those modern self-supervised representations capture information that transfers better across different speakers and languages than just standard spectral features like MFCCs do.

Paper summary: Rosa: And looking at the back-end, they found that the BiLSTM model with two layers outperformed their lightweight attention based model across all the configurations they tested. That suggests a recurrent structure is still beneficial when you're training on these lower resource EMA datasets.

Dev: It’s interesting because it implies that for real-time systems, we might want to stick with something like a BiLSTM for the inversion part if we're constrained by computational resources, even if the attention based model seems theoretically lighter.

Taro: And they did show where things were weakest overall. They found that combining MFCCs with their lightweight attention model resulted in the lowest results, with Pearson correlation dropping between zero point two four and zero point two nine. That tells you that for this kind of task, you need richer input representations than just basic spectral features to get any useful result at all.

Rosa: So to wrap up on what this paper means for us right now, it’s about setting realistic expectations for cross-lingual AAI systems. It tells us that while we can get moderate drops when switching between genders, the challenges jump significantly when you switch languages because the underlying articulatory rules are different.

Dev: The authors are showing that using self supervised features in the front end and a recurrent model in the back end gives us better performance than older methods, but they also clearly flag that tongue variables show sensitivity to language contrasts, which is a big caution for any deployment.

Taro: For autonomy researchers, this means when we design recovery protocols, we need to account for that specific degradation in tongue prediction during rapid movements when the environment shifts from one language context to another.

Rosa: So the main point of "Beyond Speaker Independence: Evaluating Cross-Lingual Acoustic-to-Articulatory Inversion Across Finnish and Russian" is that you can't just treat speaker independence as a simple thing because it breaks down differently depending on whether you're changing gender or changing the language entirely.

Dev: It’s about establishing better benchmarks for when we evaluate these complex systems, using things like tract variables to see exactly where the coupling between sound and movement is breaking down across those different groups.

Taro: It shows us that we need more diverse data than just English or standard European corpora if we want our models to be truly robust when the real world throws a mix of speakers and languages at them.

Conclusion: Rosa: So we're wrapping up this deep dive on how acoustic to articulatory inversion works when you change speakers and languages, specifically looking at the paper "Beyond Speaker Independence: Evaluating Cross-Lingual Acoustic-to-Articulatory Inversion Across Finnish and Russian."

Dev: Right. The authors are testing those boundary conditions, seeing how much performance actually degrades when you mix language and gender shifts together in this specific bilingual corpus.

Taro: What this boils down to is that speaker independence isn't as simple as we thought; the acoustic signal carries a lot of language-specific movement information.

Rosa: They found that cross-gender mismatch causes a moderate drop, about zero point zero five to zero point one in Pearson correlation compared to the in-domain baseline.

Dev: And when you switch languages, the drop is bigger, roughly zero point one to two tenths. That's a significant difference in robustness across different types of domain shift.

Taro: It suggests that the specific articulatory rules for Russian and Finnish are quite different, especially for things like tongue position.

Rosa: Exactly. For example, the lip tract variables stay relatively stable across languages, but the tongue constriction variables show a big drop when you switch between those two languages.

Dev: That points to how much of the acoustic information is tied to specific place contrasts and palatalization rules unique to one language over another.

Taro: And during fast movements, those tongue prediction errors visibly diverge from what we know is true articulation during rapid changes in movement, about zero point zero four to zero point seven seconds.

Rosa: So the authors conclude that you can't just rely on a single model trained on one set of speakers and one language.

Dev: They show that using modern self-supervised acoustic features helps preserve some articulatory information even when the input speaker changes.

Taro: But they also flag a major limitation: tongue predictions get significantly worse during rapid transitions between languages, which is something we need to worry about for real-world recovery.

Rosa: The implication is that any system designed for cross-lingual speech reconstruction needs to explicitly account for those language-specific movement variations.

University of Eastern Finland, Finland

eess.AS

Submitted: 2026-06-18

Updated: 2026-10-31

Comments: Presented at Interspeech 2026 (IS2026) held at Sydney, Australia, updated with supplementary material and additional results

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: Acoustic-to-articulatory inversion (AAI) remains challenging under domain shifts where changes in speaker attributes and cross-language conditions often degrade performance.

Key concepts

Acoustic-to-Articulatory Inversion (AAI)
This is the process of taking sound recordings and trying to figure out the physical movements of the mouth and tongue that produced those sounds. The system uses acoustic features to predict articulatory positions, which is useful for understanding speech production.
Domain Shift
This refers to changes in the conditions under which a model is tested, such as using data from different speakers or different languages. The paper investigates how robust the AAI system is when moving from one specific group of speakers (e.g., Finnish men) to another (e.g., Russian women).
Tract Variables (TVs)
These are a compact, five-dimensional representation derived from the raw physical positions of the mouth and tongue. They encode constriction locations, which helps researchers understand how different languages use specific speech sounds by showing where the physical movements occur.
Cross-Gender Mismatch
This occurs when an inversion model is tested on data from a speaker of one gender (e.g., female) but the model was primarily trained on data from a different gender (e.g., male). The results show this mismatch causes a moderate decline in performance compared to the original, in-domain baseline.

Terminology

Summary

Acoustic-to-articulatory inversion (AAI) remains challenging under domain shifts where changes in speaker attributes and cross-language conditions often degrade performance. p. 0 The results indicate that cross-gender mismatch introduces moderate Pearson correlation declines (approx. p. 0 0.05–0.10) relative to the in-domain baseline, whereas cross-language mismatch causes larger drops (approx 0. p. 0 p. 010- 0.20).

Systematic Evaluation Framework

The authors conduct a systematic evaluation under domain shifts and establish baseline benchmarks on FROST-EMA, a Finnish–Russian bilingual EMA corpus. The evaluation benchmarks include (i) articulatory targets (raw EMA coordinates vs tract variables), (ii) acoustic front-ends (MFCC vs SSL features), and (iii) inversion back-ends (BiLSTM vs a lightweight attention-based sequence model). p. 0 They further define evaluation protocols for cross-gender transfer (within language) and cross-language transfer (within gender). p. 0 The experimental design is shown in Figure 1. p. 0

Dataset Composition and Preprocessing

FROST-EMA is a recently collected electromagnetic articulography corpus designed to address the limited coverage and English bias of existing parallel acoustic–EMA resources. It contains recordings from 18 bilingual speakers (11 native Finnish, 7 native Russian; 8 female, 10 male) under three controlled speaking conditions: native language (L1), second language (L2), and imitated L2 (fake foreign accent). p. 1 These speakers form four language-gender groups: FIN-M (5), FIN-F (6), RUS-M (5), and RUS-F (2). p. 1 Articulatory data were captured using five midsagittal sensors, with no jaw sensor: upper lip (UL), lower lip (LL), tongue tip (TT), tongue blade (TB), and tongue dorsum (TD). p. 1 EMA segments are automatically screened for reliability, preprocessed with a standardized pipeline explained later. p. 1

Model Architectures and Targets

The AAI pipeline consists of an acoustic front-end and an inversion back-end. The acoustic front-ends considered are a DSP-based spectral feature baseline (MFCCs) and modern self-supervised representations (Wav2Vec 2. p. 10, XLSR53, MMS-300m). p. 1 The inversion back-ends compared are a BiLSTM with two layers and a hidden size of 256 per direction, and a lightweight Transformer encoder (Attn-lite). p. 1 Articulatory target representations evaluated include Raw EMA coordinates and tract variables (TVs). p. 1 TVs encode a compact fivedimensional constriction-based representation from the physical (non-normalized) EMA coordinates. p. 1

Key Findings on Transfer Robustness

The results indicate that cross-gender mismatch introduces moderate Pearson correlation declines (approx. 0.05–0.10) relative to the in-domain baseline, whereas cross-language mismatch causes larger drops (approx 0.10- 0.20). For Finnish speakers, FIN-F→FIN-M consistently outperforms FINM→FIN-F for tongue sensors (TT: 0. p. 246 vs. 0.34). Since TD captures tongue dorsum positioning relevant to velar constrictions, this disparity may reflect the broader palatalization contrasts in Russian, which engage the tongue dorsum more extensively than Finnish velar consonants. p. 2 Overall, cross-gender transfer introduces a systematic correlation drop of approximately 0. p. 205–0.10 relative to in-domain results. p. 0

Diagnostic Power of Tract Variables

TVs and raw EMA achieve comparable aggregate accuracy, but TVs offer a diagnostic advantage. Lip TVs remain robust across languages, while tongue CL variables degrade substantially, suggesting that constriction location is more language-specific due to differences in place contrasts and palatalization between Finnish and Russian. p. 3 Under crosslanguage transfer, the tongue CL predictions visibly diverge from the ground truth during rapid articulatory transitions ∆r ≈ 0. p. 304 − 0.07, a larger degradation than for LA. p. 3

Front-end and Back-end Performance

SSL representations consistently outperform MFCCs under both matched and mismatched conditions. The front-end ranking is preserved under domain shift, indicating that articulatory information in these representations partially transfers across speakers and languages. p. 3 BiLSTM outperformed lightweight attention model across all configurations suggesting that recurrent inductive bias is beneficial under low-resource EMA training conditions. p. 3 The MFCC-Attn-lite combination yielded the weakest results overall (r = 0. p. 324-0.29), indicating that lightweight attention model requires richer input representations. p. 3

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Standardized Benchmarking for Cross-Lingual AAI: The improved system can rigorously evaluate Acoustic-to-articulatory inversion across different speakers and languages by defining evaluation protocols that isolate cross-gender (within language) and cross-language (within gender) shifts as independent factors.

  2. Robust Front-end Selection: The system can utilize self supervised learning representations like Wav2Vec 2.0 or MMS-300m to extract rich acoustic representations without requiring language-specific articulatory data, leading to better performance even when training data is limited or under domain shifts.

  3. Diagnostic Articulatory Analysis via Tract Variables: The improved system can provide a diagnostic capability by using tract variables, which the paper notes are more speaker-invariant than raw sensor positions, allowing researchers to determine which articulatory aspects transfer across domains by analyzing their specific correlation patterns.

  4. Architecture Selection Based on Inductive Bias: The system can select the appropriate inversion back-end based on performance; specifically, using a BiLSTM is shown to be beneficial because recurrent inductive bias benefits articulatory trajectory modeling, especially with lower-dimensional features.

  5. Targeted Feature Focus: For specific articulatory tasks, the system can prioritize certain features over others; for example, it can confirm that tongue CL variables are more language-specific, indicating that constriction location is more language-specific due to differences in place contrasts and palatalization.

Sources

Related papers