PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification".
Dev: Here's the gist of the paper. We study speaker-disjoint accent identification for L2 English where the goal is to predict a speaker’s first language (L1) background from English pronunciation.
Rosa: Most existing systems classify accents using a single utterance-level representation but such global representations can obscure pronunciation cues that depend on specific English phonemes.
Dev: We propose a transcript-assisted model that makes phoneme information explicit during accent identification.
Rosa: Instead of representing an utterance only as a global speech embedding we represent it as a sequence of pronunciation units each combining acoustic evidence from a spoken segment with the aligned English phoneme for that segment.
Paper summary: Rosa: So, looking at the title and authors, PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification by Yangyang Qu, Massimiliano Todisco, and Nicholas Evans is really about how to get better speaker identification for L2 English speakers.
Dev: It’s basically suggesting that preserving that level of detail—the phoneme level evidence—is beneficial when you're trying to predict the speaker’s first language background. They found this method achieves the highest mean accuracy and macro Fone on their test set, which was the L2-ARCTIC corpus with speakers from six different L1 groups like Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese.
Taro: What that means for us is that for accent identification in L2 English contexts, having this phoneme-level information explicitly preserved in the model’s structure helps you capture those subtle pronunciation patterns better than just using a general speech embedding.
Rosa: It confirms that organizing those frozen speech encoder features into these phoneme-conditioned pronunciation tokens is a strong way to improve speaker-disjoint identification performance.
Dev: The limitation they mentioned is that this approach requires transcripts and forced phoneme alignments both for training and for evaluation, which means it adds some manual work to the pipeline. Also, they noted that the use of the phoneme identifier conditioning provides a smaller additional gain once those phoneme-aligned acoustic tokens have already been constructed.
Taro: So, while this method is strong because it gives richer evidence, you can't just throw transcripts at it and expect perfect results without having that alignment step done first.
Rosa: That’s the trade-off they highlighted; the power comes from that explicit structure, but you have to manage the requirement for those alignments.
Conclusion: Rosa: So, we’re wrapping up the look at PAID, which stands for Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification by Qu, Todisco and Evans.
Dev: It basically means they took those frozen speech features and organized them into tokens that are tied directly to specific English phonemes.
Rosa: Right, so instead of just looking at a whole utterance as one big blob of sound, they're making sure the model sees exactly which sounds are there and what those sounds mean in terms of the speaker’s first language.
Dev: That explicit connection between local pronunciation and the English phoneme is where they found their main advantage.
Rosa: They put this method through a tough test on L2-ARCTIC, and it actually outperformed other systems by about eight percent in accuracy.
Dev: Eight percent means when you’re trying to figure out a speaker’s background from just their English accent, this phoneme-level detail is adding real value beyond what a global summary can give you.
Rosa: It suggests that for speaker identification based on accent, keeping the fine-grained pronunciation evidence intact is really helpful for getting more accurate results.
Dev: That brings up the question of how much manual work this introduces—they’re saying you need those transcripts and forced alignments to get it working.
Rosa: Exactly, so it’s a trade-off between getting better performance and having to do more upfront preparation with the audio data.
Dev: It confirms that preserving phoneme-level evidence is beneficial for L1-background accent identification, but we have to manage that requirement for transcripts and alignments.
Rosa: Now we know the win, but how does this actually work in a real system when you're trying to deploy it outside a controlled lab setting?
EURECOM
eess.AS
Submitted: 2026-09-09
Updated: 2026-11-03
License: http://creativecommons.org/licenses/by-sa/4.0/
Importance score: 85/100
The gist: We study speaker-disjoint accent identification for L2 English where the goal is to predict a speaker’s first language (L1) background from English pronunciation.
Key concepts
- Speaker-Disjoint Protocol
- This is a rigorous evaluation method used to test models. It ensures that a model's performance isn't just due to memorizing specific speakers. The protocol tests if the model can correctly identify an L1 background even when it hasn't seen those specific speakers during training, making the results more general and robust.
- Phoneme-Aligned Acoustic Tokens
- This is the core innovation of PAID. It takes raw speech features and organizes them into units that are explicitly tied to specific English phonemes (sounds). This makes the model focus on how a specific sound is pronounced, rather than just a general acoustic pattern across an entire utterance.
- Transcript-Assisted Model
- The model requires text transcripts alongside the audio. These transcripts are used to force an alignment between the speech and known English phonemes. This explicit connection between what is said (the transcript) and how it sounds (the phoneme) is crucial for building the improved representation.
Terminology
Summary
We study speaker-disjoint accent identification for L2 English where the goal is to predict a speaker’s first language (L1) background from English pronunciation. p. 0 Phoneme-Aware Pronunciation Representations for
L2-English L1-Background Accent Identification
Yangyang Qu1
, Massimiliano Todisco1
, Nicholas Evans1
1EURECOM, Sophia Antipolis, France
Abstract—We study speaker-disjoint accent identification for
L2 English, where the goal is to predict a speaker’s firstlanguage (L1) background from English pronunciation. Most
existing systems classify accents using a single utterance-level
representation, but such global representations can obscure
pronunciation cues that depend on specific English phonemes. Phoneme-Aware Pronunciation Representations for
L2-English L1-Background Accent Identification
Yangyang Qu1
, Massimiliano Todisco1
, Nicholas Evans1
1EURECOM, Sophia Antipolis, France
Abstract—We study speaker-disjoint accent identification for
L2 English, where the goal is to predict a speaker’s firstlanguage (L1) background from English pronunciation. Most
existing systems classify accents using a single utterance-level
representation, but such global representations can obscure
pronunciation cues that depend on specific English phonemes. Most
existing systems classify accents using a single utterance-level
representation, but such global representations can obscure
pronunciation cues that depend on specific English phonemes. We propose a transcript-assisted model that makes phoneme
information explicit during accent identification. We propose a transcript-assisted model that makes phoneme
information explicit during accent identification. Instead of representing an utterance only as a global speech embedding, we
represent it as a sequence of pronunciation units, each combining
acoustic evidence from a spoken segment with the aligned English
phoneme for that segment. No word-level or sentence-level
text representation is passed to the accent classifier. Under a
four-fold speaker-disjoint protocol on L2-ARCTIC, our model
achieves 81. Under a
four-fold speaker-disjoint protocol on L2-ARCTIC, our model
achieves 81. 41% accuracy and 81. Under a
four-fold speaker-disjoint protocol on L2-ARCTIC, our model
achieves 81. 41% accuracy and 81. Under a
four-fold speaker-disjoint protocol on L2-ARCTIC, our model
achieves 81. 41% accuracy and 81. 41% accuracy and 81. 21% macro-F1, the highest
mean performance among the evaluated systems.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Addressing Representation Limitations
The paper addresses a representation-level limitation of global or frame-aggregated accent representations which do not explicitly connect local pronunciation evidence with the English phoneme being realized. The authors argue that utterance-level pooling mixes local pronunciation cues into a single vector making their connection to the intended English phonemes less explicit. p. 0 For
example, L1-background information may appear in how
speakers realise vowels, consonants, rhotics, or segment timing. Utterance-level pooling mixes these local pronunciation
cues into a single vector, which makes their connection to the
intended English phonemes less explicit. We propose PAID, which is a transcript-assisted model that
pairs segment-level acoustic evidence with aligned phoneme
information for accent identification. Instead of classifying
an utterance using only a global embedding, PAID organises
frozen Whisper features into phoneme-level acoustic tokens. Instead of classifying
an utterance using only a global embedding, PAID organises
frozen Whisper features into phoneme-level acoustic tokens. A forced aligner uses the transcript to provide phoneme spans
and aligned phoneme identifiers (IDs).Where the paper says it
Where the paper says it
Where the paper says it
The Training Signal Construction
The model's training signal is built by first extracting frame-level acoustic features from a frozen speech encoder like Whisper-small. These features are then grouped into phoneme-level acoustic tokens using span mean pooling to get one acoustic token for each valid aligned phoneme realisation. p. 2 (3)
Span mean pooling converts the grouped frames into one
acoustic pronunciation token:
zi =
1
|Ti
|
X
t∈Ti
ht. (4)
This produces one acoustic token for each valid aligned
phoneme realisation. (5)
The projection module maps each span-pooled acoustic token
to a task-specific token representation. In parallel, the aligned phoneme ID pi
is mapped to a
learnable phoneme-ID embedding. Padded positions have mi = 0
and are excluded from utterance-level aggregation. Our final Whisper-based model aggregates valid tokens by
masked mean pooling:
u =
1
PM
i=1 mi
X
M
i=1
mi˜ri
.Where the paper says it
Where the paper says it
Where the paper says it
Where the Gains Come From
The primary gains come from organizing frozen speech features into phoneme-aligned pronunciation tokens which makes the comparison reflect predominantly the representation structure rather than changes in the acoustic front-end. Diagnostic ablations show that phoneme-aligned token construction is important and a Whisper-based ablation shows an additional gain from phoneme information. p. 0 21% macro-F1, the highest
mean performance among the evaluated systems. Diagnostic
ablations support the importance of phoneme-aligned token
construction, while a Whisper-based ablation shows an additional
gain from phoneme information. We show that
PAID achieves the highest mean accuracy and macro-F1
among the evaluated systems. 3) The WavLM diagnostic setting shows the largest observed separation between global/frame-level summaries
and phoneme-aligned token construction, while the final
Whisper-based ablation shows a smaller additional gain
from phoneme-ID information.Where the paper says it
Where the paper says it
Performance Against Baselines
The proposed Whisper-based PAID achieves 81.41% accuracy and 81.21% macro-F1 under a four-fold speaker-disjoint protocol on L2-ARCTIC. p. 1 2) We evaluate PAID under a four-fold speaker-disjoint
protocol on L2-ARCTIC, where a speaker from each L1-
background group is held out in each fold. We show that
PAID achieves the highest mean accuracy and macro-F1
among the evaluated systems. 21% macro-F1. p. 0 41% accuracy and 81. 21% macro-F1, the highest
mean performance among the evaluated systems. 21% macro-F1. Compared with Whisper-Utt,
PAID improves accuracy by 4. Compared with Whisper-Utt,
PAID improves accuracy by 4. 29 percentage points and
macro-F1 by 4. Compared with Whisper-Utt,
PAID improves accuracy by 4. 29 percentage points and
macro-F1 by 4. Compared with Whisper-Utt,
PAID improves accuracy by 4. 29 percentage points and
macro-F1 by 4. Compared with Whisper-Utt,
PAID improves accuracy by 4. 29 percentage points and
macro-F1 by 4. 41 percentage points. Compared with VoxlectWhisper, PAID improves accuracy by 4. Compared with Whisper-Utt,
PAID improves accuracy by 4. 29 percentage points and
macro-F1 by 4. Compared with Whisper-Utt,
PAID improves accuracy by 4. 29 percentage points and
macro-F1 by 4. Its performance is
close to that of Whisper-Utt, suggesting that large-data accent
pretraining can provide competitive utterance-level representations. However, the improvement of PAID over both WhisperUtt and Voxlect-Whisper suggests that phoneme-conditioned
pronunciation tokens provide complementary evidence beyond
utterance-level representations in this setting.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Work
The main limitation is that the approach requires transcripts and forced phoneme alignments for both training and evaluation. The current evaluation is also limited to one readspeech corpus with 24 speakers. p. 5 The main limitation is that our approach requires transcripts
and forced phoneme alignments for both training and evaluation. The current evaluation is also limited to one readspeech corpus with 24 speakers. Larger and more diverse L2-
English corpora are needed to support a more comprehensive
evaluation. Future work should study the influence of ASRderived transcripts, approaches that reduce dependence on
manual transcripts, and evaluation across broader L2-English
speaking conditions. This reduces macro-F1 from 72. Thus,
phoneme-ID conditioning provides a smaller additional gain
once phoneme-aligned acoustic tokens are already constructed.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
Overall these findings suggest that preserving phoneme-level pronunciation evidence is beneficial for L1-background accent identification. The paper concludes by stating that the main limitation is the requirement for transcripts and forced phoneme alignments. p. 5 Overall, these findings suggest that preserving
phoneme-level pronunciation evidence is beneficial for L1-
background accent identification. The main limitation is that our approach requires transcripts
and forced phoneme alignments for both training and evaluation. We study a different representation
question: whether frozen speech features are more effective
for speaker-disjoint L2-English L1-background identification
when organised into phoneme-aligned pronunciation tokens. Speaker-Generalised Accent Modeling
In small accent corpora, L1-background labels can be confounded with individual speaker identity and speaker-specific
timbral characteristics.Where the paper says it
Where the paper says it
References
The paper cites numerous works supporting the context of accent identification and related acoustic models. These references cover various approaches including i-vector-based methods and pretrained speech encoders. p. 1 RELATED WORK
A. Accent Identification with Pretrained Representations
Accent identification has long been studied using acoustic, prosodic, and utterance-level factor-analysis representations, including i-vector-based approaches [10]–[12]. Such representations may be derived from speaker
encoders, automatic speech recognition (ASR) encoders, or
pooled frame-level features extracted using pretrained speech
models. Recent work has explored accent-specific embeddings,
speaker-generalised accent representations, voice-conversionbased robust accent identification, and frame- or utterancelevel aggregation over pretrained acoustic features [2]–[5]. Such representations may be derived from speaker
encoders, automatic speech recognition (ASR) encoders, or
pooled frame-level features extracted using pretrained speech
models. Recent work has explored accent-specific embeddings,
speaker-generalised accent representations, voice-conversionbased robust accent identification, and frame- or utterancelevel aggregation over pretrained acoustic features [2]–[5].Where the paper says it
Where the paper says it
Where the paper says it
Index Terms
The index terms listed are accent identification L2 English phonemeconditioned tokens. These terms summarize the core concepts of the paper's contribution. The index terms highlight the key elements of the proposed framework. p. 0 Diagnostic
ablations support the importance of phoneme-aligned token
construction, while a Whisper-based ablation shows an additional
gain from phoneme information. Index Terms—accent identification, L2 English, phonemeconditioned tokens
I. Diagnostic
ablations support the importance of phoneme-aligned token
construction, while a Whisper-based ablation shows an additional
gain from phoneme information. Index Terms—accent identification, L2 English, phonemeconditioned tokens
I.Where the paper says it
Where the paper says it
Dataset and Evaluation Protocol
Evaluation is performed using L2-ARCTIC a read-speech corpus of non-native English speakers. The corpus contains data collected from 24 speakers from six L1-background groups: Arabic Chinese Hindi Korean Spanish and Vietnamese with four speakers per group. p. 3 Dataset and Evaluation Protocol
Evaluation is performed using L2-ARCTIC, a read-speech
corpus of non-native English speakers. The corpus contains
data collected from 24 speakers from six L1-background
groups: Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese, with four speakers per group. We adopt the same
label space and define the task as six-way L1-background
accent identification from L2 English utterances. All systems and ablations reported in this paper are evaluated using the same four-fold speaker-disjoint protocol. Therefore, each test fold contains six unseen speakers, one
from each L1-background group, and no speaker appears in
both training and test sets. This protocol evaluates whether
a model can recognise L1-influenced pronunciation patterns
from unseen speakers while reducing the risk of speaker
identity-based shortcuts.Where the paper says it
Where the paper says it
Where the paper says it
Compared Systems
The paper compares its approach with speaker-embedding baselines utterance-level speech-representation baselines pretrained accent/dialect representation references and phonemetoken variants. The compared systems are grouped as ECAPA-Spk Whisper-Utt FrameAttn WavLM-based diagnostic variants GenAID and Voxlect-Whisper. p. 4 Accuracy and macro-F1 results for
ECAPA-Spk are close to the six-way chance level, suggesting
that speaker-verification embeddings are not sufficiently informative for this speaker-disjoint L1-background identification
task. WavLM-Utt and GenAID obtain a lower mean macro-F1
than the Whisper-based systems. • WavLM-based diagnostic variants. These variants use
frozen WavLM Base+ features [24] to compare different representation choices under the same encoder.Where the paper says it
Where the paper says it
Implementation Details
The full model uses a frozen Whisper-small encoder for the Whisper-based model. The acoustic front-end is fixed and only the downstream projection pooling or phoneme-ID embeddings are trained. p. 1 Only the acoustic-token projection,
phoneme-ID embedding table, and classifier are trained. Frozen Speech Encoder and Phoneme Alignment
We use a frozen speech encoder as the acoustic front-end. Forced phoneme alignments are produced offline using
MFA. The transcript is required at both training and evaluation
time to obtain phoneme spans and aligned phoneme IDs. Padding positions are
excluded from utterance-level aggregation using a token mask. For the final Whisper-based model, the projection maps
768-dimensional encoder features to 192-dimensional token
representations using a linear projection followed by layer
normalisation and dropout. We train the proposed model with cross-entropy loss using
Adam. The learning rate is 2 × 10−4
, the batch size is
16, the dropout probability is 0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Results and Analysis
The proposed Whisper-based PAID achieves 81.41% accuracy and 81.21% macro-F1. Compared with Whisper-Utt PAID improves accuracy by 4.29 percentage points and macro-F1 by 4.41 percentage points. The results show that phoneme-conditioned pronunciation tokens provide complementary evidence beyond utterance-level representations in this setting. p. 4 Its performance is
close to that of Whisper-Utt, suggesting that large-data accent
pretraining can provide competitive utterance-level representations. However, the improvement of PAID over both WhisperUtt and Voxlect-Whisper suggests that phoneme-conditioned
pronunciation tokens provide complementary evidence beyond
utterance-level representations in this setting. Under a four-fold
speaker-disjoint protocol on L2-ARCTIC, PAID achieves the
highest mean accuracy and macro-F1 among the evaluated systems. The diagnostic results support phoneme-aligned token
construction in the WavLM setting, and the Whisper-based
ablation shows a smaller additional gain from phoneme-ID
conditioning. This suggests that
phoneme-ID conditioning may be helpful on harder held-out
speaker splits, although this fold-level observation should not
be over-interpreted. The fold-wise variation also highlights why we use fourfold speaker-disjoint evaluation instead of relying on a single
random split. The results vary
across the six groups. Hindi obtains a high F1 for most
evaluated systems, whereas Chinese and Korean show more
system-dependent differences.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper presents a transcript-assisted framework for speakerdisjoint L2-English L1-background accent identification. Instead of classifying an utterance using only a global speech representation we organize frozen speech-encoder features into phoneme-conditioned pronunciation tokens. p. 5 CONCLUSIONS
We presented a transcript-assisted framework for speakerdisjoint L2-English L1-background accent identification. Instead of classifying an utterance using only a global speech
representation, we organise frozen speech-encoder features
into phoneme-conditioned pronunciation tokens. 2) We evaluate PAID under a four-fold speaker-disjoint
protocol on L2-ARCTIC, where a speaker from each L1-
background group is held out in each fold. We show that
PAID achieves the highest mean accuracy and macro-F1
among the evaluated systems. The diagnostic results support phoneme-aligned token
construction in the WavLM setting, and the Whisper-based
ablation shows a smaller additional gain from phoneme-ID
conditioning. Overall, these findings suggest that preserving
phoneme-level pronunciation evidence is beneficial for L1-
background accent identification.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Phoneme-aware pronunciation representations for L2-English L1-Background Accent Identification can be implemented by organizing frozen speech encoder features into phoneme-conditioned tokens, which combines acoustic realization with its aligned phoneme identity. This allows the classifier to receive
phoneme conditioned pronunciation evidence rather than only a global utterance embedding.
-
The improved system can perform speaker-disjoint L1-background accent identification for L2 English by leveraging transcript assistance to obtain
forced phoneme alignments, including phoneme spans and aligned phoneme IDs.
This explicitly connects local pronunciation cues with the intended English phonemes, addressing the limitation whereaccent cues can be phoneme-dependent or segmental.
-
The system achieves higher mean accuracy and macro-F1 by using a final Whisper-based model structure that aggregates valid tokens via masked mean pooling, specifically showing that
phoneme-ID conditioning provides a smaller additional gain once phoneme-aligned acoustic tokens are already constructed.
This demonstrates the benefit of pairing acoustic evidence with phoneme information.
Abstract
First-language (L1) background classification from second-language English speech aims to identify pronunciation-related acoustic cues that remain informative across speakers and corpora. Global pooling is commonly used to aggregate frame-level speech representations, but averaging features across an utterance may obscure local acoustic differences in how individual phonemes are pronounced. To model these pronunciation-related cues before utterance-level aggregation, we introduce Phoneme-Aligned Input Design (PAID), which constructs acoustic units from frozen Whisper representations using transcript-assisted forced alignment. PAID applies a shared nonlinear mapping to each acoustic unit before utterance-level aggregation and incorporates auxiliary phoneme-label embeddings. We construct a six-L1 dataset with speaker-disjoint partitions and evaluate cross-corpus performance on the Speech Accent Archive (SAA). PAID achieves 89.72% Macro-F1 on the source Test set and 80.14% on SAA. Matched comparisons show that phoneme-aligned acoustic-unit modeling improves SAA Macro-F1 by 13.44 percentage points over global pooling. To address uneven classification performance across L1 backgrounds, we additionally evaluate class-sensitive training, which improves observed weakest-class performance.
Sources
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models