PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification

arXiv:2609.10466 · eess.AS · Submitted 2026-09-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification".

Dev: Here's the gist of the paper. We study speaker-disjoint accent identification for L2 English where the goal is to predict a speaker’s first language (L1) background from English pronunciation.

Rosa: Most existing systems classify accents using a single utterance-level representation but such global representations can obscure pronunciation cues that depend on specific English phonemes.

Dev: We propose a transcript-assisted model that makes phoneme information explicit during accent identification.

Rosa: Instead of representing an utterance only as a global speech embedding we represent it as a sequence of pronunciation units each combining acoustic evidence from a spoken segment with the aligned English phoneme for that segment.

Paper summary: Rosa: So, looking at the title and authors, PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification by Yangyang Qu, Massimiliano Todisco, and Nicholas Evans is really about how to get better speaker identification for L2 English speakers.

Dev: It’s basically suggesting that preserving that level of detail—the phoneme level evidence—is beneficial when you're trying to predict the speaker’s first language background. They found this method achieves the highest mean accuracy and macro Fone on their test set, which was the L2-ARCTIC corpus with speakers from six different L1 groups like Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese.

Taro: What that means for us is that for accent identification in L2 English contexts, having this phoneme-level information explicitly preserved in the model’s structure helps you capture those subtle pronunciation patterns better than just using a general speech embedding.

Rosa: It confirms that organizing those frozen speech encoder features into these phoneme-conditioned pronunciation tokens is a strong way to improve speaker-disjoint identification performance.

Dev: The limitation they mentioned is that this approach requires transcripts and forced phoneme alignments both for training and for evaluation, which means it adds some manual work to the pipeline. Also, they noted that the use of the phoneme identifier conditioning provides a smaller additional gain once those phoneme-aligned acoustic tokens have already been constructed.

Taro: So, while this method is strong because it gives richer evidence, you can't just throw transcripts at it and expect perfect results without having that alignment step done first.

Rosa: That’s the trade-off they highlighted; the power comes from that explicit structure, but you have to manage the requirement for those alignments.

Conclusion: Rosa: So, we’re wrapping up the look at PAID, which stands for Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification by Qu, Todisco and Evans.

Dev: It basically means they took those frozen speech features and organized them into tokens that are tied directly to specific English phonemes.

Rosa: Right, so instead of just looking at a whole utterance as one big blob of sound, they're making sure the model sees exactly which sounds are there and what those sounds mean in terms of the speaker’s first language.

Dev: That explicit connection between local pronunciation and the English phoneme is where they found their main advantage.

Rosa: They put this method through a tough test on L2-ARCTIC, and it actually outperformed other systems by about eight percent in accuracy.

Dev: Eight percent means when you’re trying to figure out a speaker’s background from just their English accent, this phoneme-level detail is adding real value beyond what a global summary can give you.

Rosa: It suggests that for speaker identification based on accent, keeping the fine-grained pronunciation evidence intact is really helpful for getting more accurate results.

Dev: That brings up the question of how much manual work this introduces—they’re saying you need those transcripts and forced alignments to get it working.

Rosa: Exactly, so it’s a trade-off between getting better performance and having to do more upfront preparation with the audio data.

Dev: It confirms that preserving phoneme-level evidence is beneficial for L1-background accent identification, but we have to manage that requirement for transcripts and alignments.

Rosa: Now we know the win, but how does this actually work in a real system when you're trying to deploy it outside a controlled lab setting?

EURECOM

eess.AS

Submitted: 2026-09-09

Updated: 2026-11-03

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 85/100

The gist: We study speaker-disjoint accent identification for L2 English where the goal is to predict a speaker’s first language (L1) background from English pronunciation.

Key concepts

Speaker-Disjoint Protocol
This is a rigorous evaluation method used to test models. It ensures that a model's performance isn't just due to memorizing specific speakers. The protocol tests if the model can correctly identify an L1 background even when it hasn't seen those specific speakers during training, making the results more general and robust.
Phoneme-Aligned Acoustic Tokens
This is the core innovation of PAID. It takes raw speech features and organizes them into units that are explicitly tied to specific English phonemes (sounds). This makes the model focus on how a specific sound is pronounced, rather than just a general acoustic pattern across an entire utterance.
Transcript-Assisted Model
The model requires text transcripts alongside the audio. These transcripts are used to force an alignment between the speech and known English phonemes. This explicit connection between what is said (the transcript) and how it sounds (the phoneme) is crucial for building the improved representation.

Terminology

Summary

We study speaker-disjoint accent identification for L2 English where the goal is to predict a speaker’s first language (L1) background from English pronunciation. p. 0

Where the paper says it

Phoneme-Aware Pronunciation Representations for L2-English L1-Background Accent Identification Yangyang Qu1 , Massimiliano Todisco1 , Nicholas Evans1 1EURECOM, Sophia Antipolis, France Abstract—We study speaker-disjoint accent identification for L2 English, where the goal is to predict a speaker’s firstlanguage (L1) background from English pronunciation. Most existing systems classify accents using a single utterance-level representation, but such global representations can obscure pronunciation cues that depend on specific English phonemes.

Most existing systems classify accents using a single utterance-level representation but such global representations can obscure pronunciation cues that depend on specific English phonemes. p. 0
Where the paper says it

Phoneme-Aware Pronunciation Representations for L2-English L1-Background Accent Identification Yangyang Qu1 , Massimiliano Todisco1 , Nicholas Evans1 1EURECOM, Sophia Antipolis, France Abstract—We study speaker-disjoint accent identification for L2 English, where the goal is to predict a speaker’s firstlanguage (L1) background from English pronunciation. Most existing systems classify accents using a single utterance-level representation, but such global representations can obscure pronunciation cues that depend on specific English phonemes.

We propose a transcript-assisted model that makes phoneme information explicit during accent identification. p. 0
Where the paper says it

Most existing systems classify accents using a single utterance-level representation, but such global representations can obscure pronunciation cues that depend on specific English phonemes. We propose a transcript-assisted model that makes phoneme information explicit during accent identification.

Instead of representing an utterance only as a global speech embedding we represent it as a sequence of pronunciation units each combining acoustic evidence from a spoken segment with the aligned English phoneme for that segment. p. 0
Where the paper says it

We propose a transcript-assisted model that makes phoneme information explicit during accent identification. Instead of representing an utterance only as a global speech embedding, we represent it as a sequence of pronunciation units, each combining acoustic evidence from a spoken segment with the aligned English phoneme for that segment.

Under a four-fold speaker-disjoint protocol on L2-ARCTIC our model achieves 81. p. 0
Where the paper says it

No word-level or sentence-level text representation is passed to the accent classifier. Under a four-fold speaker-disjoint protocol on L2-ARCTIC, our model achieves 81.

41% accuracy and 81. p. 0
Where the paper says it

Under a four-fold speaker-disjoint protocol on L2-ARCTIC, our model achieves 81. 41% accuracy and 81.

p. 0
Where the paper says it

Under a four-fold speaker-disjoint protocol on L2-ARCTIC, our model achieves 81. 41% accuracy and 81.

p. 0
Where the paper says it

Under a four-fold speaker-disjoint protocol on L2-ARCTIC, our model achieves 81. 41% accuracy and 81.

21% macro-F1 the highest mean performance among the evaluated systems. p. 0
Where the paper says it

41% accuracy and 81. 21% macro-F1, the highest mean performance among the evaluated systems.

Addressing Representation Limitations

The paper addresses a representation-level limitation of global or frame-aggregated accent representations which do not explicitly connect local pronunciation evidence with the English phoneme being realized. The authors argue that utterance-level pooling mixes local pronunciation cues into a single vector making their connection to the intended English phonemes less explicit. p. 0

Where the paper says it

For example, L1-background information may appear in how speakers realise vowels, consonants, rhotics, or segment timing. Utterance-level pooling mixes these local pronunciation cues into a single vector, which makes their connection to the intended English phonemes less explicit.

They propose PAID which organizes frozen Whisper features into phoneme-level acoustic tokens. p. 0
Where the paper says it

We propose PAID, which is a transcript-assisted model that pairs segment-level acoustic evidence with aligned phoneme information for accent identification. Instead of classifying an utterance using only a global embedding, PAID organises frozen Whisper features into phoneme-level acoustic tokens.

This involves using a forced aligner to provide phoneme spans and aligned phoneme identifiers including phoneme spans and aligned phoneme IDs. p. 0
Where the paper says it

Instead of classifying an utterance using only a global embedding, PAID organises frozen Whisper features into phoneme-level acoustic tokens. A forced aligner uses the transcript to provide phoneme spans and aligned phoneme identifiers (IDs).

The Training Signal Construction

The model's training signal is built by first extracting frame-level acoustic features from a frozen speech encoder like Whisper-small. These features are then grouped into phoneme-level acoustic tokens using span mean pooling to get one acoustic token for each valid aligned phoneme realisation. p. 2

Where the paper says it

(3) Span mean pooling converts the grouped frames into one acoustic pronunciation token: zi = 1 |Ti | X t∈Ti ht. (4) This produces one acoustic token for each valid aligned phoneme realisation.

Each acoustic token is then mapped to a task-specific representation by projecting it and concatenating it with a learnable phoneme-ID embedding. p. 2
Where the paper says it

(5) The projection module maps each span-pooled acoustic token to a task-specific token representation. In parallel, the aligned phoneme ID pi is mapped to a learnable phoneme-ID embedding.

The final utterance representation is obtained by masked mean pooling these phoneme-conditioned tokens. p. 3
Where the paper says it

Padded positions have mi = 0 and are excluded from utterance-level aggregation. Our final Whisper-based model aggregates valid tokens by masked mean pooling: u = 1 PM i=1 mi X M i=1 mi˜ri .

Where the Gains Come From

The primary gains come from organizing frozen speech features into phoneme-aligned pronunciation tokens which makes the comparison reflect predominantly the representation structure rather than changes in the acoustic front-end. Diagnostic ablations show that phoneme-aligned token construction is important and a Whisper-based ablation shows an additional gain from phoneme information. p. 0

Where the paper says it

21% macro-F1, the highest mean performance among the evaluated systems. Diagnostic ablations support the importance of phoneme-aligned token construction, while a Whisper-based ablation shows an additional gain from phoneme information.

The WavLM diagnostic setting shows the largest observed separation between global/frame-level summaries and phoneme-aligned token construction. p. 1
Where the paper says it

We show that PAID achieves the highest mean accuracy and macro-F1 among the evaluated systems. 3) The WavLM diagnostic setting shows the largest observed separation between global/frame-level summaries and phoneme-aligned token construction, while the final Whisper-based ablation shows a smaller additional gain from phoneme-ID information.

Performance Against Baselines

The proposed Whisper-based PAID achieves 81.41% accuracy and 81.21% macro-F1 under a four-fold speaker-disjoint protocol on L2-ARCTIC. p. 1

Where the paper says it

2) We evaluate PAID under a four-fold speaker-disjoint protocol on L2-ARCTIC, where a speaker from each L1- background group is held out in each fold. We show that PAID achieves the highest mean accuracy and macro-F1 among the evaluated systems.

Compared with Whisper-Utt PAID improves accuracy by 4. p. 4
Where the paper says it

21% macro-F1. p. 0

Where the paper says it

41% accuracy and 81. 21% macro-F1, the highest mean performance among the evaluated systems.

Compared with Whisper-Utt, PAID improves accuracy by 4.

p. 4
Where the paper says it

21% macro-F1. Compared with Whisper-Utt, PAID improves accuracy by 4.

29 percentage points and macro-F1 by 4. p. 4
Where the paper says it

Compared with Whisper-Utt, PAID improves accuracy by 4. 29 percentage points and macro-F1 by 4.

p. 4
Where the paper says it

Compared with Whisper-Utt, PAID improves accuracy by 4. 29 percentage points and macro-F1 by 4.

41 percentage points. p. 4
Where the paper says it

Compared with Whisper-Utt, PAID improves accuracy by 4. 29 percentage points and macro-F1 by 4.

p. 4
Where the paper says it

Compared with Whisper-Utt, PAID improves accuracy by 4. 29 percentage points and macro-F1 by 4.

Compared with VoxlectWhisper PAID improves accuracy by 4. p. 4
Where the paper says it

41 percentage points. Compared with VoxlectWhisper, PAID improves accuracy by 4.

02 percentage points and macro-F1 by 4. p. 4
Where the paper says it

Compared with Whisper-Utt, PAID improves accuracy by 4. 29 percentage points and macro-F1 by 4.

45 percentage points. p. 4
Where the paper says it

Compared with Whisper-Utt, PAID improves accuracy by 4. 29 percentage points and macro-F1 by 4.

The results suggest that phoneme-conditioned pronunciation tokens provide complementary evidence beyond utterance-level representations in this setting. p. 4
Where the paper says it

Its performance is close to that of Whisper-Utt, suggesting that large-data accent pretraining can provide competitive utterance-level representations. However, the improvement of PAID over both WhisperUtt and Voxlect-Whisper suggests that phoneme-conditioned pronunciation tokens provide complementary evidence beyond utterance-level representations in this setting.

Limitations and Future Work

The main limitation is that the approach requires transcripts and forced phoneme alignments for both training and evaluation. The current evaluation is also limited to one readspeech corpus with 24 speakers. p. 5

Where the paper says it

The main limitation is that our approach requires transcripts and forced phoneme alignments for both training and evaluation. The current evaluation is also limited to one readspeech corpus with 24 speakers.

Future work should study the influence of ASR-derived transcripts and approaches that reduce dependence on manual transcripts. p. 5
Where the paper says it

Larger and more diverse L2- English corpora are needed to support a more comprehensive evaluation. Future work should study the influence of ASRderived transcripts, approaches that reduce dependence on manual transcripts, and evaluation across broader L2-English speaking conditions.

The paper also notes that phoneme-ID conditioning provides a smaller additional gain once phoneme-aligned acoustic tokens are already constructed. p. 5
Where the paper says it

This reduces macro-F1 from 72. Thus, phoneme-ID conditioning provides a smaller additional gain once phoneme-aligned acoustic tokens are already constructed.

Conclusion

Overall these findings suggest that preserving phoneme-level pronunciation evidence is beneficial for L1-background accent identification. The paper concludes by stating that the main limitation is the requirement for transcripts and forced phoneme alignments. p. 5

Where the paper says it

Overall, these findings suggest that preserving phoneme-level pronunciation evidence is beneficial for L1- background accent identification. The main limitation is that our approach requires transcripts and forced phoneme alignments for both training and evaluation.

The study confirms that organizing frozen speech encoder features into phoneme-conditioned pronunciation tokens is beneficial for speaker-disjoint L2-English L1-background accent identification. p. 1
Where the paper says it

We study a different representation question: whether frozen speech features are more effective for speaker-disjoint L2-English L1-background identification when organised into phoneme-aligned pronunciation tokens. Speaker-Generalised Accent Modeling In small accent corpora, L1-background labels can be confounded with individual speaker identity and speaker-specific timbral characteristics.

References

The paper cites numerous works supporting the context of accent identification and related acoustic models. These references cover various approaches including i-vector-based methods and pretrained speech encoders. p. 1

Where the paper says it

RELATED WORK A. Accent Identification with Pretrained Representations Accent identification has long been studied using acoustic, prosodic, and utterance-level factor-analysis representations, including i-vector-based approaches [10]–[12].

Specifically, the paper references work on accent-aware ASR and speaker verification embeddings. p. 0
Where the paper says it

Such representations may be derived from speaker encoders, automatic speech recognition (ASR) encoders, or pooled frame-level features extracted using pretrained speech models. Recent work has explored accent-specific embeddings, speaker-generalised accent representations, voice-conversionbased robust accent identification, and frame- or utterancelevel aggregation over pretrained acoustic features [2]–[5].

It also cites foundational work on forced aligners and large speech foundation models like WavLM. p. 0
Where the paper says it

Such representations may be derived from speaker encoders, automatic speech recognition (ASR) encoders, or pooled frame-level features extracted using pretrained speech models. Recent work has explored accent-specific embeddings, speaker-generalised accent representations, voice-conversionbased robust accent identification, and frame- or utterancelevel aggregation over pretrained acoustic features [2]–[5].

The references provide a broad context for the proposed method's contributions.

Index Terms

The index terms listed are accent identification L2 English phonemeconditioned tokens. These terms summarize the core concepts of the paper's contribution. The index terms highlight the key elements of the proposed framework. p. 0

Where the paper says it

Diagnostic ablations support the importance of phoneme-aligned token construction, while a Whisper-based ablation shows an additional gain from phoneme information. Index Terms—accent identification, L2 English, phonemeconditioned tokens I.

The index terms are relevant to accent identification in L2 English settings. p. 0
Where the paper says it

Diagnostic ablations support the importance of phoneme-aligned token construction, while a Whisper-based ablation shows an additional gain from phoneme information. Index Terms—accent identification, L2 English, phonemeconditioned tokens I.

Dataset and Evaluation Protocol

Evaluation is performed using L2-ARCTIC a read-speech corpus of non-native English speakers. The corpus contains data collected from 24 speakers from six L1-background groups: Arabic Chinese Hindi Korean Spanish and Vietnamese with four speakers per group. p. 3

Where the paper says it

Dataset and Evaluation Protocol Evaluation is performed using L2-ARCTIC, a read-speech corpus of non-native English speakers. The corpus contains data collected from 24 speakers from six L1-background groups: Arabic, Chinese, Hindi, Korean, Spanish, and Vietnamese, with four speakers per group.

All systems are evaluated using the same four-fold speaker-disjoint protocol. p. 3
Where the paper says it

We adopt the same label space and define the task as six-way L1-background accent identification from L2 English utterances. All systems and ablations reported in this paper are evaluated using the same four-fold speaker-disjoint protocol.

This protocol tests whether a model can recognise L1-influenced pronunciation patterns from unseen speakers while reducing the risk of speaker identity-based shortcuts. p. 3
Where the paper says it

Therefore, each test fold contains six unseen speakers, one from each L1-background group, and no speaker appears in both training and test sets. This protocol evaluates whether a model can recognise L1-influenced pronunciation patterns from unseen speakers while reducing the risk of speaker identity-based shortcuts.

Compared Systems

The paper compares its approach with speaker-embedding baselines utterance-level speech-representation baselines pretrained accent/dialect representation references and phonemetoken variants. The compared systems are grouped as ECAPA-Spk Whisper-Utt FrameAttn WavLM-based diagnostic variants GenAID and Voxlect-Whisper. p. 4

Where the paper says it

Accuracy and macro-F1 results for ECAPA-Spk are close to the six-way chance level, suggesting that speaker-verification embeddings are not sufficiently informative for this speaker-disjoint L1-background identification task. WavLM-Utt and GenAID obtain a lower mean macro-F1 than the Whisper-based systems.

These baselines serve to test different representation choices under the same frozen encoder. p. 3
Where the paper says it

• WavLM-based diagnostic variants. These variants use frozen WavLM Base+ features [24] to compare different representation choices under the same encoder.

Implementation Details

The full model uses a frozen Whisper-small encoder for the Whisper-based model. The acoustic front-end is fixed and only the downstream projection pooling or phoneme-ID embeddings are trained. p. 1

Where the paper says it

Only the acoustic-token projection, phoneme-ID embedding table, and classifier are trained. Frozen Speech Encoder and Phoneme Alignment We use a frozen speech encoder as the acoustic front-end.

Forced phoneme alignments are produced offline using MFA which requires the transcript at both training and evaluation time. p. 4
Where the paper says it

Forced phoneme alignments are produced offline using MFA. The transcript is required at both training and evaluation time to obtain phoneme spans and aligned phoneme IDs.

The projection maps 768-dimensional encoder features to 192-dimensional token representations for the final Whisper-based model. p. 4
Where the paper says it

Padding positions are excluded from utterance-level aggregation using a token mask. For the final Whisper-based model, the projection maps 768-dimensional encoder features to 192-dimensional token representations using a linear projection followed by layer normalisation and dropout.

The learning rate is set to 2 × 10−4 with a batch size of 16 and dropout probability of 0. p. 4
Where the paper says it

We train the proposed model with cross-entropy loss using Adam. The learning rate is 2 × 10−4 , the batch size is 16, the dropout probability is 0.

1.

Results and Analysis

The proposed Whisper-based PAID achieves 81.41% accuracy and 81.21% macro-F1. Compared with Whisper-Utt PAID improves accuracy by 4.29 percentage points and macro-F1 by 4.41 percentage points. The results show that phoneme-conditioned pronunciation tokens provide complementary evidence beyond utterance-level representations in this setting. p. 4

Where the paper says it

Its performance is close to that of Whisper-Utt, suggesting that large-data accent pretraining can provide competitive utterance-level representations. However, the improvement of PAID over both WhisperUtt and Voxlect-Whisper suggests that phoneme-conditioned pronunciation tokens provide complementary evidence beyond utterance-level representations in this setting.

The diagnostic results support phoneme-aligned token construction in the WavLM setting. p. 5
Where the paper says it

Under a four-fold speaker-disjoint protocol on L2-ARCTIC, PAID achieves the highest mean accuracy and macro-F1 among the evaluated systems. The diagnostic results support phoneme-aligned token construction in the WavLM setting, and the Whisper-based ablation shows a smaller additional gain from phoneme-ID conditioning.

The fold-wise behavior highlights why fourfold speaker-disjoint evaluation is used instead of relying on a single random split. p. 5
Where the paper says it

This suggests that phoneme-ID conditioning may be helpful on harder held-out speaker splits, although this fold-level observation should not be over-interpreted. The fold-wise variation also highlights why we use fourfold speaker-disjoint evaluation instead of relying on a single random split.

The results vary across L1-background groups showing that Hindi obtains a high F1 for most evaluated systems. p. 5
Where the paper says it

The results vary across the six groups. Hindi obtains a high F1 for most evaluated systems, whereas Chinese and Korean show more system-dependent differences.

The paper presents a transcript-assisted framework for speakerdisjoint L2-English L1-background accent identification. Instead of classifying an utterance using only a global speech representation we organize frozen speech-encoder features into phoneme-conditioned pronunciation tokens. p. 5

Where the paper says it

CONCLUSIONS We presented a transcript-assisted framework for speakerdisjoint L2-English L1-background accent identification. Instead of classifying an utterance using only a global speech representation, we organise frozen speech-encoder features into phoneme-conditioned pronunciation tokens.

Under a four-fold speaker-disjoint protocol on L2-ARCTIC PAID achieves the highest mean accuracy and macro-F1 among the evaluated systems. p. 1
Where the paper says it

2) We evaluate PAID under a four-fold speaker-disjoint protocol on L2-ARCTIC, where a speaker from each L1- background group is held out in each fold. We show that PAID achieves the highest mean accuracy and macro-F1 among the evaluated systems.

The overall findings suggest that preserving phoneme-level pronunciation evidence is beneficial for L1-background accent identification. p. 5
Where the paper says it

The diagnostic results support phoneme-aligned token construction in the WavLM setting, and the Whisper-based ablation shows a smaller additional gain from phoneme-ID conditioning. Overall, these findings suggest that preserving phoneme-level pronunciation evidence is beneficial for L1- background accent identification.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Phoneme-aware pronunciation representations for L2-English L1-Background Accent Identification can be implemented by organizing frozen speech encoder features into phoneme-conditioned tokens, which combines acoustic realization with its aligned phoneme identity. This allows the classifier to receive phoneme conditioned pronunciation evidence rather than only a global utterance embedding.

  2. The improved system can perform speaker-disjoint L1-background accent identification for L2 English by leveraging transcript assistance to obtain forced phoneme alignments, including phoneme spans and aligned phoneme IDs. This explicitly connects local pronunciation cues with the intended English phonemes, addressing the limitation where accent cues can be phoneme-dependent or segmental.

  3. The system achieves higher mean accuracy and macro-F1 by using a final Whisper-based model structure that aggregates valid tokens via masked mean pooling, specifically showing that phoneme-ID conditioning provides a smaller additional gain once phoneme-aligned acoustic tokens are already constructed. This demonstrates the benefit of pairing acoustic evidence with phoneme information.

Abstract

First-language (L1) background classification from second-language English speech aims to identify pronunciation-related acoustic cues that remain informative across speakers and corpora. Global pooling is commonly used to aggregate frame-level speech representations, but averaging features across an utterance may obscure local acoustic differences in how individual phonemes are pronounced. To model these pronunciation-related cues before utterance-level aggregation, we introduce Phoneme-Aligned Input Design (PAID), which constructs acoustic units from frozen Whisper representations using transcript-assisted forced alignment. PAID applies a shared nonlinear mapping to each acoustic unit before utterance-level aggregation and incorporates auxiliary phoneme-label embeddings. We construct a six-L1 dataset with speaker-disjoint partitions and evaluate cross-corpus performance on the Speech Accent Archive (SAA). PAID achieves 89.72% Macro-F1 on the source Test set and 80.14% on SAA. Matched comparisons show that phoneme-aligned acoustic-unit modeling improves SAA Macro-F1 by 13.44 percentage points over global pooling. To address uneven classification performance across L1 backgrounds, we additionally evaluate class-sensitive training, which improves observed weakest-class performance.

Sources

Related papers