Edge Phoneme Recognition for Children's Speech through Age-Aware Training

Detecting phonemes from children’s speech has historically been difficult due to the scarcity of training data, and unique characteristics of children’s speech. During a phoneme detection competition, we found that training a lightweight model to predict the age of the learner, as well as the phoneme sequence, enabled a 94M-parameter model to outperform WavLM Large models (317M) on the target DrivenData distribution, and fall within approximately 0.04 CER of competition ensembles with 90 times the parameters. This has enabled the creation of PhonemeTrainer, an application that can run on most modern cellular phones. This will ultimately enable better Automated Speech Recognition (ASR) and pronunciation helper apps for children’s speech, with the privacy and compliance benefits that come with edge processing.

The Age-aware training utilized a WavLM Base+ (94M parameters) pretrained encoder and two decoder heads, one that predicts the age bucket, and one that uses Connectionist Temporal Classification to predict International Phonetic Alphabet symbols. Predicting age was not part of the competition criteria, but there was some hope that adding this head would result in the model learning a regularization signal. The age prediction was discarded for the purposes of the competition. While training, this model converged three times faster than any other model we trained. All runs shared the same 117,500 utterance training set (10,696 DrivenData + 106,804 TalkBank), split by child ID to prevent data leakage. As Table 1 shows, the age-aware model was not merely competitive with WavLM Large (317M parameters) configurations—it matched the Large RNN-T on mixed CER (both 0.306) and outperformed it on the target DrivenData distribution (0.306 vs. 0.343), despite being 3.4× smaller. The WavLM Large + RNN-T model did perform better on TalkBank (TB) data, no doubt due to the training data imbalance. On the actual competition DrivenData (DD), the age-aware model performed significantly better, neither under- nor overfitting on either subset of the data. This suggests that the age-auxiliary objective encourages the encoder to learn age-invariant phoneme representations rather than overfit on the more numerous (10:1 ratio) TalkBank data.

Post-training tests of the Age-aware head showed that it was accurately predicting the age bucket of the speech samples with 72% accuracy, which essentially meant predicting the dominant class every time. So while the head clearly sent a useful signal to the loss function, it was a subpar predictor. We conjecture that the age gradient may encourage age-invariant phoneme representations, preventing the encoder from overfitting to the acoustic characteristics of the majority age group. Future research will apply mechanistic interpretability techniques to explore the mechanisms by which this age-aware objective helps training and inference. The most important implication of this unique combination of size, accuracy, and quick convergence means that the model takes up a relatively small amount of device memory and can easily be trained and retrained as necessary to mitigate distribution drift. This model can then serve as a stable backbone for a number of speech recognition and pronunciation feedback applications, without sending children’s speech data over the internet. This has significant privacy implications in countries with strict regulations (laws like COPPA, FERPA, GDPR) governing the transmission of children’s speech and academic records. A key technical limitation arises when trying this with larger, more accurate models such as the finetuned WavLM-large (317M parameters) backbones. Those models often exceed 1 GB, creating challenges for mobile deployment including longer initial download times, RAM requirements that exceed Android’s per-app memory limits, and inference delays that make real-time feedback impractical on mid-range devices.

This demonstration will show the phoneme transcription capabilities in real-time on an Android phone, as part of an application that gives users feedback on their pronunciation of short English, Spanish, or Korean sentences. During this demonstration participants will be asked to speak phrases into an Android device. These phrases will then be analyzed via an application called PhonemeTrainer. They will then be able to analyze the phonetic output of their speech, and analyze the accuracy to the reference phonetic sequence for that particular phrase. The application performs this analysis of participant speech by routing the.wav file to the finetuned WavLM Base+ acoustic model, which outputs a symbolic phoneme sequence. This spoken sequence is then mapped to ARPAbet phonemes and compared to the ground truth from the CMU Pronouncing Dictionary. Users are then shown a visual representation of their spoken sequence, and where, if at all, their phonetic output diverges from the ground truth sequence. The demonstration video shows two audio samples, a three year old child and an adult each saying, “this is a test”, to illustrate how the tool responds to different speaker profiles and voice characteristics. There will likely be no children in attendance. To show how the application performs on children’s speech, we will pre-load several examples of children speaking the phrases for demonstration. The participants will speak target sentences drawn from a language proficiency assessment used in many U.S. schools, selected to cover a representative range of phonemes. To manage the model’s 360 MB size, the model weights are downloaded once from a remote host on first launch and saved to the device’s internal storage, so that all subsequent sessions run fully offline without re-downloading. Future work will focus on using this speech-to-phoneme model as a complement to a language model that can take multilingual phrases and generate plausible phoneme sequences. This will allow PhonemeTrainer to work on any sentences, not just those that have been pre-loaded.

Improvements to AI Systems:

1. Age-Aware Multi-Task Learning for Robustness: Integrate an auxiliary age-prediction head into speech recognition models during training. This forces the encoder to learn age-invariant acoustic features, reducing overfitting to dominant age groups in imbalanced datasets. The improved system can generalize better across diverse speaker demographics (e.g., children vs. adults) without requiring additional labeled data for the target age group.

2. Efficient Edge-Deployable ASR Backbone: Use a lightweight pretrained encoder (e.g., WavLM Base+ at 94M parameters) with dual heads (age classification + phoneme CTC) instead of larger models (e.g., WavLM Large at 317M). The improved system achieves comparable or better accuracy on target distributions while being 3.4× smaller, enabling real-time inference on mid-range mobile devices with <1 GB memory footprint.

3. Rapid Convergence via Auxiliary Objectives: Adopt the age-prediction auxiliary loss to accelerate training convergence (3× faster than standard fine-tuning). The improved system can be retrained quickly on new data to mitigate distribution drift, making it practical for continuous deployment in dynamic environments (e.g., classroom settings with changing accents or noise conditions).

4. Privacy-Preserving On-Device Processing: Design the ASR pipeline to run entirely offline after a one-time model download (360 MB). The improved system processes children’s speech locally, avoiding transmission of sensitive audio over the internet, thus complying with regulations like COPPA, FERPA, and GDPR while enabling real-time pronunciation feedback.

5. Cross-Lingual Phoneme Mapping for Flexible Feedback: Combine the speech-to-phoneme model with a language model that generates plausible phoneme sequences for arbitrary multilingual phrases. The improved system can provide pronunciation feedback for any sentence (not just pre-loaded ones) by mapping output to ARPAbet and comparing against reference sequences from dictionaries like CMU, supporting English, Spanish, and Korean.

6. Scalable Training with Data Leakage Prevention: Use child-ID-based split for training/validation to prevent data leakage, ensuring the model learns speaker-independent features. The improved system can be trained on heterogeneous datasets (e.g., TalkBank + competition data) without overfitting to specific children, leading to more reliable phoneme detection in unseen speakers.

What the Improved AI System Can Do:

• Accurately transcribe phonemes from children’s speech in real-time on a standard Android phone, matching or exceeding larger models on target distributions (e.g., 0.306 CER on DrivenData).

• Provide immediate, visual pronunciation feedback for short sentences in English, Spanish, or Korean, highlighting phonetic divergences from reference sequences.

• Operate fully offline after initial download, ensuring child speech data never leaves the device, enhancing privacy and regulatory compliance.

• Adapt quickly to new acoustic conditions or demographic shifts via rapid retraining, maintaining high accuracy without requiring massive computational resources.

• Extend to arbitrary sentences via a companion language model, enabling dynamic, personalized pronunciation coaching beyond fixed phrase lists.

Detecting phonemes from children's speech has historically been difficult due to the scarcity of training data, and unique characteristics of children's speech. During a phoneme detection competition, we found that training a lightweight model to predict the age of the learner, as well as the phoneme sequence, enabled a 94M-parameter model to outperform WavLM Large models (317M) on the target DrivenData distribution, and fall within approximately 0.04 CER of competition ensembles with 90 times the parameters. This has enabled the creation of PhonemeTrainer, an application that can run on most modern cellular phones. This will ultimately enable better Automated Speech Recognition (ASR) and pronunciation helper apps for children's speech, with the privacy and compliance benefits that come with edge processing.