BabAR: from phoneme recognition to developmental measures of young children's speech production

arXiv:2603.05213 · eess.AS · Submitted 2026-03-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "BabAR: from phoneme recognition to developmental measures of young children's speech production".

Dev: Here's the gist of the paper. Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved<ref:2603.05213#pg2> Building on decades of data collection, we curate TinyVox, a corpus of more than half a million phonetically transcribed child vocalizations in English, French, Portuguese, German, and Spanish<ref:2603.05213#pg2> We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives<ref:2603.05213#pg5> BabAR’s automatic measures of speech maturity align with developmental estimates from the literature<ref:2603.05213#pg10>.

Rosa: Data Curation and Standardization The authors address the data barrier by curating TinyVox, a large-scale standardization of PhonBank comprising over half a million IPA-transcribed vocalizations from 560 children aged 6 months to 8 years across five languages<ref:2603.05213#pg3> The dataset was created from PhonBank

21: , and all available data in English, French, Portuguese, German, and Spanish were downloaded and audio files were paired with their transcript (CHAT format

22: )<ref:2603.05213#pg5> To create a more consistent cross-linguistic target inventory, they created a 57-sound set (30 consonants and 27 vowels) based on the adult phonemic inventories of all five languages<ref:2603.05213#pg9>.

Dev: Post-processing involved removing utterances with extreme durations or containing unidentified sounds, and implementing a two-pass sampling procedure to screen for systematic alignment issues in problematic corpora<ref:2603.05213#pg8>.

Rosa: Model Selection and Training Strategy The study systematically compared multiple self-supervised models that varied in architecture, training set size, language coverage, and whether they were trained on adult-only speech or both child and adult speech<ref:2603.05213#pg5> Among the models pretrained on child-centered recordings, BabyHuBERT achieved the best performance at 46.2 ± 0.15% PER, showing a 6.0% absolute improvement over W2V2 XLSR despite using a base rather than a large architecture and four times less pretraining data<ref:2603.05213#pg8> The authors also explored the impact of input context duration, demonstrating that longer context windows improve performance, with validation PER decreasing from 46.2 ± 0.15% (no context) to 43.5 ± 0.13% (20 seconds)<ref:2603.05213#pg9>.

Title and authors: Rosa: So we’re talking about this paper called "BabAR: from phoneme recognition to developmental measures of young children's speech production." It sounds like they've built something that goes beyond just identifying sounds; it aims to give us actual data on how speech develops at a massive scale.

Dev: Right, it’s about using automatic tools for early speech development, which is always a tough spot because we need these tools to work outside of the lab and in real-world settings, which is where I like to be. This paper tackles that difficulty by focusing on phoneme recognition for young children.

Taro: What I'm curious about is how they handle the sheer volume of data needed for something this ambitious, especially when you’re dealing with different languages and very young infants whose speech is so variable.

Rosa: Well, the paper starts by curating this dataset called TinyVox. They put together over half a million phonetically transcribed child vocalizations in five languages: English, French, Portuguese, German, and Spanish across five hundred sixty children from six months to eight years old <ref:2603.05213#pg2,languages: English, French, Portuguese, German, and Spanish>.

Dev: That’s a huge standardization effort because they took existing data from PhonBank and cleaned it up to create this consistent set of sounds. They even made a specific set of fifty-seven sounds—thirty consonants and twenty-seven vowels—based on the adult phonemic inventories from all five languages.

Taro: I wonder how much that standardization actually helps when you’re trying to recognize speech that’s changing every single day in a child. Is it just about making sure the input format is neat?

Rosa: It’s more than just neatness; they say this multilingual approach, pretraining the system on child-centered daylong recordings, substantially outperforms other models they tested. They found that this way of training gives BabAR better initial representations for child speech than alternatives like W2V2 XLSR.

Dev: That’s a big deal because it shows that focusing the AI training specifically on what children actually say, rather than just adult speech, provides a much stronger foundation for the recognition system. It points toward how much context matters in these learning models.

Title and authors: Taro: So they found this pretraining strategy is key to getting better performance before you even look at the actual phoneme recognition part of the system? Does that mean we can skip some of the heavy work on adult speech data when building these tools?

Rosa: Exactly, and they also looked at how much surrounding audio context helps during fine-tuning. They showed that providing twenty seconds of surrounding audio context during fine-tuning actually improves performance, boosting the accuracy from about forty-six point two percent down to forty-three point five percent.

Dev: Twenty seconds is a concrete number, so that's a measurable improvement in how well the system adapts to things like competing sounds or adult prompts during that training phase. That’s what I look for when I’m designing real-time loops; understanding the latency and context window is critical for stability.

Taro: Okay, so we have better data curation, a better pretraining strategy using BabyHuBERT on multilingual recordings, and a proven benefit from longer audio context during fine-tuning. What’s next for them?

Rosa: Next they move into error analysis to see how the system actually performs on test sets. They look at substitution errors and find that these substitutions tend to stay within the same broad phonetic categories, which suggests the system might be more reliable than just its overall accuracy score would suggest.

Dev: That’s a subtle but important finding for developmental science; if substitutions stay in the same category, it means we can trust those coarse-grained estimates more when we try to map them onto actual speech maturity measures. It moves it beyond just getting a high percentage number.

Taro: So, even though the overall phoneme error rate might be around forty-two point one percent, the way those errors are distributed across sounds is what matters for tracking development over time. What kind of categories were they looking at?

Rosa: They looked at vowel height and consonant manner of articulation. For vowels, they saw that when nasalized vowels are substituted, the most frequent substitute is their oral counterpart, like a nasal vowel substituting an oral one.

Title and authors: Dev: That pattern tells us something about how the model is confusing those specific acoustic features; it's not just making random mistakes but it’s hitting predictable phonetic hurdles.

Taro: And for consonants, they observed a similar structure where substitutions tend to stay within the same manner of articulation, even when changing voicing or place of articulation. It seems the system has a general understanding of how sounds are grouped structurally.

Rosa: So that brings us to the validation part, which is really where this paper gets its final punch. They applied BabAR to a held-out set of forty-four American English-learning children recorded monthly from six months up to eighteen months.

Dev: That longitudinal dataset is what really proves their claim about aligning with literature; they showed that the automatically extracted proportions of canonical vocalizations, which are a widely used measure of speech maturity, fall within the ninety-five percent confidence interval of estimates from language development literature.

Taro: That’s powerful because it shows that this automated pipeline produces group-level canonical proportion estimates that match what we know about American English-learning children. It bridges the gap between raw audio and established developmental theory.

Rosa: So, to wrap up on "BabAR: from phoneme recognition to developmental measures of young children's speech production," they’ve shown that by using multilingual pretraining and longer context windows, they can build a system that not only recognizes phonemes but also gives us statistically sound estimates of speech development across different languages.

Dev: It’s a big step toward automated phoneme-level transcription for young children, opening doors for large-scale screening for speech and language delays in ways that were previously only possible through manual work.

Taro: It opens up cross-linguistic comparisons of phonological development, which is something we really need to explore when looking at how early vocal patterns relate to later language outcomes.

Rosa: Yeah, they’ve shared TinyVox and BabAR with the community, hoping to lower that barrier for large-scale phonetic analysis of child speech and encourage collaboration between speech technology and developmental science.

The paper's summary: Rosa: So, to recap, BabAR is this new system that takes child speech recognition and turns it into actual measures of how speech develops over time using data from five languages and a huge number of children.

Dev: Yeah, it’s moving past just getting a percentage error rate on words or sounds; they’re actually producing estimates for speech maturity that line up with what we already know about kids learning English.

Rosa: It's not just about the numbers, though. They looked at the actual mistakes the AI makes and found that when it messes up, it usually stays within a broad phonetic category.

Dev: That’s interesting because if the errors are clustered in specific ways—like confusing a nasal vowel with an oral one—it suggests we can use those coarse categories for developmental tracking.

Rosa: Exactly. It means this system might be more reliable for big picture analysis than just looking at the overall error score alone. It’s like being able to tell if the system is consistently misidentifying nasal sounds across different recordings, which is a real insight into how the model understands speech structure.

Dev: And they did this by using BabyHuBERT as a starting point and adding twenty seconds of surrounding audio during fine-tuning, which improved performance. So the system gets better contextually before it even tries to identify the phoneme itself.

Rosa: Right, that’s a key part of how they built it up. It shows that leveraging multilingual data and giving the model rich context helps it handle the variability inherent in young children's speech much better than previous attempts.

Dev: The implication here for us is that we can start thinking about how to build these large-scale systems to be more robust against those real-world noises and variations you see when you’re testing outside a controlled lab setting.

Rosa: Definitely. Because this system aims to open up big things, like cross-linguistic comparisons of phonological development, which used to require tedious manual transcription for small samples.

Dev: It really is a step toward automated phoneme-level transcription at scale, which could change how quickly we can screen for speech and language delays in different populations.

Rosa: So it’s not just a tool; it’s a way to get objective data on speech development that was previously too expensive to gather.

Dev: It’s about bridging the gap between raw audio and established developmental science literature, which is what they aimed for with these validation tests on those forty-four children.

Rosa: Next up, we're going to look at how they actually put this system into practice and what kind of real-world failure modes it encounters when things get messy.

The paper's improvements: Rosa: So, to wrap up on improvements, the authors are suggesting that future work should focus on quantifying how much agreement there is between humans and this AI system when they look at the same speech samples.

Dev: That makes sense from a controls standpoint; if we can't quantify how much the machine and a human agree on an error, we can't really know where the system is failing for developmental tracking.

Rosa: Right. They pointed out that phonetic transcription isn't an objective record of what the child actually made, which is a big limitation they acknowledged.

Dev: So they aren't claiming perfect accuracy, which is realistic, but they’re suggesting a way to measure the uncertainty of those errors compared to human annotators.

Rosa: That means future studies need to look at both human-human and human-machine agreement on the same speech recordings so we can establish how much the system errors actually exceed the natural variability among different transcribers.

Dev: That moves it from just a performance metric, like phoneme error rate, to a reliability metric that accounts for human subjectivity. It’s about establishing a confidence interval for what the AI is actually saying.

Rosa: It’s crucial because this is about using the system for scientific tracking, and we need to know when we can trust those automatic estimates over the manual ones.

Dev: I agree. If we can map out that discrepancy between human and machine consensus, it tells us exactly how much noise is in that automatic pipeline.

Rosa: And this opens up a lot of avenues for future research into how these vocal patterns relate to later language outcomes, which used to be locked behind the cost of manual transcription.

Dev: Exactly. It lets us explore those connections at a scale we couldn't reach before, but we have to keep that human-machine agreement in mind when we use the data for clinical markers.

Rosa: So, it’s not just about making the AI better; it’s about building a framework to assess its trustworthiness for real developmental science applications.

Dev: That’s the direction they're pushing toward, moving beyond just high numbers and focusing on how robust those estimates are in practice.

Rosa: And that leads us into looking at how this all fits into the larger ecosystem of speech technology, especially when we consider other papers on feedback delay networks and foundation models.

Conclusion: Tom: So we're wrapping up on "BabAR: from phoneme recognition to developmental measures of young children's speech production." It’s really about using this system to get objective data on how speech develops across different languages.

Rosa: Right, it’s a big step because it lets us move past just guessing at development and start seeing group-level estimates that actually match what the literature suggests for American English-learning children.

Dev: It’s interesting because they showed that by using babyhubert as a base and adding twenty seconds of surrounding audio during fine-tuning, the system gets better contextually before it even tries to identify the phoneme itself.

Taro: I just want to push on what this means for real-world testing; if we can get these estimates right, it’s a huge validation tool for autonomy systems that need to understand human speech in diverse settings.

Rosa: Totally. And they also showed that when the system makes substitution errors, those errors tend to stay within the same broad phonetic categories, which suggests those coarse category estimates are more reliable than just looking at the overall error score.

Dev: That’s a key point because if we can't trust the substitution error counts, then all that accuracy in the phoneme recognition part doesn't matter for our safety case.

Taro: It’s about graceful recovery; if the system is consistently making predictable errors within certain sound groups, it means its failure modes are more structured and easier to design around.

Rosa: So they’ve created a system that not only recognizes phonemes but also provides a measurable way to track speech production development, which used to require expensive manual transcription for small samples.

Dev: It’s about making large-scale phonetic analysis of child speech accessible, and they shared both the dataset TinyVox and the BabAR model with the community for that purpose.

Taro: For me, I think this opens up cross-linguistic comparisons of phonological development, which is something we really need to explore when looking at how early vocal patterns relate to later language outcomes.

Rosa: Exactly. So while this paper is a first step toward automated phoneme-level transcription for young children, it’s laying the groundwork for much bigger studies in speech technology and developmental science.

Dev: We'll keep an eye on how they quantify that human-machine agreement we talked about, because that’s where the real reliability check will be.

Taro: Yeah, I'm looking forward to seeing how this framework helps us design more robust systems for interacting with children in complex environments.

Marvin Lavechin, Elika Bergelson, Roger Levy

Laboratoire d’Informatique et Systemes, Université Aix-Marseille, CNRS · Department of Psychology, Harvard University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology

eess.AS

Submitted: 2026-03-05

Updated: 2026-06-12

Journal ref: Proc. Interspeech 2026 [Long Track], 4354-4363

DOI: 10.21437/Interspeech.2026-1132

Code: https://github.com/MarvinLvn/BabAR

Project page: https://marvinlvn.github.io/projects/babar

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 94/100

The gist: Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children…

Key concepts

TinyVox
This is a large, standardized database of over half a million vocalizations from 560 children aged 6 months to 8 years across five languages (English, French, Portuguese, German, Spanish). It was created by curating data from PhonBank to overcome the barrier of needing costly manual phonetic transcription for child speech.
Self-Supervised Models
These are types of AI models trained on massive amounts of raw audio data without needing human labels. The study compared several models, finding that pretraining on multilingual child-centered recordings significantly outperformed alternatives, leading to the best performance with BabyHuBERT.
Canonical Proportion Estimates
These are group-level estimates derived from BabAR's automatic analysis of speech maturity. The paper validated these estimates by showing they fall within the 95% confidence interval of manual annotations, proving the system produces results consistent with established literature on American English learners.

Terminology

Summary

Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved<ref:2603. p. 0

Where the paper says it

edu, rplevy@mit. edu Abstract Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved.

05213#pg2> Building on decades of data collection, we curate TinyVox, a corpus of more than half a million phonetically transcribed child vocalizations in English, French, Portuguese, German, and Spanish<ref:2603. p. 0
Where the paper says it

edu Abstract Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved. Building on decades of data collection, we curate TinyVox, a corpus of more than half a million phonetically transcribed child vocalizations in English, French, Portuguese, German, and Spanish.

05213#pg2> We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives<ref:2603. p. 0
Where the paper says it

We use TinyVox to train BabAR, a cross-linguistic phoneme recognition system for child speech. We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives, and that providing 20 seconds of surrounding audio context during fine-tuning further improves performance.

05213#pg5> BabAR’s automatic measures of speech maturity align with developmental estimates from the literature<ref:2603. p. 0
Where the paper says it

Error analyses show that substitutions predominantly fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses. We validate BabAR by showing that its automatic measures of speech maturity align with developmental estimates from the literature.

05213#pg10>.

Data Curation and Standardization

The authors address the data barrier by curating TinyVox, a large-scale standardization of PhonBank comprising over half a million IPA-transcribed vocalizations from 560 children aged 6 months to 8 years across five languages<ref:2603.05213#pg3> The dataset was created from PhonBank [21], and all available data in English, French, Portuguese, German, and Spanish were downloaded and audio files were paired with their transcript (CHAT format [22])<ref:2603. p. 1

Where the paper says it

TinyVox (training, validation, and test sets) Our dataset of IPA-transcribed children’s vocalizations was created from PhonBank [21], a large-scale database dedicated to the study of child phonology. All available data in English, French, Portuguese, German, and Spanish were downloaded and audio files were paired with their transcript (CHAT format [22]).

05213#pg5> To create a more consistent cross-linguistic target inventory, they created a 57-sound set (30 consonants and 27 vowels) based on the adult phonemic inventories of all five languages<ref:2603. p. 1
Where the paper says it

However, these categories reflect heterogeneous annotation practices, with varying levels of phonetic detail across corpora. To create a more consistent cross-linguistic target inventory, we created a 57-sound set (30 consonants and 27 vowels) based on the adult phonemic inventories of all five languages.

05213#pg9>. Post-processing involved removing utterances with extreme durations or containing unidentified sounds, and implementing a two-pass sampling procedure to screen for systematic alignment issues in problematic corpora<ref:2603. p. 1
Where the paper says it

We therefore implemented a two-pass sampling procedure. In the first pass, we randomly sampled 10% of files from each corpus and listened to 10 randomly selected utterances per file to identify corpora with systematic alignment issues.

05213#pg8>.

Model Selection and Training Strategy

The study systematically compared multiple self-supervised models that varied in architecture, training set size, language coverage, and whether they were trained on adult-only speech or both child and adult speech<ref:2603.05213#pg5> Among the models pretrained on child-centered recordings, BabyHuBERT achieved the best performance at 46. p. 3

Where the paper says it

BabyHuBERT adapts the HuBERT base architecture and was trained on 13,000 hours of multilingual child-centered daylong recordings containing both child and adult speech [25]. To the best of our knowledge, W2V2 LL4300 and BabyHuBERT are the only two self-supervised models pretrained on child-centered daylong recordings.

2 ± 0.15% PER, showing a 6. p. 0
Where the paper says it

Error analyses show that substitutions predominantly fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses. We validate BabAR by showing that its automatic measures of speech maturity align with developmental estimates from the literature.

0% absolute improvement over W2V2 XLSR despite using a base rather than a large architecture and four times less pretraining data<ref:2603. p. 4
Where the paper says it

15% PER: a 6. 0% absolute improvement over W2V2 XLSR despite using a base rather than a large architecture and four times less pretraining data.

05213#pg8> The authors also explored the impact of input context duration, demonstrating that longer context windows improve performance, with validation PER decreasing from 46. p. 1
Where the paper says it

multilingual), and whether they were trained on adult-only speech or both child and adult speech. Third, we study the impact of input context duration on phoneme recognition accuracy, demonstrating that longer context windows improve performance.

2 ± 0.15% (no context) to 43. p. 0
Where the paper says it

We use TinyVox to train BabAR, a cross-linguistic phoneme recognition system for child speech. We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives, and that providing 20 seconds of surrounding audio context during fine-tuning further improves performance.

5 ± 0.13% (20 seconds)<ref:2603. p. 0
Where the paper says it

In this work, we introduce BabAR (BABbling Automatic Recognition), a phoneme recognition system for child speech that substantially extends prior work in both scope and scale. arXiv:2603.

05213#pg9>.

Error Analysis and Developmental Validation

BabAR's error analysis showed that substitutions predominantly fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses When applied to a held-out set of naturalistic recordings of 44 infants, BabAR’s derived canonical proportion estimates fall within the 95% confidence interval of a large-scale metaanalysis based on manual annotations. This demonstrates that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature. p. 6

Where the paper says it

Both the average automatic and manual trajectories exhibit a clear increasing trend from 6 to 17 months. Crucially, BabAR’s automatic trajectory falls within the 95% confidence interval of the manual trajectory across the entire age range, indicating that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature.

Conclusion and Future Directions

BabAR represents a first step toward automated phoneme-level transcription of young children’s speech at scale, opening new possibilities for large-scale screening for speech and language delays The authors acknowledge that phonetic transcription is not an objective record of what was produced by the child, but a subjective perceptual judgment Future work should quantify both human-human and human-machine agreement on the same speech samples to establish the extent to which system errors exceed the variability present among human transcribers.

--- Page 1 ---

BabAR: from phoneme recognition to developmental measures of young children’s speech production The first years of life mark a period of dramatic transformation, as most children progress from simple sounds and syllables to words and phrases, developing the ability to combine basic sound units into increasingly complex sequences [1]. Decades of research on child speech production have produced a wealth of findings with both theoretical and practical implications regarding how universal the order of learning of speech sounds is [2], how early babbling relates to later word production [3], how speech production is linked to various clinical conditions, and how to remediate speech difficulties [4]. p. 0

Where the paper says it

Introduction The first years of life mark a period of dramatic transformation, as most children progress from simple sounds and syllables to words and phrases, developing the ability to combine basic sound units into increasingly complex sequences [1]. Decades of research on child speech production have produced a wealth of findings with both theoretical and practical implications regarding how universal the order of learning of speech sounds is [2], how early babbling relates to later word production [3], how speech production is linked to various clinical conditions, and how to remediate speech difficulties [4].

Audio recordings collected in controlled or naturalistic settings are the prevailing method for studying early speech development, providing precise data on individual speech sounds and how they change over development [5]. p. 0
Where the paper says it

Decades of research on child speech production have produced a wealth of findings with both theoretical and practical implications regarding how universal the order of learning of speech sounds is [2], how early babbling relates to later word production [3], how speech production is linked to various clinical conditions, and how to remediate speech difficulties [4]. Audio recordings collected in controlled or naturalistic settings are the prevailing method for studying early speech development, providing precise data on individual speech sounds and how they change over development [5].

However, this approach requires costly manual phonetic transcription, which becomes infeasible at scale This annotation burden creates a tradeoff between depth and breadth that impedes scientific progress: longitudinal studies capturing everyday experiences provide extensive data from few children, while more controlled settings enable larger samples at the cost of sparse, brief recordings per participant (Table 1 illustrates this tradeoff across the 31 corpora compiled for this study) Automatic speech recognition (ASR) algorithms, fueled by advances in self-supervised learning, have now reached impressive performance on adult speech, even in noisy recording conditions [6]. p. 0
Where the paper says it

However, this approach requires costly manual phonetic transcription, which becomes infeasible at scale. This annotation burden creates a tradeoff between depth and breadth that impedes scientific progress: longitudinal studies capturing everyday experiences provide extensive data from few children, while more controlled settings enable larger samples at the cost of sparse, brief recordings per **indicates the corresponding author.

Building such systems for child speech could enable detailed phonetic tracking across thousands of hours and diverse populations, in turn accelerating research on speech acquisition and related clinical markers, opening up analyses that were previously impractical at scale. p. 0
Where the paper says it

Automatic speech recognition (ASR) algorithms, fueled by advances in self-supervised learning, have now reached impressive performance on adult speech, even in noisy recording conditions [6]. Building such systems for child speech could enable detailed phonetic tracking across thousands of hours and diverse populations, in turn accelerating research on speech acquisition and related clinical markers, opening up analyses that were previously impractical at scale.

However, child speech, especially from young infants, poses unique challenges. p. 0
Where the paper says it

Building such systems for child speech could enable detailed phonetic tracking across thousands of hours and diverse populations, in turn accelerating research on speech acquisition and related clinical markers, opening up analyses that were previously impractical at scale. However, child speech, especially from young infants, poses unique challenges.

A newborn’s vocal tract differs substantially from its adult shape: the larynx is positioned higher and the tongue fills much of the space of the oral cavity [7, 8]. p. 0
Where the paper says it

However, child speech, especially from young infants, poses unique challenges. A newborn’s vocal tract differs substantially from its adult shape: the larynx is positioned higher and the tongue fills much of the space of the oral cavity [7, 8].

Throughout early childhood, articulatory motor control and perceptual systems are still developing, resulting in highly variable acoustic output that differs markedly from adult speech [1, 9]. p. 0
Where the paper says it

A newborn’s vocal tract differs substantially from its adult shape: the larynx is positioned higher and the tongue fills much of the space of the oral cavity [7, 8]. Throughout early childhood, articulatory motor control and perceptual systems are still developing, resulting in highly variable acoustic output that differs markedly from adult speech [1, 9].

Various techniques have been proposed to address the gap between child and adult speech, including vocal tract length normalization [10], data augmentation/generation [11, 12], multi-task learning [13], and transfer learning using selfsupervised models pretrained on adult speech [14, 15]. p. 0
Where the paper says it

Throughout early childhood, articulatory motor control and perceptual systems are still developing, resulting in highly variable acoustic output that differs markedly from adult speech [1, 9]. Various techniques have been proposed to address the gap between child and adult speech, including vocal tract length normalization [10], data augmentation/generation [11, 12], multi-task learning [13], and transfer learning using selfsupervised models pretrained on adult speech [14, 15].

Most of this work has focused on children older than 6 years, with applications to educational technology, literacy assessment, and language learning tools [16, 17, 18]. p. 0
Where the paper says it

Various techniques have been proposed to address the gap between child and adult speech, including vocal tract length normalization [10], data augmentation/generation [11, 12], multi-task learning [13], and transfer learning using selfsupervised models pretrained on adult speech [14, 15]. Most of this work has focused on children older than 6 years, with applications to educational technology, literacy assessment, and language learning tools [16, 17, 18].

A notable exception, and the closest to our work, is Li et al. p. 0
Where the paper says it

Most of this work has focused on children older than 6 years, with applications to educational technology, literacy assessment, and language learning tools [16, 17, 18]. A notable exception, and the closest to our work, is Li et al.

(2024), who trained a phoneme recognition system on the Providence corpus (including children aged 11 to 48 months) using wav2vec 2. p. 0
Where the paper says it

A notable exception, and the closest to our work, is Li et al. (2024), who trained a phoneme recognition system on the Providence corpus (including children aged 11 to 48 months) using wav2vec 2.

0 pretrained on 4,300 hours of child-directed daylong recordings from American English-speaking families However, they achieve a phoneme error rate of around 60% [19, 20], illustrating that child speech remains a formidable challenge for ASR systems. p. 0
Where the paper says it

0 pretrained on 4,300 hours of child-directed daylong recordings from American English-speaking families. However, they achieve a phoneme error rate of around 60% [19, 20], illustrating that child speech remains a formidable challenge for ASR systems.

The difficulty building high-performing child ASR systems stems largely from the scarcity of publicly available annotated data, especially for non-English languages and young children. p. 0
Where the paper says it

However, they achieve a phoneme error rate of around 60% [19, 20], illustrating that child speech remains a formidable challenge for ASR systems. The difficulty building high-performing child ASR systems stems largely from the scarcity of publicly available annotated data, especially for non-English languages and young children.

Indeed, among 27 child ASR corpora recently reviewed, only 13 include non-English data, and only one (the Providence corpus) includes children younger than 4 years old [17]. p. 0
Where the paper says it

The difficulty building high-performing child ASR systems stems largely from the scarcity of publicly available annotated data, especially for non-English languages and young children. Indeed, among 27 child ASR corpora recently reviewed, only 13 include non-English data, and only one (the Providence corpus) includes children younger than 4 years old [17].

Yet, publicly available repositories such as PhonBank contain decades of child speech recordings, laboriously collected and annotated by generations of researchers [21]. p. 0
Where the paper says it

Indeed, among 27 child ASR corpora recently reviewed, only 13 include non-English data, and only one (the Providence corpus) includes children younger than 4 years old [17]. Yet, publicly available repositories such as PhonBank contain decades of child speech recordings, laboriously collected and annotated by generations of researchers [21].

These resources remain largely untapped by the speech processing community due to the large technical debt required to aggregate disparate sources: reconciling diverse phonetic transcriptions, resolving human inconsistencies, and extracting machineactionable segments from varied audio. p. 0
Where the paper says it

Yet, publicly available repositories such as PhonBank contain decades of child speech recordings, laboriously collected and annotated by generations of researchers [21]. These resources remain largely untapped by the speech processing community due to the large technical debt required to aggregate disparate sources: reconciling diverse phonetic transcriptions, resolving human inconsistencies, and extracting machineactionable segments from varied audio.

In this work, we introduce BabAR (BABbling Automatic Recognition), a phoneme recognition system for child speech that substantially extends prior work in both scope and scale. p. 0
Where the paper says it

These resources remain largely untapped by the speech processing community due to the large technical debt required to aggregate disparate sources: reconciling diverse phonetic transcriptions, resolving human inconsistencies, and extracting machineactionable segments from varied audio. In this work, we introduce BabAR (BABbling Automatic Recognition), a phoneme recognition system for child speech that substantially extends prior work in both scope and scale.

--- Page 2 ---

We make the following contributions: First, we address the data barrier by curating TinyVox, a large-scale standardization of PhonBank comprising over half a million IPA-transcribed vocalizations from 560 children aged 6 months to 8 years across 5 languages (English, French, Portuguese, German, and Spanish) Second, we systematically compare multiple self-supervised models that vary in their architecture, training set size, language coverage (English-only vs. multilingual), and whether they were trained on adult-only speech or both child and adult speech Third, we study the impact of input context duration on phoneme recognition accuracy, demonstrating that longer context windows improve performance Fourth, we provide detailed analyses of our best-performing model Finally, we validate BabAR by applying it to a held-out longitudinal dataset of 44 American English-learning children recorded monthly from 6 to 18 months, and show that automatically extracted proportions of canonical vocalizations (a widely used measure of speech maturity) align with estimates from the language development literature. p. 1

Where the paper says it

Fourth, we provide detailed analyses of our best-performing model. Finally, we validate BabAR by applying it to a held-out longitudinal dataset of 44 American English-learning children recorded monthly from 6 to 18 months, and show that automatically extracted proportions of canonical vocalizations (a widely used measure of speech maturity) align with estimates from the language development literature.

--- Page 3 ---

We next present our approach to recognizing phonemes in children’s speech. We begin with describing the dataset, the phoneme recognition system, and the self-supervised speech representations it builds upon We then present our contextaware fine-tuning approach Finally, we present the baselines against which BabAR is compared and our evaluation metric. p. 1

Where the paper says it

We begin with describing the dataset, the phoneme recognition system, and the self-supervised speech representations it builds upon. We then present our contextaware fine-tuning approach.

We conclude with implementation details. p. 1
Where the paper says it

Finally, we present the baselines against which BabAR is compared and our evaluation metric. We conclude with implementation details.

--- Page 4 ---

We begin with describing the dataset, the phoneme recognition system, and the self-supervised speech representations it builds upon We then present our contextaware fine-tuning approach Finally, we present the baselines against which BabAR is compared and our evaluation metric.

--- Page 5 ---

We evaluate six self-supervised models that vary in their training data characteristics (Table 2) and their architecture We then examine the effect of providing extended audio context during training We then compare the performance of our best-performing system, BabAR, against two baselines before analyzing its error types. Finally, we close the loop from model development to real-world application by using BabAR to measure speech production development on a held-out set of naturalistic child-centered recordings unseen during training, validation, and evaluation. p. 4

Where the paper says it

We then compare the performance of our best-performing system, BabAR, against two baselines, before analyzing its error types. Finally, we close the loop from model development to real-world application by using BabAR to measure speech production development on a held-out set of naturalistic child-centered recordings unseen during training, validation, and evaluation.

p. 4
Where the paper says it

We then compare the performance of our best-performing system, BabAR, against two baselines, before analyzing its error types. Finally, we close the loop from model development to real-world application by using BabAR to measure speech production development on a held-out set of naturalistic child-centered recordings unseen during training, validation, and evaluation.

--- Page 6 ---

We first compare the performance of six self-supervised models fine-tuned on TinyVox, then examine the effect of providing extended audio context during training We then compare the performance of our best-performing system, BabAR, against two baselines before analyzing its error types. Finally, we close the loop from model development to real-world application by using BabAR to measure speech production development on a held-out set of naturalistic child-centered recordings unseen during training, validation, and evaluation.

--- Page 7 ---

We now analyze the errors made by BabAR on the test set, first comparing performance to baselines, then examining how error rates vary across languages, and finally investigating whether substitution errors tend to remain within or fall across broad phonetic categories.

--- Page 8 ---

We now analyze the errors made by BabAR on the test set, first comparing performance to baselines, then examining how error rates vary across languages, and finally investigating whether substitution errors tend to remain within or fall across broad phonetic categories.

--- Page 9 ---

Figure 5 shows the results Both the average automatic and manual trajectories exhibit a clear increasing trend from 6 to 17 months Crucially, BabAR’s automatic trajectory falls within the 95% confidence interval of the manual trajectory across the entire age range, indicating that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature.

--- Page 10 ---

Figure 5 shows the results Both the average automatic and manual trajectories exhibit a clear increasing trend from 6 to 17 months Crucially, BabAR’s automatic trajectory falls within the 95% confidence interval of the manual trajectory across the entire age range, indicating that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature.

--- Page 11 ---

Figure 4: Substitution matrices for vowels (panel a) and consonants (panel b) indicating which substitution errors BabAR makes. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37. p. 5

Where the paper says it

For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37.

5% to /I/, differing from /˜I/ in tenseness or nasalization. p. 5
Where the paper says it

At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37. 5% to /I/, differing from /˜I/ in tenseness or nasalization.

Similarly, when /˜u/ is substituted, 71. p. 5
Where the paper says it

5% to /I/, differing from /˜I/ in tenseness or nasalization. Similarly, when /˜u/ is substituted, 71.

4% of errors map to /u/, both close back rounded vowels differing in nasalization. p. 5
Where the paper says it

Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both close back rounded vowels differing in nasalization.

These examples illustrate a broader pattern: when nasalized vowels are substituted, the most frequent substitute tends to be their oral counterpart (e. p. 5
Where the paper says it

4% of errors map to /u/, both close back rounded vowels differing in nasalization. These examples illustrate a broader pattern: when nasalized vowels are substituted, the most frequent substitute tends to be their oral counterpart (e.

g., /˜I/ → /I/, /˜u/ → /u/, / ˜œ/ → /œ/), suggesting that the nasalization contrast is a primary source of confusion for the model. p. 5
Where the paper says it

These examples illustrate a broader pattern: when nasalized vowels are substituted, the most frequent substitute tends to be their oral counterpart (e. , /˜I/ → /I/, /˜u/ → /u/, / ˜œ/ → /œ/), suggesting that the nasalization contrast is a primary source of confusion for the model.

--- Page 12 ---

At the level of vowel height, we compute for each category the proportion of substitution errors that remain within the same height category (i.e., within the colored blocks in Figure 4a) versus crossing into another. p. 5

Where the paper says it

At the level of vowel height, we compute for each category the proportion of substitution errors that remain within the same height category (i. , within the colored blocks in Figure 4a) versus crossing into another.

For close vowels, 52. p. 5
Where the paper says it

Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both close back rounded vowels differing in nasalization.

2% of substitution errors map to another close vowel, 38. p. 5
Where the paper says it

For close vowels, 52. 2% of substitution errors map to another close vowel, 38.

4% to a mid vowel, and only 9. p. 5
Where the paper says it

2% of substitution errors map to another close vowel, 38. 4% to a mid vowel, and only 9.

3% to an open vowel. p. 5
Where the paper says it

4% to a mid vowel, and only 9. 3% to an open vowel.

Mid vowels show a similar within-category rate (52. p. 5
Where the paper says it

3% to an open vowel. Mid vowels show a similar within-category rate (52.

6%), but with more symmetric spillover toward both close (19. p. 6
Where the paper says it

Substitutions were more likely within vowel/consonant categories (in each outlined square) than across them. spillover toward both close (19.

9%) and open (27. p. 5
Where the paper says it

4% to a mid vowel, and only 9. 3% to an open vowel.

4%) vowels. p. 1
Where the paper says it

However, these categories reflect heterogeneous annotation practices, with varying levels of phonetic detail across corpora. To create a more consistent cross-linguistic target inventory, we created a 57-sound set (30 consonants and 27 vowels) based on the adult phonemic inventories of all five languages.

Open vowels have the lowest within-category rate: only 37. p. 6
Where the paper says it

4%) vowels. Open vowels have the lowest within-category rate: only 37.

9% of substitution errors map to another open vowel, while 54. p. 6
Where the paper says it

Open vowels have the lowest within-category rate: only 37. 9% of substitution errors map to another open vowel, while 54.

p. 6
Where the paper says it

Open vowels have the lowest within-category rate: only 37. 9% of substitution errors map to another open vowel, while 54.

2% map to mid vowels and 7. p. 6
Where the paper says it

9% of substitution errors map to another open vowel, while 54. 2% map to mid vowels and 7.

p. 6
Where the paper says it

9% of substitution errors map to another open vowel, while 54. 2% map to mid vowels and 7.

9% to close vowels. p. 5
Where the paper says it

Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both close back rounded vowels differing in nasalization.

p. 5
Where the paper says it

Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both close back rounded vowels differing in nasalization.

For consonants (Figure 4b), we observe a similar structure. p. 6
Where the paper says it

9% to close vowels. For consonants (Figure 4b), we observe a similar structure.

p. 6
Where the paper says it

9% to close vowels. For consonants (Figure 4b), we observe a similar structure.

At the level of individual sounds, consider /N/: when it is substituted, 46. p. 5
Where the paper says it

For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37.

p. 5
Where the paper says it

For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37.

5% of errors map to /n/, both nasal consonants differing only in place of articulation Similarly, when /t/ is substituted, 27. p. 6
Where the paper says it

5% of errors map to /n/, both nasal consonants differing only in place of articulation. Similarly, when /t/ is substituted, 27.

p. 6
Where the paper says it

5% of errors map to /n/, both nasal consonants differing only in place of articulation. Similarly, when /t/ is substituted, 27.

6% of errors map to /d/ and 17. p. 5
Where the paper says it

For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37.

p. 5
Where the paper says it

For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37.

8% to /k/, both stops different in voicing or place. p. 6
Where the paper says it

6% of errors map to /d/ and 17. 8% to /k/, both stops different in voicing or place.

p. 6
Where the paper says it

6% of errors map to /d/ and 17. 8% to /k/, both stops different in voicing or place.

More broadly, substitution errors tend to stay within the same manner of articulation. p. 6
Where the paper says it

8% to /k/, both stops different in voicing or place. More broadly, substitution errors tend to stay within the same manner of articulation.

p. 6
Where the paper says it

8% to /k/, both stops different in voicing or place. More broadly, substitution errors tend to stay within the same manner of articulation.

--- Page 13 ---

Thus, although BabAR’s test PER of 42.1% reflects frequent phoneme-level errors, substitutions tend to remain within broad phonetic categories, suggesting it may be more reliable than its PER alone implies for downstream analyses at coarser levels of granularity Open vowels have the lowest within-category rate: only 37. p. 6

Where the paper says it

Thus, although BabAR’s test PER of 42. 1% reflects frequent phoneme-level errors, substitutions tend to remain within broad phonetic categories, suggesting it may be more reliable than its PER alone implies for downstream analyses at coarser levels of granularity, such as consonant-to-vowel ratios or manner-class distributions.

9% of substitution errors map to another open vowel, while 54.2% map to mid vowels and 7.9% to close vowels. For consonants (Figure 4b), we observe a similar structure. At the level of individual sounds, consider /N/: when it is substituted, 46.5% of errors map to /n/, both nasal consonants differing only in place of articulation Similarly, when /t/ is substituted, 27.6% of errors map to /d/ and 17.8% to /k/, both stops different in voicing or place. More broadly, substitution errors tend to stay within the same manner of articulation.

--- Page 14 ---

BabAR represents a first step toward automated phoneme-level transcription of young children’s speech at scale, opening new possibilities for large-scale screening for speech and language delays, cross-linguistic comparisons of phonological development, and investigation of how early vocal patterns relate to later language outcomes — analyses that have until now relied on manual transcription, limiting them to small samples.

--- Page 15 ---

We introduced TinyVox, a cross-linguistic corpus compilation of over half a million phonetically transcribed child vocalizations, and BabAR, a phoneme recognition system that substantially outperforms existing baselines on child speech Our experiments revealed that pretraining on multilingual childcentered daylong recordings and providing extended audio context during fine-tuning both improve performance, and that BabAR’s errors tend to preserve broad phonetic categories. By sharing both TinyVox and BabAR with the community, we hope to lower the barrier to large-scale phonetic analysis of child speech and foster tighter collaboration between speech technology and developmental science. p. 7

Where the paper says it

Applied to held-out naturalistic recordings of 44 infants, BabAR readily recovers established developmental trends without any manual annotation. By sharing both TinyVox and BabAR with the community, we hope to lower the barrier to large-scale phonetic analysis of child speech and foster tighter collaboration between speech technology and developmental science.

--- Page 16 ---

M.L. acknowledges funding from The Simons Foundation International (034070-00033). p. 7

Where the paper says it

Acknowledgments M. acknowledges funding from The Simons Foundation International (034070-00033).

E.B. acknowledges funding fro… p. 7
Where the paper says it

Acknowledgments M. acknowledges funding from The Simons Foundation International (034070-00033).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved Pretraining Strategy: Utilize BabyHuBERT as a starting point for all subsequent experiments because BabyHuBERT (13K hours, multilingual, base architecture) achieves the best performance at 46.2 ± 0.15% PER. This leverages pretraining on multilingual child-centered daylong recordings containing both child and adult speech to provide superior initial representations compared to other models.

  2. Extended Context Fine-tuning: Implement a context window of 20 seconds during fine-tuning for all inference tasks, as the paper shows that providing 20 seconds of surrounding audio context during fine-tuning further improves performance. This helps the model adapt to competing sources (adults, other children, or environmental sounds) and leveraging surrounding linguistic context such as adult prompts (e.g., 'say daddy').

  3. Error Analysis for Coarse Metrics: Use the substitution matrix analysis to guide downstream developmental metrics, as it shows that substitutions tend to remain within broad phonetic categories, suggesting this approach is more reliable than its PER alone implies for downstream analyses at coarser levels of granularity.

Abstract

Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved. Building on decades of data collection, we curate TinyVox, a corpus of more than half a million phonetically transcribed child vocalizations in English, French, Portuguese, German, and Spanish. We use TinyVox to train BabAR, a cross-linguistic phoneme recognition system for child speech. We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives, and that providing 20 seconds of surrounding audio context during fine-tuning further improves performance. Error analyses show that substitutions predominantly fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses. We validate BabAR by showing that its automatic measures of speech maturity align with developmental estimates from the literature.

Related papers