BabAR: from phoneme recognition to developmental measures of young children's speech production
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "BabAR: from phoneme recognition to developmental measures of young children's speech production".
Dev: Here's the gist of the paper. Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved<ref:2603.05213#pg2> Building on decades of data collection, we curate TinyVox, a corpus of more than half a million phonetically transcribed child vocalizations in English, French, Portuguese, German, and Spanish<ref:2603.05213#pg2> We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives<ref:2603.05213#pg5> BabAR’s automatic measures of speech maturity align with developmental estimates from the literature<ref:2603.05213#pg10>.
Rosa: Data Curation and Standardization The authors address the data barrier by curating TinyVox, a large-scale standardization of PhonBank comprising over half a million IPA-transcribed vocalizations from 560 children aged 6 months to 8 years across five languages<ref:2603.05213#pg3> The dataset was created from PhonBank
21: , and all available data in English, French, Portuguese, German, and Spanish were downloaded and audio files were paired with their transcript (CHAT format
22: )<ref:2603.05213#pg5> To create a more consistent cross-linguistic target inventory, they created a 57-sound set (30 consonants and 27 vowels) based on the adult phonemic inventories of all five languages<ref:2603.05213#pg9>.
Dev: Post-processing involved removing utterances with extreme durations or containing unidentified sounds, and implementing a two-pass sampling procedure to screen for systematic alignment issues in problematic corpora<ref:2603.05213#pg8>.
Rosa: Model Selection and Training Strategy The study systematically compared multiple self-supervised models that varied in architecture, training set size, language coverage, and whether they were trained on adult-only speech or both child and adult speech<ref:2603.05213#pg5> Among the models pretrained on child-centered recordings, BabyHuBERT achieved the best performance at 46.2 ± 0.15% PER, showing a 6.0% absolute improvement over W2V2 XLSR despite using a base rather than a large architecture and four times less pretraining data<ref:2603.05213#pg8> The authors also explored the impact of input context duration, demonstrating that longer context windows improve performance, with validation PER decreasing from 46.2 ± 0.15% (no context) to 43.5 ± 0.13% (20 seconds)<ref:2603.05213#pg9>.
Title and authors: Rosa: So we’re talking about this paper called "BabAR: from phoneme recognition to developmental measures of young children's speech production." It sounds like they've built something that goes beyond just identifying sounds; it aims to give us actual data on how speech develops at a massive scale.
Dev: Right, it’s about using automatic tools for early speech development, which is always a tough spot because we need these tools to work outside of the lab and in real-world settings, which is where I like to be. This paper tackles that difficulty by focusing on phoneme recognition for young children.
Taro: What I'm curious about is how they handle the sheer volume of data needed for something this ambitious, especially when you’re dealing with different languages and very young infants whose speech is so variable.
Rosa: Well, the paper starts by curating this dataset called TinyVox. They put together over half a million phonetically transcribed child vocalizations in five languages: English, French, Portuguese, German, and Spanish across five hundred sixty children from six months to eight years old <ref:2603.05213#pg2,languages: English, French, Portuguese, German, and Spanish>.
Dev: That’s a huge standardization effort because they took existing data from PhonBank and cleaned it up to create this consistent set of sounds. They even made a specific set of fifty-seven sounds—thirty consonants and twenty-seven vowels—based on the adult phonemic inventories from all five languages.
Taro: I wonder how much that standardization actually helps when you’re trying to recognize speech that’s changing every single day in a child. Is it just about making sure the input format is neat?
Rosa: It’s more than just neatness; they say this multilingual approach, pretraining the system on child-centered daylong recordings, substantially outperforms other models they tested. They found that this way of training gives BabAR better initial representations for child speech than alternatives like W2V2 XLSR.
Dev: That’s a big deal because it shows that focusing the AI training specifically on what children actually say, rather than just adult speech, provides a much stronger foundation for the recognition system. It points toward how much context matters in these learning models.
Title and authors: Taro: So they found this pretraining strategy is key to getting better performance before you even look at the actual phoneme recognition part of the system? Does that mean we can skip some of the heavy work on adult speech data when building these tools?
Rosa: Exactly, and they also looked at how much surrounding audio context helps during fine-tuning. They showed that providing twenty seconds of surrounding audio context during fine-tuning actually improves performance, boosting the accuracy from about forty-six point two percent down to forty-three point five percent.
Dev: Twenty seconds is a concrete number, so that's a measurable improvement in how well the system adapts to things like competing sounds or adult prompts during that training phase. That’s what I look for when I’m designing real-time loops; understanding the latency and context window is critical for stability.
Taro: Okay, so we have better data curation, a better pretraining strategy using BabyHuBERT on multilingual recordings, and a proven benefit from longer audio context during fine-tuning. What’s next for them?
Rosa: Next they move into error analysis to see how the system actually performs on test sets. They look at substitution errors and find that these substitutions tend to stay within the same broad phonetic categories, which suggests the system might be more reliable than just its overall accuracy score would suggest.
Dev: That’s a subtle but important finding for developmental science; if substitutions stay in the same category, it means we can trust those coarse-grained estimates more when we try to map them onto actual speech maturity measures. It moves it beyond just getting a high percentage number.
Taro: So, even though the overall phoneme error rate might be around forty-two point one percent, the way those errors are distributed across sounds is what matters for tracking development over time. What kind of categories were they looking at?
Rosa: They looked at vowel height and consonant manner of articulation. For vowels, they saw that when nasalized vowels are substituted, the most frequent substitute is their oral counterpart, like a nasal vowel substituting an oral one.
Title and authors: Dev: That pattern tells us something about how the model is confusing those specific acoustic features; it's not just making random mistakes but it’s hitting predictable phonetic hurdles.
Taro: And for consonants, they observed a similar structure where substitutions tend to stay within the same manner of articulation, even when changing voicing or place of articulation. It seems the system has a general understanding of how sounds are grouped structurally.
Rosa: So that brings us to the validation part, which is really where this paper gets its final punch. They applied BabAR to a held-out set of forty-four American English-learning children recorded monthly from six months up to eighteen months.
Dev: That longitudinal dataset is what really proves their claim about aligning with literature; they showed that the automatically extracted proportions of canonical vocalizations, which are a widely used measure of speech maturity, fall within the ninety-five percent confidence interval of estimates from language development literature.
Taro: That’s powerful because it shows that this automated pipeline produces group-level canonical proportion estimates that match what we know about American English-learning children. It bridges the gap between raw audio and established developmental theory.
Rosa: So, to wrap up on "BabAR: from phoneme recognition to developmental measures of young children's speech production," they’ve shown that by using multilingual pretraining and longer context windows, they can build a system that not only recognizes phonemes but also gives us statistically sound estimates of speech development across different languages.
Dev: It’s a big step toward automated phoneme-level transcription for young children, opening doors for large-scale screening for speech and language delays in ways that were previously only possible through manual work.
Taro: It opens up cross-linguistic comparisons of phonological development, which is something we really need to explore when looking at how early vocal patterns relate to later language outcomes.
Rosa: Yeah, they’ve shared TinyVox and BabAR with the community, hoping to lower that barrier for large-scale phonetic analysis of child speech and encourage collaboration between speech technology and developmental science.
The paper's summary: Rosa: So, to recap, BabAR is this new system that takes child speech recognition and turns it into actual measures of how speech develops over time using data from five languages and a huge number of children.
Dev: Yeah, it’s moving past just getting a percentage error rate on words or sounds; they’re actually producing estimates for speech maturity that line up with what we already know about kids learning English.
Rosa: It's not just about the numbers, though. They looked at the actual mistakes the AI makes and found that when it messes up, it usually stays within a broad phonetic category.
Dev: That’s interesting because if the errors are clustered in specific ways—like confusing a nasal vowel with an oral one—it suggests we can use those coarse categories for developmental tracking.
Rosa: Exactly. It means this system might be more reliable for big picture analysis than just looking at the overall error score alone. It’s like being able to tell if the system is consistently misidentifying nasal sounds across different recordings, which is a real insight into how the model understands speech structure.
Dev: And they did this by using BabyHuBERT as a starting point and adding twenty seconds of surrounding audio during fine-tuning, which improved performance. So the system gets better contextually before it even tries to identify the phoneme itself.
Rosa: Right, that’s a key part of how they built it up. It shows that leveraging multilingual data and giving the model rich context helps it handle the variability inherent in young children's speech much better than previous attempts.
Dev: The implication here for us is that we can start thinking about how to build these large-scale systems to be more robust against those real-world noises and variations you see when you’re testing outside a controlled lab setting.
Rosa: Definitely. Because this system aims to open up big things, like cross-linguistic comparisons of phonological development, which used to require tedious manual transcription for small samples.
Dev: It really is a step toward automated phoneme-level transcription at scale, which could change how quickly we can screen for speech and language delays in different populations.
Rosa: So it’s not just a tool; it’s a way to get objective data on speech development that was previously too expensive to gather.
Dev: It’s about bridging the gap between raw audio and established developmental science literature, which is what they aimed for with these validation tests on those forty-four children.
Rosa: Next up, we're going to look at how they actually put this system into practice and what kind of real-world failure modes it encounters when things get messy.
The paper's improvements: Rosa: So, to wrap up on improvements, the authors are suggesting that future work should focus on quantifying how much agreement there is between humans and this AI system when they look at the same speech samples.
Dev: That makes sense from a controls standpoint; if we can't quantify how much the machine and a human agree on an error, we can't really know where the system is failing for developmental tracking.
Rosa: Right. They pointed out that phonetic transcription isn't an objective record of what the child actually made, which is a big limitation they acknowledged.
Dev: So they aren't claiming perfect accuracy, which is realistic, but they’re suggesting a way to measure the uncertainty of those errors compared to human annotators.
Rosa: That means future studies need to look at both human-human and human-machine agreement on the same speech recordings so we can establish how much the system errors actually exceed the natural variability among different transcribers.
Dev: That moves it from just a performance metric, like phoneme error rate, to a reliability metric that accounts for human subjectivity. It’s about establishing a confidence interval for what the AI is actually saying.
Rosa: It’s crucial because this is about using the system for scientific tracking, and we need to know when we can trust those automatic estimates over the manual ones.
Dev: I agree. If we can map out that discrepancy between human and machine consensus, it tells us exactly how much noise is in that automatic pipeline.
Rosa: And this opens up a lot of avenues for future research into how these vocal patterns relate to later language outcomes, which used to be locked behind the cost of manual transcription.
Dev: Exactly. It lets us explore those connections at a scale we couldn't reach before, but we have to keep that human-machine agreement in mind when we use the data for clinical markers.
Rosa: So, it’s not just about making the AI better; it’s about building a framework to assess its trustworthiness for real developmental science applications.
Dev: That’s the direction they're pushing toward, moving beyond just high numbers and focusing on how robust those estimates are in practice.
Rosa: And that leads us into looking at how this all fits into the larger ecosystem of speech technology, especially when we consider other papers on feedback delay networks and foundation models.
Conclusion: Tom: So we're wrapping up on "BabAR: from phoneme recognition to developmental measures of young children's speech production." It’s really about using this system to get objective data on how speech develops across different languages.
Rosa: Right, it’s a big step because it lets us move past just guessing at development and start seeing group-level estimates that actually match what the literature suggests for American English-learning children.
Dev: It’s interesting because they showed that by using babyhubert as a base and adding twenty seconds of surrounding audio during fine-tuning, the system gets better contextually before it even tries to identify the phoneme itself.
Taro: I just want to push on what this means for real-world testing; if we can get these estimates right, it’s a huge validation tool for autonomy systems that need to understand human speech in diverse settings.
Rosa: Totally. And they also showed that when the system makes substitution errors, those errors tend to stay within the same broad phonetic categories, which suggests those coarse category estimates are more reliable than just looking at the overall error score.
Dev: That’s a key point because if we can't trust the substitution error counts, then all that accuracy in the phoneme recognition part doesn't matter for our safety case.
Taro: It’s about graceful recovery; if the system is consistently making predictable errors within certain sound groups, it means its failure modes are more structured and easier to design around.
Rosa: So they’ve created a system that not only recognizes phonemes but also provides a measurable way to track speech production development, which used to require expensive manual transcription for small samples.
Dev: It’s about making large-scale phonetic analysis of child speech accessible, and they shared both the dataset TinyVox and the BabAR model with the community for that purpose.
Taro: For me, I think this opens up cross-linguistic comparisons of phonological development, which is something we really need to explore when looking at how early vocal patterns relate to later language outcomes.
Rosa: Exactly. So while this paper is a first step toward automated phoneme-level transcription for young children, it’s laying the groundwork for much bigger studies in speech technology and developmental science.
Dev: We'll keep an eye on how they quantify that human-machine agreement we talked about, because that’s where the real reliability check will be.
Taro: Yeah, I'm looking forward to seeing how this framework helps us design more robust systems for interacting with children in complex environments.
Marvin Lavechin, Elika Bergelson, Roger Levy
Laboratoire d’Informatique et Systemes, Université Aix-Marseille, CNRS · Department of Psychology, Harvard University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
eess.AS
Submitted: 2026-03-05
Updated: 2026-06-12
Journal ref: Proc. Interspeech 2026 [Long Track], 4354-4363
DOI: 10.21437/Interspeech.2026-1132
Code: https://github.com/MarvinLvn/BabAR
Project page: https://marvinlvn.github.io/projects/babar
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 94/100
The gist: Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children…
Key concepts
- TinyVox
- This is a large, standardized database of over half a million vocalizations from 560 children aged 6 months to 8 years across five languages (English, French, Portuguese, German, Spanish). It was created by curating data from PhonBank to overcome the barrier of needing costly manual phonetic transcription for child speech.
- Self-Supervised Models
- These are types of AI models trained on massive amounts of raw audio data without needing human labels. The study compared several models, finding that pretraining on multilingual child-centered recordings significantly outperformed alternatives, leading to the best performance with BabyHuBERT.
- Canonical Proportion Estimates
- These are group-level estimates derived from BabAR's automatic analysis of speech maturity. The paper validated these estimates by showing they fall within the 95% confidence interval of manual annotations, proving the system produces results consistent with established literature on American English learners.
Terminology
Summary
Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved<ref:2603. p. 0 edu, rplevy@mit. edu
Abstract
Studying early speech development at scale requires automatic
tools, yet automatic phoneme recognition, especially for young
children, remains largely unsolved. edu
Abstract
Studying early speech development at scale requires automatic
tools, yet automatic phoneme recognition, especially for young
children, remains largely unsolved. Building on decades of data
collection, we curate TinyVox, a corpus of more than half a
million phonetically transcribed child vocalizations in English,
French, Portuguese, German, and Spanish. We use TinyVox
to train BabAR, a cross-linguistic phoneme recognition system
for child speech. We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives, and that providing 20 seconds of surrounding audio context during fine-tuning further improves performance. Error analyses show that substitutions predominantly
fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses. We validate
BabAR by showing that its automatic measures of speech maturity align with developmental estimates from the literature.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data Curation and Standardization
The authors address the data barrier by curating TinyVox, a large-scale standardization of PhonBank comprising over half a million IPA-transcribed vocalizations from 560 children aged 6 months to 8 years across five languages<ref:2603.05213#pg3> The dataset was created from PhonBank [21], and all available data in English, French, Portuguese, German, and Spanish were downloaded and audio files were paired with their transcript (CHAT format [22])<ref:2603. p. 1 TinyVox (training, validation, and test sets)
Our dataset of IPA-transcribed children’s vocalizations was created from PhonBank [21], a large-scale database dedicated to
the study of child phonology. All available data in English,
French, Portuguese, German, and Spanish were downloaded
and audio files were paired with their transcript (CHAT format [22]). However, these categories reflect heterogeneous annotation practices, with varying
levels of phonetic detail across corpora. To create a more consistent cross-linguistic target inventory, we created a 57-sound
set (30 consonants and 27 vowels) based on the adult phonemic
inventories of all five languages. We therefore implemented
a two-pass sampling procedure. In the first pass, we randomly
sampled 10% of files from each corpus and listened to 10 randomly selected utterances per file to identify corpora with systematic alignment issues.Where the paper says it
Where the paper says it
Where the paper says it
Model Selection and Training Strategy
The study systematically compared multiple self-supervised models that varied in architecture, training set size, language coverage, and whether they were trained on adult-only speech or both child and adult speech<ref:2603.05213#pg5> Among the models pretrained on child-centered recordings, BabyHuBERT achieved the best performance at 46. p. 3 BabyHuBERT adapts the HuBERT
base architecture and was trained on 13,000 hours of multilingual child-centered daylong recordings containing both child
and adult speech [25]. To the best of our knowledge, W2V2
LL4300 and BabyHuBERT are the only two self-supervised
models pretrained on child-centered daylong recordings. Error analyses show that substitutions predominantly
fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses. We validate
BabAR by showing that its automatic measures of speech maturity align with developmental estimates from the literature. 15% PER: a 6. 0% absolute improvement over W2V2 XLSR despite using a base rather than a large
architecture and four times less pretraining data. multilingual), and whether
they were trained on adult-only speech or both child and adult
speech. Third, we study the impact of input context duration
on phoneme recognition accuracy, demonstrating that longer
context windows improve performance. We use TinyVox
to train BabAR, a cross-linguistic phoneme recognition system
for child speech. We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives, and that providing 20 seconds of surrounding audio context during fine-tuning further improves performance. In this work, we introduce BabAR (BABbling Automatic
Recognition), a phoneme recognition system for child speech
that substantially extends prior work in both scope and scale. arXiv:2603.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Error Analysis and Developmental Validation
BabAR's error analysis showed that substitutions predominantly fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses When applied to a held-out set of naturalistic recordings of 44 infants, BabAR’s derived canonical proportion estimates fall within the 95% confidence interval of a large-scale metaanalysis based on manual annotations. This demonstrates that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature. p. 6 Both the average automatic and
manual trajectories exhibit a clear increasing trend from 6 to 17
months. Crucially, BabAR’s automatic trajectory falls within
the 95% confidence interval of the manual trajectory across the
entire age range, indicating that our fully automatic pipeline
produces group-level canonical proportion estimates consistent
with what we know about American English-learning children
from the literature.Where the paper says it
Conclusion and Future Directions
BabAR represents a first step toward automated phoneme-level transcription of young children’s speech at scale, opening new possibilities for large-scale screening for speech and language delays The authors acknowledge that phonetic transcription is not an objective record of what was produced by the child, but a subjective perceptual judgment Future work should quantify both human-human and human-machine agreement on the same speech samples to establish the extent to which system errors exceed the variability present among human transcribers.
--- Page 1 ---
BabAR: from phoneme recognition to developmental measures of young children’s speech production The first years of life mark a period of dramatic transformation, as most children progress from simple sounds and syllables to words and phrases, developing the ability to combine basic sound units into increasingly complex sequences [1]. Decades of research on child speech production have produced a wealth of findings with both theoretical and practical implications regarding how universal the order of learning of speech sounds is [2], how early babbling relates to later word production [3], how speech production is linked to various clinical conditions, and how to remediate speech difficulties [4]. p. 0 Introduction
The first years of life mark a period of dramatic transformation,
as most children progress from simple sounds and syllables
to words and phrases, developing the ability to combine basic
sound units into increasingly complex sequences [1]. Decades
of research on child speech production have produced a wealth
of findings with both theoretical and practical implications regarding how universal the order of learning of speech sounds
is [2], how early babbling relates to later word production [3],
how speech production is linked to various clinical conditions,
and how to remediate speech difficulties [4]. Decades
of research on child speech production have produced a wealth
of findings with both theoretical and practical implications regarding how universal the order of learning of speech sounds
is [2], how early babbling relates to later word production [3],
how speech production is linked to various clinical conditions,
and how to remediate speech difficulties [4]. Audio recordings collected in controlled or naturalistic settings are the prevailing method for studying early speech development, providing precise data on individual speech sounds and
how they change over development [5]. However, this approach
requires costly manual phonetic transcription, which becomes
infeasible at scale. This annotation burden creates a tradeoff between depth and breadth that impedes scientific progress: longitudinal studies capturing everyday experiences provide extensive data from few children, while more controlled settings enable larger samples at the cost of sparse, brief recordings per
**indicates the corresponding author. Automatic speech recognition (ASR) algorithms, fueled by
advances in self-supervised learning, have now reached impressive performance on adult speech, even in noisy recording conditions [6]. Building such systems for child speech could enable detailed phonetic tracking across thousands of hours and
diverse populations, in turn accelerating research on speech acquisition and related clinical markers, opening up analyses that
were previously impractical at scale. Building such systems for child speech could enable detailed phonetic tracking across thousands of hours and
diverse populations, in turn accelerating research on speech acquisition and related clinical markers, opening up analyses that
were previously impractical at scale. However, child speech, especially from young infants,
poses unique challenges. However, child speech, especially from young infants,
poses unique challenges. A newborn’s vocal tract differs substantially from its adult shape: the larynx is positioned higher
and the tongue fills much of the space of the oral cavity [7, 8]. A newborn’s vocal tract differs substantially from its adult shape: the larynx is positioned higher
and the tongue fills much of the space of the oral cavity [7, 8]. Throughout early childhood, articulatory motor control and perceptual systems are still developing, resulting in highly variable
acoustic output that differs markedly from adult speech [1, 9]. Throughout early childhood, articulatory motor control and perceptual systems are still developing, resulting in highly variable
acoustic output that differs markedly from adult speech [1, 9]. Various techniques have been proposed to address the
gap between child and adult speech, including vocal tract
length normalization [10], data augmentation/generation [11,
12], multi-task learning [13], and transfer learning using selfsupervised models pretrained on adult speech [14, 15]. Various techniques have been proposed to address the
gap between child and adult speech, including vocal tract
length normalization [10], data augmentation/generation [11,
12], multi-task learning [13], and transfer learning using selfsupervised models pretrained on adult speech [14, 15]. Most of
this work has focused on children older than 6 years, with applications to educational technology, literacy assessment, and
language learning tools [16, 17, 18]. Most of
this work has focused on children older than 6 years, with applications to educational technology, literacy assessment, and
language learning tools [16, 17, 18]. A notable exception,
and the closest to our work, is Li et al. A notable exception,
and the closest to our work, is Li et al. (2024), who trained
a phoneme recognition system on the Providence corpus (including children aged 11 to 48 months) using wav2vec 2. 0
pretrained on 4,300 hours of child-directed daylong recordings from American English-speaking families. However, they
achieve a phoneme error rate of around 60% [19, 20], illustrating that child speech remains a formidable challenge for
ASR systems. However, they
achieve a phoneme error rate of around 60% [19, 20], illustrating that child speech remains a formidable challenge for
ASR systems. The difficulty building high-performing child
ASR systems stems largely from the scarcity of publicly available annotated data, especially for non-English languages and
young children. The difficulty building high-performing child
ASR systems stems largely from the scarcity of publicly available annotated data, especially for non-English languages and
young children. Indeed, among 27 child ASR corpora recently reviewed, only 13 include non-English data, and only one
(the Providence corpus) includes children younger than 4 years
old [17]. Indeed, among 27 child ASR corpora recently reviewed, only 13 include non-English data, and only one
(the Providence corpus) includes children younger than 4 years
old [17]. Yet, publicly available repositories such as PhonBank
contain decades of child speech recordings, laboriously collected and annotated by generations of researchers [21]. Yet, publicly available repositories such as PhonBank
contain decades of child speech recordings, laboriously collected and annotated by generations of researchers [21]. These
resources remain largely untapped by the speech processing
community due to the large technical debt required to aggregate disparate sources: reconciling diverse phonetic transcriptions, resolving human inconsistencies, and extracting machineactionable segments from varied audio. These
resources remain largely untapped by the speech processing
community due to the large technical debt required to aggregate disparate sources: reconciling diverse phonetic transcriptions, resolving human inconsistencies, and extracting machineactionable segments from varied audio. In this work, we introduce BabAR (BABbling Automatic
Recognition), a phoneme recognition system for child speech
that substantially extends prior work in both scope and scale.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
We make the following contributions: First, we address the data barrier by curating TinyVox, a large-scale standardization of PhonBank comprising over half a million IPA-transcribed vocalizations from 560 children aged 6 months to 8 years across 5 languages (English, French, Portuguese, German, and Spanish) Second, we systematically compare multiple self-supervised models that vary in their architecture, training set size, language coverage (English-only vs. multilingual), and whether they were trained on adult-only speech or both child and adult speech Third, we study the impact of input context duration on phoneme recognition accuracy, demonstrating that longer context windows improve performance Fourth, we provide detailed analyses of our best-performing model Finally, we validate BabAR by applying it to a held-out longitudinal dataset of 44 American English-learning children recorded monthly from 6 to 18 months, and show that automatically extracted proportions of canonical vocalizations (a widely used measure of speech maturity) align with estimates from the language development literature. p. 1 Fourth, we provide detailed analyses of our best-performing model. Finally, we validate BabAR by applying it to a held-out longitudinal dataset of
44 American English-learning children recorded monthly from
6 to 18 months, and show that automatically extracted proportions of canonical vocalizations (a widely used measure of
speech maturity) align with estimates from the language development literature.Where the paper says it
--- Page 3 ---
We next present our approach to recognizing phonemes in children’s speech. We begin with describing the dataset, the phoneme recognition system, and the self-supervised speech representations it builds upon We then present our contextaware fine-tuning approach Finally, we present the baselines against which BabAR is compared and our evaluation metric. p. 1 We begin with describing the dataset, the
phoneme recognition system, and the self-supervised speech
representations it builds upon. We then present our contextaware fine-tuning approach. Finally, we present the baselines
against which BabAR is compared and our evaluation metric. We conclude with implementation details.Where the paper says it
Where the paper says it
--- Page 4 ---
We begin with describing the dataset, the phoneme recognition system, and the self-supervised speech representations it builds upon We then present our contextaware fine-tuning approach Finally, we present the baselines against which BabAR is compared and our evaluation metric.
--- Page 5 ---
We evaluate six self-supervised models that vary in their training data characteristics (Table 2) and their architecture We then examine the effect of providing extended audio context during training We then compare the performance of our best-performing system, BabAR, against two baselines before analyzing its error types. Finally, we close the loop from model development to real-world application by using BabAR to measure speech production development on a held-out set of naturalistic child-centered recordings unseen during training, validation, and evaluation. p. 4 We then compare the performance of our best-performing system, BabAR, against two
baselines, before analyzing its error types. Finally, we close
the loop from model development to real-world application by
using BabAR to measure speech production development on
a held-out set of naturalistic child-centered recordings unseen
during training, validation, and evaluation. We then compare the performance of our best-performing system, BabAR, against two
baselines, before analyzing its error types. Finally, we close
the loop from model development to real-world application by
using BabAR to measure speech production development on
a held-out set of naturalistic child-centered recordings unseen
during training, validation, and evaluation.Where the paper says it
Where the paper says it
--- Page 6 ---
We first compare the performance of six self-supervised models fine-tuned on TinyVox, then examine the effect of providing extended audio context during training We then compare the performance of our best-performing system, BabAR, against two baselines before analyzing its error types. Finally, we close the loop from model development to real-world application by using BabAR to measure speech production development on a held-out set of naturalistic child-centered recordings unseen during training, validation, and evaluation.
--- Page 7 ---
We now analyze the errors made by BabAR on the test set, first comparing performance to baselines, then examining how error rates vary across languages, and finally investigating whether substitution errors tend to remain within or fall across broad phonetic categories.
--- Page 8 ---
We now analyze the errors made by BabAR on the test set, first comparing performance to baselines, then examining how error rates vary across languages, and finally investigating whether substitution errors tend to remain within or fall across broad phonetic categories.
--- Page 9 ---
Figure 5 shows the results Both the average automatic and manual trajectories exhibit a clear increasing trend from 6 to 17 months Crucially, BabAR’s automatic trajectory falls within the 95% confidence interval of the manual trajectory across the entire age range, indicating that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature.
--- Page 10 ---
Figure 5 shows the results Both the average automatic and manual trajectories exhibit a clear increasing trend from 6 to 17 months Crucially, BabAR’s automatic trajectory falls within the 95% confidence interval of the manual trajectory across the entire age range, indicating that our fully automatic pipeline produces group-level canonical proportion estimates consistent with what we know about American English-learning children from the literature.
--- Page 11 ---
Figure 4: Substitution matrices for vowels (panel a) and consonants (panel b) indicating which substitution errors BabAR makes. At the level of individual sounds, consider /˜I/: when substituted, 40% of errors map to /i/ and 37. p. 5 For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds,
consider /˜I/: when substituted, 40% of errors map to /i/ and
37. At the level of individual sounds,
consider /˜I/: when substituted, 40% of errors map to /i/ and
37. 5% to /I/, differing from /˜I/ in tenseness or nasalization. 5% to /I/, differing from /˜I/ in tenseness or nasalization. Similarly, when /˜u/ is substituted, 71. Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both
close back rounded vowels differing in nasalization. 4% of errors map to /u/, both
close back rounded vowels differing in nasalization. These examples illustrate a broader pattern: when nasalized vowels are
substituted, the most frequent substitute tends to be their oral
counterpart (e. These examples illustrate a broader pattern: when nasalized vowels are
substituted, the most frequent substitute tends to be their oral
counterpart (e. , /˜I/ → /I/, /˜u/ → /u/, / ˜œ/ → /œ/), suggesting
that the nasalization contrast is a primary source of confusion
for the model.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 12 ---
At the level of vowel height, we compute for each category the proportion of substitution errors that remain within the same height category (i.e., within the colored blocks in Figure 4a) versus crossing into another. p. 5 At the level of vowel height, we compute for each
category the proportion of substitution errors that remain within
the same height category (i. , within the colored blocks in Figure 4a) versus crossing into another. Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both
close back rounded vowels differing in nasalization. For close vowels, 52. 2% of
substitution errors map to another close vowel, 38. 2% of
substitution errors map to another close vowel, 38. 4% to a mid
vowel, and only 9. 4% to a mid
vowel, and only 9. 3% to an open vowel. 3% to an open vowel. Mid vowels show a
similar within-category rate (52. Substitutions were more likely
within vowel/consonant categories (in each outlined square) than across them. spillover toward both close (19. 4% to a mid
vowel, and only 9. 3% to an open vowel. However, these categories reflect heterogeneous annotation practices, with varying
levels of phonetic detail across corpora. To create a more consistent cross-linguistic target inventory, we created a 57-sound
set (30 consonants and 27 vowels) based on the adult phonemic
inventories of all five languages. 4%) vowels. Open vowels have the lowest within-category rate: only 37. Open vowels have the lowest within-category rate: only 37. 9%
of substitution errors map to another open vowel, while 54. Open vowels have the lowest within-category rate: only 37. 9%
of substitution errors map to another open vowel, while 54. 9%
of substitution errors map to another open vowel, while 54. 2%
map to mid vowels and 7. 9%
of substitution errors map to another open vowel, while 54. 2%
map to mid vowels and 7. Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both
close back rounded vowels differing in nasalization. Similarly, when /˜u/ is substituted, 71. 4% of errors map to /u/, both
close back rounded vowels differing in nasalization. 9% to close vowels. For consonants (Figure 4b), we observe a similar structure. 9% to close vowels. For consonants (Figure 4b), we observe a similar structure. For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds,
consider /˜I/: when substituted, 40% of errors map to /i/ and
37. For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds,
consider /˜I/: when substituted, 40% of errors map to /i/ and
37. 5% of errors map to /n/, both nasal consonants differing only in place of articulation. Similarly, when /t/ is substituted, 27. 5% of errors map to /n/, both nasal consonants differing only in place of articulation. Similarly, when /t/ is substituted, 27. For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds,
consider /˜I/: when substituted, 40% of errors map to /i/ and
37. For vowels (Figure 4a), we highlight several patterns at increasing levels of granularity. At the level of individual sounds,
consider /˜I/: when substituted, 40% of errors map to /i/ and
37. 6% of errors map to /d/ and 17. 8% to /k/, both stops
different in voicing or place. 6% of errors map to /d/ and 17. 8% to /k/, both stops
different in voicing or place. 8% to /k/, both stops
different in voicing or place. More broadly, substitution errors
tend to stay within the same manner of articulation. 8% to /k/, both stops
different in voicing or place. More broadly, substitution errors
tend to stay within the same manner of articulation.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 13 ---
Thus, although BabAR’s test PER of 42.1% reflects frequent phoneme-level errors, substitutions tend to remain within broad phonetic categories, suggesting it may be more reliable than its PER alone implies for downstream analyses at coarser levels of granularity Open vowels have the lowest within-category rate: only 37. p. 6 Thus, although BabAR’s test PER of 42. 1% reflects frequent phoneme-level errors, substitutions tend to remain within
broad phonetic categories, suggesting it may be more reliable than its PER alone implies for downstream analyses at
coarser levels of granularity, such as consonant-to-vowel ratios
or manner-class distributions.Where the paper says it
--- Page 14 ---
BabAR represents a first step toward automated phoneme-level transcription of young children’s speech at scale, opening new possibilities for large-scale screening for speech and language delays, cross-linguistic comparisons of phonological development, and investigation of how early vocal patterns relate to later language outcomes — analyses that have until now relied on manual transcription, limiting them to small samples.
--- Page 15 ---
We introduced TinyVox, a cross-linguistic corpus compilation of over half a million phonetically transcribed child vocalizations, and BabAR, a phoneme recognition system that substantially outperforms existing baselines on child speech Our experiments revealed that pretraining on multilingual childcentered daylong recordings and providing extended audio context during fine-tuning both improve performance, and that BabAR’s errors tend to preserve broad phonetic categories. By sharing both TinyVox and BabAR with the community, we hope to lower the barrier to large-scale phonetic analysis of child speech and foster tighter collaboration between speech technology and developmental science. p. 7 Applied to held-out naturalistic recordings of 44 infants, BabAR
readily recovers established developmental trends without any
manual annotation. By sharing both TinyVox and BabAR with
the community, we hope to lower the barrier to large-scale phonetic analysis of child speech and foster tighter collaboration
between speech technology and developmental science.Where the paper says it
--- Page 16 ---
M.L. acknowledges funding from The Simons Foundation International (034070-00033). p. 7 Acknowledgments
M. acknowledges funding from The Simons Foundation International (034070-00033). Acknowledgments
M. acknowledges funding from The Simons Foundation International (034070-00033).Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved Pretraining Strategy: Utilize BabyHuBERT as a starting point for all subsequent experiments because
BabyHuBERT (13K hours, multilingual, base architecture) achieves the best performance at 46.2 ± 0.15% PER.
This leverages pretraining onmultilingual child-centered daylong recordings containing both child and adult speech
to provide superior initial representations compared to other models. -
Extended Context Fine-tuning: Implement a context window of 20 seconds during fine-tuning for all inference tasks, as the paper shows that
providing 20 seconds of surrounding audio context during fine-tuning further improves performance.
This helps the model adapt tocompeting sources (adults, other children, or environmental sounds)
andleveraging surrounding linguistic context such as adult prompts (e.g., 'say daddy').
-
Error Analysis for Coarse Metrics: Use the substitution matrix analysis to guide downstream developmental metrics, as it shows that
substitutions tend to remain within broad phonetic categories,
suggesting this approach ismore reliable than its PER alone implies for downstream analyses at coarser levels of granularity.
Abstract
Studying early speech development at scale requires automatic tools, yet automatic phoneme recognition, especially for young children, remains largely unsolved. Building on decades of data collection, we curate TinyVox, a corpus of more than half a million phonetically transcribed child vocalizations in English, French, Portuguese, German, and Spanish. We use TinyVox to train BabAR, a cross-linguistic phoneme recognition system for child speech. We find that pretraining the system on multilingual child-centered daylong recordings substantially outperforms alternatives, and that providing 20 seconds of surrounding audio context during fine-tuning further improves performance. Error analyses show that substitutions predominantly fall within the same broad phonetic categories, suggesting suitability for coarse-grained developmental analyses. We validate BabAR by showing that its automatic measures of speech maturity align with developmental estimates from the literature.
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models