BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings

arXiv:2509.15001 · eess.AS, cs.LG, cs.SD · Submitted 2025-09-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings".

Jane: The gist BabyHuBERT introduces a self-supervised speech model pre-trained on 13,000 hours of multilingual child-centered recordings from over 40 languages to achieve substantial performance improvements in voice type classification compared…

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So to wrap up, we're talking about "BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings". The authors are Theo Charlot, Tarek Kunze, Maxime Poli, Alejandrina Cristia, Emmanuel Dupoux and Marvin Lavechin.

Tom: The main point of this paper is introducing BabyHuBERT to address the poor performance of existing speech models on child recordings because they were trained only on clean adult data. It claims this multilingual model achieves F1-scores from fifty-five point zero percent to seventy-six point one percent across six corpora when classifying speakers into key child, other children, male adult, and female adult categories.

Lu: What matters is that it consistently outperforms W2V2-LL4300 and HuBERT which was pretrained on English daylongs and clean adult speech. The notable gains are fourteen point zero and eighteen point three absolute F1 points over HuBERT on Vanuatu and Solomon Islands.

Meng: From a deployment perspective, the model seems capable of handling the complexity of naturalistic recordings, but they flag that increasing the size and diversity in their pre-training dataset is where they think you get the biggest performance lifts for classes like other children and male adults.

Jane: So what this means for us listening to this show is that we have a new self-supervised speech model available that can help automate the identification of who speaks when in these complex recordings, moving past anonymous speaker labels to meaningful developmental categories.

Conclusion: Tom: So we've been digging into BabyHuBERT, this multilingual self-supervised speech model from those authors—Charlot and his team—and now we’re getting to the end of it all.

Jane: Right, so to recap what we just covered, they took a massive collection of child recordings from over forty languages and trained one single model on it.

Tom: Exactly. They're calling it BabyHuBERT because the whole point is that existing models fail on this kind of audio because the sounds, the voices, and the language are just too different from what they were trained on—which was mostly clean adult speech.

Jane: And what did they find? They showed that this model can actually do voice type classification—figuring out if it’s a key child or an adult male or female—with F1 scores hitting seventy-six point one percent on average across all the different language groups they tested.

Tom: That's a big jump from where models like HuBERT were performing, which was hovering around fifty point seven percent on the same tasks. It sounds like their data approach really helped bridge that gap.

Jane: It really does. The implication here is that we can start building tools for language development research without needing massive, perfectly clean datasets for every single language we care about.

Tom: From a machine learning standpoint, it's impressive how they managed to handle the noise in those daylong recordings by using HuBERT as their base architecture and then doing those two iterations of pseudo-label generation. It’s a smart way to use what we already know about speech representations without needing perfect human labels upfront.

Jane: I think that makes it accessible for people who aren't deep into the math to actually get some useful analysis done on these recordings. It moves the problem from requiring expert phoneticians to something that just needs this robust representation learning approach.

Tom: Right, so they’ve basically built a tool that starts solving a huge logistical bottleneck in how we study early language acquisition across different cultures and languages. But what does this mean for the actual research?

Jane: Well, it means researchers can finally look at those complex daylong recordings and get some automated, relatively accurate insights into who is speaking when. It opens up new avenues for studying how children interact with each other in real-world settings.

Tom: Yeah, it definitely shifts the focus from just labeling every single utterance to actually analyzing patterns across the whole recording. We need to see if this can scale up beyond just voice type classification, though.

Jane: That's what we’ll be looking at next—where they suggest it might take us for other kinds of language tasks in these recordings.

Theo Charlot, Tarek Kunze, Maxime Poli, Alejandrina Cristia, Emmanuel Dupoux, Marvin Lavechin

Laboratoire d’Informatique et Systemes, Université Aix-Marseille, CNRS, PSL University

eess.AS, cs.LG, cs.SD

Submitted: 2025-09-18

Updated: 2026-06-29

Comments: 6 pages, 1 figure

DOI: 10.21437/Interspeech.2026-2772

Code: https://github.com/LAAC-LSCP/VTC

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 83/100

The gist: The gist BabyHuBERT introduces a self-supervised speech model pre-trained on 13,000 hours of multilingual child-centered recordings from over 40 languages to achieve substantial performance

Key concepts

Child-Centered Daylong Recordings: These are naturalistic audio recordings captured by devices worn by children throughout their daily lives. They present major challenges for speech models due to high non-speech content (like noise) and fragmented, non-standard speech patterns from young speakers.
Self-Supervised Speech Model: This is a type of AI model trained on massive amounts of raw audio data without needing human labels initially. BabyHuBERT uses this technique to learn robust speech representations by predicting masked parts of the audio, allowing it to adapt better to diverse languages and child voices.
Voice Type Classification (VTC): This is the specific task the model performs: identifying who is speaking and when in a recording. The model classifies speech into categories like 'key child,' 'other children,' 'male adult,' or 'female adult' based on acoustic features learned during training.
Pre-training Strategy (BabyHuBERT-1 & -2): The model learns in two stages. First, it generates pseudo-labels using K-means clustering on speech features. Second, it refines these representations by clustering features from the first iteration's transformer layer using standard HuBERT methods to improve accuracy.

Terminology

Self-Supervised Learning — 1× in this paper · explained in 8 episodes

Summary

The gist BabyHuBERT introduces a self-supervised speech model pre-trained on 13,000 hours of multilingual child-centered recordings from over 40 languages to achieve substantial performance improvements in voice type classification compared to existing models.

Introduction and Motivation

Child-centered daylong recordings are essential for studying early language development, but existing speech models trained on clean adult data perform poorly due to acoustic and linguistic differences Child-centered daylong recordings, naturalistic audio captured by devices worn by children throughout their daily lives, present a perfect storm of challenges that systematically break existing models. These challenges include 80% non-speech content (silence, noise, or environmental sounds), alongside fragmented speech characterized by short vocalizations and overlapping speakers. Furthermore, the children themselves produce speech with higher fundamental frequencies and non-standard pronunciations that confound adult-trained systems. This technological bottleneck has prevented automated analysis at the scale needed for developmental research, particularly affecting underrepresented languages. BabyHuBERT is introduced as a self-supervised speech model trained on 13,000 hours of multilingual childcentered recordings from 40+ languages to address this mismatch.

Model Architecture and Pre-training Strategy

The approach fundamentally differs from existing speech representation models in both training data and scale. The pre-training set comprises 19 diverse corpora spanning multiple continents across 40+ languages, capturing speech from the child wearing the recording device, other children, and adults present in the environment. To handle the high proportion of non-speech content (around 80% raw audio), PyanNet-VTC is used to detect and extract speech segments. Segments shorter than 2 seconds are padded with surrounding audio, and consecutive segments separated by less than 2 seconds of silence are merged into a single chunk capped at 30 seconds to provide sufficient context. The model leverages HuBERT to leverage its masked prediction approach, considering the base architecture due to computational constraints. The pre-training follows a two-iteration approach; for BabyHuBERT-1, discrete pseudo-labels are generated via K-means clustering of speech features extracted from WavLM-base-plus2, and for BabyHuBERT-2, features from BabyHuBERT-1’s transformer layer are clustered following standard HuBERT methodology. Both iterations are trained for approximately 30 hours each, completing 45 and 44 epochs, respectively.

Fine-tuning and Evaluation

To fine-tune on the voice type classification task, independent binary classification heads are added for each speaker type following [20]. This design allows multiple labels to be activated simultaneously and enables each head to specialize on its respective speaker category. The models are evaluated using the F1-score, the harmonic mean between precision and recall as implemented in pyannote.metrics. Baselines include systems specifically designed for VTC, such as LENATM [7], PyanNet-VTC [18], and Whisper-VTC [20]. General SSL models fine-tuned on BabyTrain-2025 are also included, such as HuBERT pre-trained on clean adult speech [47] and W2V2-LL4300 pre-trained on English daylong recordings [10]. The best model selected is BabyHuBERT-VTC, which achieves an average F1-score of 66.9% across six corpora.

Key Results and Discussion

Table 2 reports the hold-out set performance for all fine-tuned models, showing that HuBERT base and HuBERT large achieve the lowest performance (average F1-score of 50.7±1.7% and 49.1±3.6%, respectively). The substantial performance gain observed with W2V2-LL4300 (+7.7% average F1 over HuBERT base) likely stems from its pre-training on child-centered recordings collected in English-speaking families. Both BabyHuBERT iterations substantially outperform W2V2-LL4300 (63.5±1.6% and 64.2±1.0% vs. 58.4±1.1%). BabyHuBERT-VTC achieves an average F1-score of 66.9%, representing 13.3 and 16.0 absolute percentage point improvements over Whisper-VTC and PyanNet-VTC, respectively. The model shows notable progress on the challenging OCH class, achieving 56.1% F1 compared to much lower performance from existing systems. Comparison with a second human annotator reveals that BabyHuBERT approaches human-level performance, achieving 66.9% average F1-score compared to the second annotator’s 69.8%, representing a gap of only 2.9 percentage points. Performance varies significantly across datasets, with English achieving a midrange score despite being the most represented language in our training data (57%) and the only one in W2V2-LL4300’s pretraining set.

Future Directions and Ethics

Future work suggests that increasing size and/or diversity in the pre-training dataset yields the largest gains for underrepresented classes like other children and male adults. The 56.1% F1-score on other children’s speech is anticipated to enable new research on peer interaction. Due to computational cost, systematic optimization of hyperparameters during pre-training remains an area for improvement. Regarding ethical considerations, BabyHuBERT is released under a custom license informed by an independent ethics assessment covering risks such as participants’ consent and indigenous data sovereignty. The license prohibits commercial use and surveillance while ensuring any derivative model inherits the same conditions. Beyond voice type classification, the authors expect BabyHuBERT to benefit a broad range of downstream tasks in child language research, including addressee classification or phoneme recognition for underrepresented languages where adult-trained models fall short.

Acknowledgments

This work was performed using HPC resources from GENCIIDRIS (2024-AD01101545 and 2025-AD011016414) and was supported in part by the Agence Nationale pour la Recherche (ANR-17-EURE-0017 Frontcog, ANR10-IDEX0001-02 PSL. ED, TC, AC, and TK acknowledge funding from the European Research Council (InfantSimulator101142705 and ExELang-10195). ML acknowledges funding from Simons Foundation International (034070-00033) and MP from Agence de l’Innovation de Defense.

References

[1] P. Garc´ıa, J. Villalba, H. Bredin, J. Du, D. Castan, A. Cristia, L. Bullock, L. Guo, K. Okabe, P. S. Nidadavolu, S Kataria, S Chen.

[2] A Cristia and J Coffey “Long-form recordings to study children’s language input and output in under-resourced contexts” in The Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING, 2024.

[3] L Peurey, M Lavechin T Kunze M Khentout L Gautheron E Dupoux and A Cristia “Fifteen Years of Child-Centered Long-Form Recordings: Promises, Resources, and Remaining Challenges to Validity” in Interspeech, 2025.

[4] S Lee A Potamianos and S Narayanan “Acoustics of children’s speech: Developmental changes of temporal and spectral parameters” The Journal of the Acoustical Society of America, vol. 105, no. 3, pp. 1455–1468, 1999.

[7] D Xu U H Yapanel S S Gray J Gilkerson J A Richards and J H Hansen “Signal processing for young child speech language development” in Workshop on Child Computer Interaction, 2008, p. 20.

[10] J Li M Hasegawa-Johnson and N L McElwain “Towards robust family-infant audio analysis based on unsupervised pretraining of wav2vec 2.0 on large-scale unlabeled family audio” in Interspeech, 2023, pp. 1035–1039.

[11] A Baevski Y Zhou A Mohamed and M Auli “wav2vec 2.0: A framework for self-supervised learning of speech representations” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12449–12460.

Improvements for AI systems

Bold header: Improved representation learning for child-centered speech

This system can achieve substantial performance improvements on voice type classification by leveraging domain-specific pre-training on child-centered speech data, enabling it to better discriminate between speaker categories like key child (KCHI), other children (OCH), male adult (MAL), and female adult (FEM).

Bold header: Enhanced multilingual generalization

BabyHuBERT can be deployed across 40+ languages by utilizing a pre-training set comprising 19 diverse corpora spanning multiple continents, allowing it to perform effectively even on underrepresented languages including Yeli Dnye, Tsimane, and Quechua.

Bold header: Substantial performance gains over state-of-the-art

The improved system can achieve an F1-score of 66.9% average F1-score, representing a 13.3 and 16.0 absolute percentage point improvements over Whisper-VTC and PyanNet-VTC, respectively.

Bold header: Superior discrimination for challenging speaker classes

The model shows notable progress on the challenging OCH class, achieving 56.1% F1 compared to much lower performance from existing systems, which is crucial for research into sibling speech for development.

Abstract

Child-centered daylong recordings are essential for studying early language development, but existing speech models trained on clean adult data perform poorly due to acoustic and linguistic differences. We introduce BabyHuBERT, a self-supervised speech model trained on 13,000 hours of multilingual child-centered recordings from 40+ languages. Evaluated on voice type classification, the task of identifying who produces speech and when in child-centered recordings (key child, other children, male, and female adults), BabyHuBERT-VTC achieves F1-scores from 55.0% to 76.1% across six corpora, consistently outperforming W2V2-LL4300 and HuBERT (pretrained on English daylongs and clean adult speech, respectively). Notable gains include 14.0 and 18.3 absolute F1 points over HuBERT on Vanuatu and Solomon Islands, demonstrating effectiveness on underrepresented languages. We share code and models to support researchers working with child-centered recordings across diverse linguistic contexts.