An open-source voice type classifier for child-centered daylong recordings
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "An open-source voice type classifier for child-centered daylong recordings".
Dev: Here's the gist of the paper. Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.
Rosa: This paper presents an open-source neural network designed to classify audio segments into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male speech, and adult female speech.
Dev: The model architecture combines SincNet filters with a stack of recurrent layers and outperforms the state-of-the-art system LENA.
Rosa: The investigation of language phenomenon in daylong recordings can lead to biased observations.
Title and authors: Rosa: Moving on to what the paper actually lays out in terms of results—the summary section says they trained this model using multiple child-centered corpora to test it against LENA. They focused heavily on comparing their approach to how LENA performs across these diverse settings.
Dev: The paper mentions they used datasets like BabyTrain, which covers a wide range of conditions including different languages and recording devices, and they kept ACLEW-Random aside as a hold-out set for that final comparison.
Taro: So the implication there is that they're testing robustness by seeing how well this system holds up when you move from one specific recording setup to another, which is important for real-world deployment.
Rosa: Right, and they also describe their data augmentation process where they add noise extracted from the MUSAN database with a random signal to noise ratio between five and twenty decibels directly onto the waveform. That helps make the model more generalizable.
The paper's summary: Dev: Now let's talk about what they suggest improving, because they point out that multitask approaches were significantly better than binary ones in their findings. They suggest training a single model to handle all those voice types at once instead of training five separate binary models.
Rosa: That makes sense from a learning perspective; sharing weights during training helps the model learn better boundaries between those different classes simultaneously, which should lead to a more robust overall system than running multiple independent classifiers.
Taro: If you do that multitask approach, they mention that it actually increases the performance for the less represented classes like OCH and MAL compared to binary models, which is a big deal for capturing those specific vocalizations.
Dev: They also highlight that by using this multitask method, their model beats LENA by a large margin, especially when looking at performance on the KCHI, FEM, and SPEECH classes.
The paper's improvements: Rosa: So to wrap up the whole paper "An open-source voice type classifier for child-centered daylong recordings," it seems like their main takeaway is that a multitask approach using this specific neural network architecture significantly outperforms the binary methods they were compared against, especially when dealing with those spontaneous conversations.
Dev: The implication for researchers is that they can now have access to a model that handles this wide variety of conditions and languages better than what was available in the previous closed-source LENA system.
Taro: From my side, I think the biggest change is moving away from just one label per time step to a multi-label task where multiple classes can be active at once, which helps detect overlapping speech instances more accurately than what LENA does.
Conclusion: Rosa: So we've covered the title, the summary, and those specific improvements mentioned in "An open-source voice type classifier for child-centered daylong recordings." It really shows how using a multitask approach with this SincNet LSTM structure can give you much better results when classifying those complex vocalizations.
Dev: The paper concludes that their multitask approach leads to more accurate observations of early linguistic environments, which is something that could help researchers who are trying to measure the quantity and quality of speech children hear.
Taro: And for us in terms of future work, they point out a limitation: since LENA is closed source, you can't really build upon their models to improve them directly without their cooperation, which limits how much further that specific pipeline can go.
Rosa: That’s a fair point; it shows the current challenge in moving forward with these types of audio processing tools when you rely on existing software pipelines.
Dev: Well, that covers what this paper has to offer us regarding voice classification, and we're all ready for whatever new audio challenges come up next.
Marvin Lavechin, Ruben Bousbib, Herve Bredin, Emmanuel Dupoux, Alejandrina Cristia
ENS-PSL/CNRS/EHESS · INRIA Paris, France · LIMSI, CNRS, University of Paris-Sud, Universite Paris-Saclay
eess.AS
Submitted: 2020-05-26
Updated: 2021-01-22
Comments: accepted to Interspeech 2020
Journal ref: Proc. Interspeech 2020, 3072-3076
DOI: 10.21437/Interspeech.2020-1690
Code: https://github.com/MarvinLvn/voice-type-classifier
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.
Key concepts
- Voice Type Classification
- This is the task of identifying the source of a voice signal within an audio stream. Instead of just saying 'speech,' this model distinguishes between different types of sounds, such as a child's specific vocalization versus adult male or female speech.
- SincNet Filters
- These are specialized filters used in the network architecture designed specifically for learning meaningful filter banks. They help the model extract low-level signal representations from the raw audio data in a way that is optimized for voice type identification.
- Multitask Approach
- Instead of training five separate models (one for each voice type), this method trains a single, unified model to handle all five classes simultaneously. This sharing of knowledge between tasks results in a more robust and better-performing final system.
Terminology
Summary
Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process. p. 0 com, alecristia@gmail. com
Abstract
Spontaneous conversations in real-world settings such as those
found in child-centered recordings have been shown to be
amongst the most challenging audio files to process. Nevertheless, building speech processing models handling such a wide
variety of conditions would be particularly useful for language
acquisition studies in which researchers are interested in the
quantity and quality of the speech that children hear and produce, as well as for early diagnosis and measuring effects of
remediation. In this paper, we present our approach to designing an open-source neural network to classify audio segments
into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male
speech, and adult female speech. Our model can be
used as input for downstream tasks such as estimating the number of words produced by adult speakers, or the number of linguistic units produced by children. Our architecture combines
SincNet filters with a stack of recurrent layers and outperforms
by a large margin the state-of-the-art system, the Language ENvironment Analysis (LENA) that has been used in numerous
child language studies.Where the paper says it
Where the paper says it
Where the paper says it
Introduction to the Problem
The investigation of language phenomenon in daylong recordings can lead to biased observations. Technology now allows researchers to efficiently collect and analyze recordings over a whole day. p. 0 However, investigating the language phenomenon in this manner can lead to biased observations, potentially resulting in divergent conclusions [3]. More recently, technology has allowed
researchers to efficiently collect and analyze recordings over a
whole day. By the combined use of a small wearable device
and speech processing algorithms, one can get meaningful insights of children’s daily language experiences. While daylong
recordings are becoming a central tool for studying how children learn language, a relatively small effort has been made to
propose robust and bias-free speech processing models to analyze such data. In
particular, the speech activity detection task [8] or the acoustic event detection problem [9, 10] are similar to the voice type
classification task we address in this paper. Given the lack of open-source and easy-to-use speech
processing models for treating child-centered recordings, researchers have been relying, for the most part, on the Language
ENvironment Analysis (LENA) software [11] to extract meaningful information about children’s language environment.Where the paper says it
Where the paper says it
Where the paper says it
Model Architecture and Classification Task
The voice type classification problem is described as the task of identifying voice signal sources in a given audio stream. The input to the network is the audio stream divided into N frames S = s1, s2,. p. 1 End-to-end voice type classification
The voice type classification problem can be described as the
task of identifying voice signal sources in a given audio stream. It can be tackled as a multi-label classification problem where
the input is the audio stream divided into N frames S =
{s1, s2, . It can be tackled as a multi-label classification problem where
the input is the audio stream divided into N frames S =
{s1, s2, . , sN } and the expected output is the corresponding
sequence of labels y = {y1, y2, . Note that, in the multilabel setup, multiple classes can be activated at the same time. At training time, fixed-length sub-sequences made of multiple successive frames, are drawn randomly from the training
set to form mini-batches of size M. At training time, fixed-length sub-sequences made of multiple successive frames, are drawn randomly from the training
set to form mini-batches of size M. As illustrated in Figure 1, these fixed-length sub-sequences
are processed by a SincNet [12] that aims at learning meaningful filter banks specifically customized to solve the voice type
classification task. As illustrated in Figure 1, these fixed-length sub-sequences
are processed by a SincNet [12] that aims at learning meaningful filter banks specifically customized to solve the voice type
classification task. These low-level signal representations are
then fed into a stack of bi-directional long short-term memory
(LSTM) layers followed by a stack of feed-forward (FF) layers. These low-level signal representations are
then fed into a stack of bi-directional long short-term memory
(LSTM) layers followed by a stack of feed-forward (FF) layers. Finally, the sigmoid activation function is applied to the final
output layer of dimension K so that each predicted score yˆi,j
consists of a number ranging between 0 and 1.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data Collection and Training Strategy
The model was trained using multiple child-centered corpora covering a wide range of conditions to train the model and compare it against LENA. This data set is described in further details in 2. p. 1 Additionally, we gathered multiple child-centered
corpora covering a wide range of conditions to train our model
and compare it against LENA. This data set is described in further details in 2. 40h08m 10h56m 0m 0m 0m 0m
Tsay public (HB) no Mandarin 132h02m 34h07m 2h08m 10m 57h31m 28m
Tsimane upon agreement mostly Tsimane 9h30m 37m 23m 11m 28m 0m
Vanuatu upon agreement no Mixture 2h29m 12m 5m 5m 9m 1m
WAR2 public (DB) yes English (US) 50m 14m 0m 0m 0m 9m
Hold-out set
ACLEW-Random private (DB) yes Mixture 20h 1h39m 45m 43m 2h48m 0m
wide range of environments, conditions and languages and have
been collected and annotated by numerous field researchers. We will refer to this data set as BabyTrain, of which a broad
description is given in Table 1. We will refer to this data set as BabyTrain, of which a broad
description is given in Table 1. We split the BabyTrain data set into a training, development
and test sets, containing approximately 60%, 20% and 20% of
the audio duration respectively. Child-centered corpora included cover a wide range of conditions (including different
languages and recording devices). ACLEW-Random is kept as a hold-out data set on which LENA and our model are compared. The
last feed-forward layer uses a sigmoid activation function. Data augmentation is applied directly on the waveform using additive noise extracted from the MUSAN database [32]
with a random target signal-to-noise ratio ranging from 5 to 20
dB.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Performance Evaluation Metrics
For each class, the F-measure between precision and recall is used to evaluate our systems. The F-measure is calculated as 2 × precision × recall where precision = tp/(tp + fp) and recall = tp/(tp + fn). p. 2 In order to ensure that our models generalize well enough
to unseen data, and to compare the performance with the LENA
system, we kept the ACLEW-Random as a hold-out data set. Evaluation metric
For each class, we use the F-measure between precision and
recall, such as implemented in pyannote. Evaluation metric
For each class, we use the F-measure between precision and
recall, such as implemented in pyannote. metrics [31] to
evaluate our systems:
F-measure =
2 × precision × recall
precision × recall
where precision = tp/(tp + fp) and recall = tp/(tp + fn) with:
• tp the duration of true positives
• fp the duration of false positives
• fn the duration of false negatives
We select our models by averaging the F-measure across the
5 classes. This class does contribute in activating the SPEECH class, but our model does not return a score for
it. Evaluation protocol
For all experiments, the neural network is trained for 10 epochs
(approximately 2400 hours of audio) on the training set. The
development set is used to choose the actual epoch and thresholds {σj}
K
j=1 that maximizes the average F-measure between
precision and recall across classes. We report both the in-domain performance (computed on
the test set of BabyTrain) and the out-of-domain performance
(computed on the hold-out set, ACLEW-Random).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Comparative Results and Conclusion
Results indicated that multitask approaches were significantly better than binary ones. The model trained in a multi-task fashion shows better performance than the models trained in a binary fashion. p. 3 Performance on the hold-out data set
Table 3 shows performance of LENA, our binary variant, and
our multitask variant on the hold-out data set. As observed on
the test set, the model trained in a multi-task fashion shows better performance than the models trained in a binary fashion. Our model can be
used as input for downstream tasks such as estimating the number of words produced by adult speakers, or the number of linguistic units produced by children. Our architecture combines
SincNet filters with a stack of recurrent layers and outperforms
by a large margin the state-of-the-art system, the Language ENvironment Analysis (LENA) that has been used in numerous
child language studies. Table 2 shows results in terms of F-measure between precision and recall on the test set for each of the 5 classes. The
best performance is obtained for the KCHI, FEM, and SPEECH
classes, which correspond to the 3 classes that are the most
present in BabyTrain (See Table 1). In addition to being underrepresented
in the training set, utterances belonging to the OCH class can
easily be confused with KCHI utterances since the main feature
that differentiates these two classes is the average distance to
the microphone. The multitask model consistently outperforms binary ones. Our model outperforms LENA by a large margin and will, we hope, lead to more
accurate observations of early linguistic environments. Our
work is part of an effort to strengthen collaborations between
the speech processing and the child language acquisition communities. We compared our approach
with a homologous system, the LENA software, which has been
used in numerous child language studies. Our model outperforms LENA by a large margin and will, we hope, lead to more
accurate observations of early linguistic environments.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Reproducible Research
All the code has been implemented using pyannote.audio [34]. p. 0 com, alecristia@gmail. com
Abstract
Spontaneous conversations in real-world settings such as those
found in child-centered recordings have been shown to be
amongst the most challenging audio files to process. audio
[34], a python open-source toolkit for speaker diarization. Our
own code, easy-to-use scripts to apply the pretrained model can
be found on our GitHub repository 1
, which also includes confusion matrices and a more extensive comparison with LENA. Our
own code, easy-to-use scripts to apply the pretrained model can
be found on our GitHub repository 1
, which also includes confusion matrices and a more extensive comparison with LENA. As soon as required agreements will be obtained, we plan to
facilitate access to the data by hosting them on Homebank. The latter have provided data as that used here, as
well as interesting challenges [6, 7]. Our paper is an example of
the speech processing community returning the favor by providing robust models that can handle spontaneous conversations in
real-world settings.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve model learning by using a multitask approach instead of training five separate binary models, as
multitask approaches were significantly better than binary ones
andsharing weights during training helps better learn the boundaries between the different classes.
This leads to a single, more robust model capable of handling all voice types simultaneously. -
Enhance generalization across languages and recording devices by removing the Paido data from both training and development sets, as this
led to improvements on the other test domains, as well as the hold-out set,
suggesting better performance on unseen real-world data. -
Improve performance on underrepresented classes like OCH and MAL by using a multitask model, which shows
increases are higher for the lesser represented classes
compared to binary models. -
Increase classification accuracy for key child vocalizations (KCHI) by training the model with Paido data, as this resulted in an
absolute improvement in terms of F-measure of 13.8 on the KCHI class.
-
Enable detection of overlapping speech instances by treating the problem as a multi-label classification task where multiple classes can be activated at once, unlike LENA which
treat[s] the problem as a multi-class classification task where only one class can be activated at a given time step.
Abstract
Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process. Nevertheless, building speech processing models handling such a wide variety of conditions would be particularly useful for language acquisition studies in which researchers are interested in the quantity and quality of the speech that children hear and produce, as well as for early diagnosis and measuring effects of remediation. In this paper, we present our approach to designing an open-source neural network to classify audio segments into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male speech, and adult female speech. To this end, we gathered diverse child-centered corpora which sums up to a total of 260 hours of recordings and covers 10 languages. Our model can be used as input for downstream tasks such as estimating the number of words produced by adult speakers, or the number of linguistic units produced by children. Our architecture combines SincNet filters with a stack of recurrent layers and outperforms by a large margin the state-of-the-art system, the Language ENvironment Analysis (LENA) that has been used in numerous child language studies.
Sources
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models