An open-source voice type classifier for child-centered daylong recordings

arXiv:2005.12656 · eess.AS · Submitted 2020-05-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "An open-source voice type classifier for child-centered daylong recordings".

Dev: Here's the gist of the paper. Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.

Rosa: This paper presents an open-source neural network designed to classify audio segments into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male speech, and adult female speech.

Dev: The model architecture combines SincNet filters with a stack of recurrent layers and outperforms the state-of-the-art system LENA.

Rosa: The investigation of language phenomenon in daylong recordings can lead to biased observations.

Title and authors: Rosa: Moving on to what the paper actually lays out in terms of results—the summary section says they trained this model using multiple child-centered corpora to test it against LENA. They focused heavily on comparing their approach to how LENA performs across these diverse settings.

Dev: The paper mentions they used datasets like BabyTrain, which covers a wide range of conditions including different languages and recording devices, and they kept ACLEW-Random aside as a hold-out set for that final comparison.

Taro: So the implication there is that they're testing robustness by seeing how well this system holds up when you move from one specific recording setup to another, which is important for real-world deployment.

Rosa: Right, and they also describe their data augmentation process where they add noise extracted from the MUSAN database with a random signal to noise ratio between five and twenty decibels directly onto the waveform. That helps make the model more generalizable.

The paper's summary: Dev: Now let's talk about what they suggest improving, because they point out that multitask approaches were significantly better than binary ones in their findings. They suggest training a single model to handle all those voice types at once instead of training five separate binary models.

Rosa: That makes sense from a learning perspective; sharing weights during training helps the model learn better boundaries between those different classes simultaneously, which should lead to a more robust overall system than running multiple independent classifiers.

Taro: If you do that multitask approach, they mention that it actually increases the performance for the less represented classes like OCH and MAL compared to binary models, which is a big deal for capturing those specific vocalizations.

Dev: They also highlight that by using this multitask method, their model beats LENA by a large margin, especially when looking at performance on the KCHI, FEM, and SPEECH classes.

The paper's improvements: Rosa: So to wrap up the whole paper "An open-source voice type classifier for child-centered daylong recordings," it seems like their main takeaway is that a multitask approach using this specific neural network architecture significantly outperforms the binary methods they were compared against, especially when dealing with those spontaneous conversations.

Dev: The implication for researchers is that they can now have access to a model that handles this wide variety of conditions and languages better than what was available in the previous closed-source LENA system.

Taro: From my side, I think the biggest change is moving away from just one label per time step to a multi-label task where multiple classes can be active at once, which helps detect overlapping speech instances more accurately than what LENA does.

Conclusion: Rosa: So we've covered the title, the summary, and those specific improvements mentioned in "An open-source voice type classifier for child-centered daylong recordings." It really shows how using a multitask approach with this SincNet LSTM structure can give you much better results when classifying those complex vocalizations.

Dev: The paper concludes that their multitask approach leads to more accurate observations of early linguistic environments, which is something that could help researchers who are trying to measure the quantity and quality of speech children hear.

Taro: And for us in terms of future work, they point out a limitation: since LENA is closed source, you can't really build upon their models to improve them directly without their cooperation, which limits how much further that specific pipeline can go.

Rosa: That’s a fair point; it shows the current challenge in moving forward with these types of audio processing tools when you rely on existing software pipelines.

Dev: Well, that covers what this paper has to offer us regarding voice classification, and we're all ready for whatever new audio challenges come up next.

Marvin Lavechin, Ruben Bousbib, Herve Bredin, Emmanuel Dupoux, Alejandrina Cristia

ENS-PSL/CNRS/EHESS · INRIA Paris, France · LIMSI, CNRS, University of Paris-Sud, Universite Paris-Saclay

eess.AS

Submitted: 2020-05-26

Updated: 2021-01-22

Comments: accepted to Interspeech 2020

Journal ref: Proc. Interspeech 2020, 3072-3076

DOI: 10.21437/Interspeech.2020-1690

Code: https://github.com/MarvinLvn/voice-type-classifier

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.

Key concepts

Voice Type Classification
This is the task of identifying the source of a voice signal within an audio stream. Instead of just saying 'speech,' this model distinguishes between different types of sounds, such as a child's specific vocalization versus adult male or female speech.
SincNet Filters
These are specialized filters used in the network architecture designed specifically for learning meaningful filter banks. They help the model extract low-level signal representations from the raw audio data in a way that is optimized for voice type identification.
Multitask Approach
Instead of training five separate models (one for each voice type), this method trains a single, unified model to handle all five classes simultaneously. This sharing of knowledge between tasks results in a more robust and better-performing final system.

Terminology

Summary

Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process. p. 0

Where the paper says it

com, alecristia@gmail. com Abstract Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.

This paper presents an open-source neural network designed to classify audio segments into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male speech, and adult female speech. p. 0
Where the paper says it

Nevertheless, building speech processing models handling such a wide variety of conditions would be particularly useful for language acquisition studies in which researchers are interested in the quantity and quality of the speech that children hear and produce, as well as for early diagnosis and measuring effects of remediation. In this paper, we present our approach to designing an open-source neural network to classify audio segments into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male speech, and adult female speech.

The model architecture combines SincNet filters with a stack of recurrent layers and outperforms the state-of-the-art system LENA. p. 0
Where the paper says it

Our model can be used as input for downstream tasks such as estimating the number of words produced by adult speakers, or the number of linguistic units produced by children. Our architecture combines SincNet filters with a stack of recurrent layers and outperforms by a large margin the state-of-the-art system, the Language ENvironment Analysis (LENA) that has been used in numerous child language studies.

Introduction to the Problem

The investigation of language phenomenon in daylong recordings can lead to biased observations. Technology now allows researchers to efficiently collect and analyze recordings over a whole day. p. 0

Where the paper says it

However, investigating the language phenomenon in this manner can lead to biased observations, potentially resulting in divergent conclusions [3]. More recently, technology has allowed researchers to efficiently collect and analyze recordings over a whole day.

A relatively small effort has been made to propose robust and bias-free speech processing models for this data. p. 0
Where the paper says it

By the combined use of a small wearable device and speech processing algorithms, one can get meaningful insights of children’s daily language experiences. While daylong recordings are becoming a central tool for studying how children learn language, a relatively small effort has been made to propose robust and bias-free speech processing models to analyze such data.

Researchers have often relied on the Language ENvironment Analysis (LENA) software to extract meaningful information about children’s language environment. p. 0
Where the paper says it

In particular, the speech activity detection task [8] or the acoustic event detection problem [9, 10] are similar to the voice type classification task we address in this paper. Given the lack of open-source and easy-to-use speech processing models for treating child-centered recordings, researchers have been relying, for the most part, on the Language ENvironment Analysis (LENA) software [11] to extract meaningful information about children’s language environment.

Model Architecture and Classification Task

The voice type classification problem is described as the task of identifying voice signal sources in a given audio stream. The input to the network is the audio stream divided into N frames S = s1, s2,. p. 1

Where the paper says it

End-to-end voice type classification The voice type classification problem can be described as the task of identifying voice signal sources in a given audio stream. It can be tackled as a multi-label classification problem where the input is the audio stream divided into N frames S = {s1, s2, .

.., sN and the expected output is the corresponding sequence of labels y = y1, y2,. p. 1
Where the paper says it

It can be tackled as a multi-label classification problem where the input is the audio stream divided into N frames S = {s1, s2, . , sN } and the expected output is the corresponding sequence of labels y = {y1, y2, .

.., yN where each yi is of dimension K At training time, fixed-length sub-sequences made of multiple successive frames are drawn randomly from the training set to form mini-batches of size M. p. 1
Where the paper says it

Note that, in the multilabel setup, multiple classes can be activated at the same time. At training time, fixed-length sub-sequences made of multiple successive frames, are drawn randomly from the training set to form mini-batches of size M.

The architecture processes these sub-sequences by feeding them into a SincNet [12] that aims at learning meaningful filter banks specifically customized to solve the voice type classification task. p. 1
Where the paper says it

At training time, fixed-length sub-sequences made of multiple successive frames, are drawn randomly from the training set to form mini-batches of size M. As illustrated in Figure 1, these fixed-length sub-sequences are processed by a SincNet [12] that aims at learning meaningful filter banks specifically customized to solve the voice type classification task.

These low-level signal representations are then fed into a stack of bi-directional long short-term memory (LSTM) layers followed by a stack of feed-forward (FF) layers. p. 1
Where the paper says it

As illustrated in Figure 1, these fixed-length sub-sequences are processed by a SincNet [12] that aims at learning meaningful filter banks specifically customized to solve the voice type classification task. These low-level signal representations are then fed into a stack of bi-directional long short-term memory (LSTM) layers followed by a stack of feed-forward (FF) layers.

Finally, the sigmoid activation function is applied to the final output layer of dimension K so that each predicted score yˆi,j consists of a number ranging between 0 and 1. p. 1
Where the paper says it

These low-level signal representations are then fed into a stack of bi-directional long short-term memory (LSTM) layers followed by a stack of feed-forward (FF) layers. Finally, the sigmoid activation function is applied to the final output layer of dimension K so that each predicted score yˆi,j consists of a number ranging between 0 and 1.

Data Collection and Training Strategy

The model was trained using multiple child-centered corpora covering a wide range of conditions to train the model and compare it against LENA. This data set is described in further details in 2. p. 1

Where the paper says it

Additionally, we gathered multiple child-centered corpora covering a wide range of conditions to train our model and compare it against LENA. This data set is described in further details in 2.

2. The BabyTrain data set covers a wide range of environments, conditions and languages and have been collected and annotated by numerous field researchers. p. 2
Where the paper says it

40h08m 10h56m 0m 0m 0m 0m Tsay public (HB) no Mandarin 132h02m 34h07m 2h08m 10m 57h31m 28m Tsimane upon agreement mostly Tsimane 9h30m 37m 23m 11m 28m 0m Vanuatu upon agreement no Mixture 2h29m 12m 5m 5m 9m 1m WAR2 public (DB) yes English (US) 50m 14m 0m 0m 0m 9m Hold-out set ACLEW-Random private (DB) yes Mixture 20h 1h39m 45m 43m 2h48m 0m wide range of environments, conditions and languages and have been collected and annotated by numerous field researchers. We will refer to this data set as BabyTrain, of which a broad description is given in Table 1.

The data set is split into a training, development and test sets containing approximately 60%, 20% and 20% of the audio duration respectively. p. 2
Where the paper says it

We will refer to this data set as BabyTrain, of which a broad description is given in Table 1. We split the BabyTrain data set into a training, development and test sets, containing approximately 60%, 20% and 20% of the audio duration respectively.

The ACLEW-Random is kept as a hold-out data set on which LENA and our model are compared. p. 2
Where the paper says it

Child-centered corpora included cover a wide range of conditions (including different languages and recording devices). ACLEW-Random is kept as a hold-out data set on which LENA and our model are compared.

Data augmentation is applied directly on the waveform using additive noise extracted from the MUSAN database [32] with a random target signal-to-noise ratio ranging from 5 to 20 dB. p. 2
Where the paper says it

The last feed-forward layer uses a sigmoid activation function. Data augmentation is applied directly on the waveform using additive noise extracted from the MUSAN database [32] with a random target signal-to-noise ratio ranging from 5 to 20 dB.

Performance Evaluation Metrics

For each class, the F-measure between precision and recall is used to evaluate our systems. The F-measure is calculated as 2 × precision × recall where precision = tp/(tp + fp) and recall = tp/(tp + fn). p. 2

Where the paper says it

In order to ensure that our models generalize well enough to unseen data, and to compare the performance with the LENA system, we kept the ACLEW-Random as a hold-out data set. Evaluation metric For each class, we use the F-measure between precision and recall, such as implemented in pyannote.

Models are selected by averaging the F-measure across the 5 classes. p. 2
Where the paper says it

Evaluation metric For each class, we use the F-measure between precision and recall, such as implemented in pyannote. metrics [31] to evaluate our systems: F-measure = 2 × precision × recall precision × recall where precision = tp/(tp + fp) and recall = tp/(tp + fn) with: • tp the duration of true positives • fp the duration of false positives • fn the duration of false negatives We select our models by averaging the F-measure across the 5 classes.

The evaluation protocol involves training the neural network for 10 epochs (approximately 2400 hours of audio) on the training set. p. 2
Where the paper says it

This class does contribute in activating the SPEECH class, but our model does not return a score for it. Evaluation protocol For all experiments, the neural network is trained for 10 epochs (approximately 2400 hours of audio) on the training set.

We report both in-domain performance (computed on the test set of BabyTrain) and out-of-domain performance (computed on the hold-out set, ACLEW-Random). p. 2
Where the paper says it

The development set is used to choose the actual epoch and thresholds {σj} K j=1 that maximizes the average F-measure between precision and recall across classes. We report both the in-domain performance (computed on the test set of BabyTrain) and the out-of-domain performance (computed on the hold-out set, ACLEW-Random).

Comparative Results and Conclusion

Results indicated that multitask approaches were significantly better than binary ones. The model trained in a multi-task fashion shows better performance than the models trained in a binary fashion. p. 3

Where the paper says it

Performance on the hold-out data set Table 3 shows performance of LENA, our binary variant, and our multitask variant on the hold-out data set. As observed on the test set, the model trained in a multi-task fashion shows better performance than the models trained in a binary fashion.

Our model outperforms the LENA model by a large margin. p. 0
Where the paper says it

Our model can be used as input for downstream tasks such as estimating the number of words produced by adult speakers, or the number of linguistic units produced by children. Our architecture combines SincNet filters with a stack of recurrent layers and outperforms by a large margin the state-of-the-art system, the Language ENvironment Analysis (LENA) that has been used in numerous child language studies.

The best performance is obtained for the KCHI, FEM, and SPEECH classes. p. 3
Where the paper says it

Table 2 shows results in terms of F-measure between precision and recall on the test set for each of the 5 classes. The best performance is obtained for the KCHI, FEM, and SPEECH classes, which correspond to the 3 classes that are the most present in BabyTrain (See Table 1).

The multitask model consistently outperforms binary ones. p. 3
Where the paper says it

In addition to being underrepresented in the training set, utterances belonging to the OCH class can easily be confused with KCHI utterances since the main feature that differentiates these two classes is the average distance to the microphone. The multitask model consistently outperforms binary ones.

Our work is part of an effort to strengthen collaborations between the speech processing and the child language acquisition communities. p. 3
Where the paper says it

Our model outperforms LENA by a large margin and will, we hope, lead to more accurate observations of early linguistic environments. Our work is part of an effort to strengthen collaborations between the speech processing and the child language acquisition communities.

The paper concludes that our approach will lead to more accurate observations of early linguistic environments. p. 3
Where the paper says it

We compared our approach with a homologous system, the LENA software, which has been used in numerous child language studies. Our model outperforms LENA by a large margin and will, we hope, lead to more accurate observations of early linguistic environments.

Reproducible Research

All the code has been implemented using pyannote.audio [34]. p. 0

Where the paper says it

com, alecristia@gmail. com Abstract Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process.

Our own code, easy-to-use scripts to apply the pretrained model can be found on our GitHub repository 1. p. 3
Where the paper says it

audio [34], a python open-source toolkit for speaker diarization. Our own code, easy-to-use scripts to apply the pretrained model can be found on our GitHub repository 1 , which also includes confusion matrices and a more extensive comparison with LENA.

As soon as required agreements will be obtained, we plan to facilitate access to the data by hosting them on Homebank. p. 3
Where the paper says it

Our own code, easy-to-use scripts to apply the pretrained model can be found on our GitHub repository 1 , which also includes confusion matrices and a more extensive comparison with LENA. As soon as required agreements will be obtained, we plan to facilitate access to the data by hosting them on Homebank.

The paper is an example of the speech processing community returning the favor by providing robust models that can handle spontaneous conversations in real-world settings. p. 3
Where the paper says it

The latter have provided data as that used here, as well as interesting challenges [6, 7]. Our paper is an example of the speech processing community returning the favor by providing robust models that can handle spontaneous conversations in real-world settings.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve model learning by using a multitask approach instead of training five separate binary models, as multitask approaches were significantly better than binary ones and sharing weights during training helps better learn the boundaries between the different classes. This leads to a single, more robust model capable of handling all voice types simultaneously.

  2. Enhance generalization across languages and recording devices by removing the Paido data from both training and development sets, as this led to improvements on the other test domains, as well as the hold-out set, suggesting better performance on unseen real-world data.

  3. Improve performance on underrepresented classes like OCH and MAL by using a multitask model, which shows increases are higher for the lesser represented classes compared to binary models.

  4. Increase classification accuracy for key child vocalizations (KCHI) by training the model with Paido data, as this resulted in an absolute improvement in terms of F-measure of 13.8 on the KCHI class.

  5. Enable detection of overlapping speech instances by treating the problem as a multi-label classification task where multiple classes can be activated at once, unlike LENA which treat[s] the problem as a multi-class classification task where only one class can be activated at a given time step.

Abstract

Spontaneous conversations in real-world settings such as those found in child-centered recordings have been shown to be amongst the most challenging audio files to process. Nevertheless, building speech processing models handling such a wide variety of conditions would be particularly useful for language acquisition studies in which researchers are interested in the quantity and quality of the speech that children hear and produce, as well as for early diagnosis and measuring effects of remediation. In this paper, we present our approach to designing an open-source neural network to classify audio segments into vocalizations produced by the child wearing the recording device, vocalizations produced by other children, adult male speech, and adult female speech. To this end, we gathered diverse child-centered corpora which sums up to a total of 260 hours of recordings and covers 10 languages. Our model can be used as input for downstream tasks such as estimating the number of words produced by adult speakers, or the number of linguistic units produced by children. Our architecture combines SincNet filters with a stack of recurrent layers and outperforms by a large margin the state-of-the-art system, the Language ENvironment Analysis (LENA) that has been used in numerous child language studies.

Sources

Related papers