XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection

arXiv:2011.01612 · cs.CL · Submitted 2020-11-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection".

Jane: The gist The XED dataset introduces a multilingual fine-grained emotion dataset consisting of human-annotated Finnish and English sentences, along with projected annotations for 30 additional languages,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper titled "XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection," and what this paper is really doing is creating a foundation for emotion work that spans many languages. It’s not just focusing on one language; it’s about expanding the scope significantly across different tongues.

Jane: I get why that matters, because we know that most existing datasets are heavily biased towards English or just a few specific languages. This paper seems to be trying to fix that by offering something usable across a much wider range of languages.

Lu: They are using movie subtitle data from OPUS as their source material, which is interesting because it provides parallel sentences across different languages at the same level of granularity. This structure helps them evaluate things consistently.

Meng: I wonder about the scale of this projection—how many new languages are we talking about? Because if you're projecting annotations for thirty others, that’s a lot of data to manage and test at scale.

Lalam: They specifically mention human-annotated Finnish sentences are twenty-five thousand and English sentences are thirty thousand plus those projected annotations across thirty more languages. That’s the core composition of XED.

The paper's summary: Tom: So to summarize what they found in "XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection," they created this dataset using Plutchik’s Wheel of Emotions—anger, anticipation, disgust, fear, joy, sadness, surprise and trust—plus neutral. They used this to build a multilabel multiclass system.

Jane: The key finding is that when they test this dataset with language-specific models like BERT and SVMs against existing benchmarks like GoEmotions by Demszky et al., two thousand twenty XED performs on par with them, which is a pretty strong result for a cross-lingual resource <ref:2011.01612#pg1>.

Lu: They found that the human-annotated Finnish data gets a macro F1 score of zero point five one when evaluated with FinBERT, and the projected Finnish data has an accuracy of zero point four five in the same evaluation scenario, showing how close they are to human annotation quality for that language.

Meng: So it’s not just that they built a dataset; it’s about proving that these resources are useful tools for sentiment analysis and emotion detection, which is what we need to know if we want to use this in production.

Lalam: They also noted some interesting label combinations, like anger with disgust or joy with trust being the most common beyond single labels, which gives us insight into how people actually express emotions in these texts.

The paper's improvements: Tom: The authors highlight a few improvements, mainly focusing on how they handle the projection aspect of the data. They show that using unique alignment IDs for parallel sentences helped guide the English annotations onto the Finnish sentences, which resulted in an average annotation correlation using Cohen’s kappa of zero point four four.

Jane: That correlation number is interesting because it’s not perfect, but they point out that accuracy by percentage was over ninety percent for that projection, and highest for joy at zero point six five, suggesting the projection method is quite effective even if it differs from human annotation slightly.

Lu: The paper notes that this difference in annotation projection is similar to how human annotations differ from each other, which suggests the projected data isn't completely divorced from reality but rather represents a similar degree of variance.

Meng: If you’re deploying a model, you need to know what the limitations are. They mention that for binary classification—mapping emotions onto positive and negative categories—their model gets an accuracy of zero point eight four zero, which is solid, but they also point out confusion between anger and disgust in the confusion matrix.

Lalam: They also show results with a linear SVM classifier where one class is fear or trust getting a per-class F1 score as high as zero point eight eight three two, which shows that for certain emotions, their method is quite good at distinguishing them.

Conclusion: Tom: So to wrap up the XED paper, it establishes XED as a useful multilingual fine-grained emotion dataset by combining human Finnish and English data with projected annotations for thirty other languages. It shows that this resource performs on par with other state-of-the-art datasets when evaluated by models like BERT and SVMs.

Jane: The implication for us is that we have a new tool to tackle emotion detection in many low-resource languages, moving beyond just English or Finnish data. It’s a real step toward cross-lingual understanding.

Lu: And the fact that they used parallel movie subtitle data to create these datasets means this methodology could be applied to other domains where parallel text exists, which is ambitious for representation learning.

Meng: For deployment, the focus should be on how easily these projected annotations can be scaled up without losing that quality we saw in the correlation scores. We need to see if it translates smoothly from research data to actual live traffic scenarios.

Lalam: Lalam thinks the biggest contribution is opening up emotion datasets for under-resourced languages, giving them a chance to build models and apply sentiment analysis where they previously had very little material.

Tom: So that’s the XED paper. It gives us a resource to work with when building classifiers, especially in those less represented linguistic areas. We'll keep an eye on how the community builds on this foundation for emotion detection in the future.

Emily Ohman, Marc Pamies, Kaisla Kajava, Jorg Tiedemann

University of Helsinki

cs.CL

Submitted: 2020-11-03

Updated: 2020-11-06

Comments: Accepted at COLING 2020

DOI: 10.18653/v1/2020.coling-main.575

Code: https://github.com/Helsinki-NLP/XED

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: The gist The XED dataset introduces a multilingual fine-grained emotion dataset consisting of human-annotated Finnish and English sentences, along with projected annotations for 30 additional

Key concepts

Plutchik’s Wheel of Emotions: This is the specific framework used to label the dataset. It identifies eight core emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. The researchers added a 'neutral' category to this set to create a multilabel classification system.
Multilabel Multiclass Dataset: This means each piece of text can be assigned more than one emotion label simultaneously. Instead of picking just one feeling (like happy or sad), the dataset allows for combinations, such as 'joy with trust' or 'anger with disgust'.
Data Projection: Since direct parallel sentences for all languages were unavailable, the researchers projected English annotations onto Finnish sentences using unique alignment IDs. This method allowed them to create parallel data across different languages while maintaining similar levels of detail.

Terminology

Core — 1× in this paper · explained in 2 episodes
Low-Resource Languages — 1× in this paper · explained in 4 episodes

Summary

The gist The XED dataset introduces a multilingual fine-grained emotion dataset consisting of human-annotated Finnish and English sentences, along with projected annotations for 30 additional languages, providing new resources for many low-resource languages.

Dataset Composition and Annotation Scheme

The dataset consists of human-annotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 additional languages, providing new resources for many low-resource languages We use Plutchik’s core emotions to annotate the dataset with the addition of neutral to create a multilabel multiclass dataset The annotation scheme uses Plutchik’s Wheel of Emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) (Plutchik 1980) as our annotation scheme with the addition of neutral on movie subtitle data from OPUS (Lison and Tiedemann 2016)

Evaluation Methodology

The dataset is carefully evaluated using language-specific BERT models and SVMs to show that XED performs on par with other similar datasets and is therefore a useful tool for sentiment analysis and emotion detection We perform evaluations with fine-tuned cased multilingual and language specific BERT (Bidirectional Encoder Representations from Transformers) models, as well as Suport Vector Machines (SVMs) Our evaluations show that the human-annotated datasets behave on par with comparable state-of-the-art datasets such as the GoEmotions dataset (Demszky et al. 2020) Furthermore, the projected datasets have accuracies that closely resemble human-annotated data with macro f1 scores of 0.51 for the human annotated Finnish data and 0.45 for the projected Finnish data when evaluating with FinBERT (Virtanen et al. 2019)

Data Characteristics

The XED dataset contains 24,164 emotion annotations excluding neutral on 17,520 unique sentences A majority of the subtitles for English were assigned one emotion label (78%), 17% were assigned two, and roughly 5% had three or more categories (see also Table 3) The number of unique label combinations is 147, including single-label The most common label combinations beyond single-label are anger with disgust (2.4%) and joy with trust (2.1%) followed by different combinations of the positive emotions of anticipation, joy, and trust

Data Source and Projection

We use the OPUS (Lison and Tiedemann 2016) parallel movie subtitle corpus of subtitles collected from opensubtitles.org as a multi-domain proxy Because OPUS open subtitles is a parallel corpus we are able to evaluate our annotated datasets across languages and at identical levels of granularity We use the 11,128 Finnish sentences for which directly parallel sentences existed and projected the English annotations on them using the unique alignment IDs for both languages as guide The average annotation correlation using Cohen’s kappa is 0.44 (although accuracy by percentage is over 90%), and highest for joy at 0.65, showing that annotation projection differs from human annotation to a similar degree as human annotations differ from each other

Evaluation Results

We achieve macro f1 scores of 0.54 for our multilabel classification with a fine-tuned BERT model Using named-entity recognition increases the accuracy slightly For binary data mapped from the emotion classifications onto positive and negative (non-multilabel classification) our model achieves a macro f1 score of 0.838 and accuracy of 0.840 Our linear SVM classifier using one-vs-rest achieves an f1 score of 0.502 with per class f1 scores between 0.8073 (anger) and 0.8832 (fear & trust) (see tables 7 and 8) The confusion matrix reveals that disgust is often confused with anger, and to some extent this is true in the other direction as well

Discussion

The results from the dataset evaluations show that the XED is on par with other similar datasets, but they also stress that reliable emotion detection is still a very challenging task The best accuracies are those that are in line with annotator agreement Perhaps the greatest contribution of all is that, for the first time, many under-resourced languages have emotion datasets that can be used in other possible downstream applications as well

--- Page 1 ---

XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection Emily Ohman ¨ and Marc Pamies & Kaisla Kajava and Jorg Tiedemann University of Helsinki firstname.lastname@helsinki.fi Abstract We introduce XED, a multilingual fine-grained emotion dataset. The dataset consists of humanannotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 additional languages, providing new resources for many low-resource languages. We use Plutchik’s core emotions to annotate the dataset with the addition of neutral to create a multilabel multiclass dataset. The dataset is carefully evaluated using language-specific BERT models and SVMs to show that XED performs on par with other similar datasets and is therefore a useful tool for sentiment analysis and emotion detection. 1 Introduction There is an ever increasing need for labeled datasets for machine learning. This is true for English as well as other, often under-resourced, languages. We provide a cross-lingual fine-grained sentence-level emotion and sentiment dataset. The dataset consists of parallel manually annotated data for English and Finnish, with additional parallel datasets of varying sizes for a total of 32 languages created by annotation projection. We use Plutchik’s Wheel of Emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) (Plutchik 1980) as our annotation scheme with the addition of neutral on movie subtitle data from OPUS (Lison and Tiedemann 2016). We perform evaluations with fine-tuned cased multilingual and language specific BERT (Bidirectional Encoder Representations from Transformers) models, as well as Suport Vector Machines (SVMs). Our evaluations show that the human-annotated datasets behave on par with comparable state-of-the-art datasets such as the GoEmotions dataset (Demszky et al. 2020). Furthermore, the projected datasets have accuracies that closely resemble human-annotated data with macro f1 scores of 0.51 for the human annotated Finnish data and 0.45 for the projected Finnish data when evaluating with FinBERT (Virtanen et al. 2019). The XED dataset can be used in emotion classification tasks and other applications that can benefit from sentiment analysis and emotion detection such as offensive language identification. The data is open source licensed under a Creative Commons Attribution 4.0 International License (CC-BY). In the following sections we discuss related work and describe our datasets. The datasets are then evaluated and the results discussed in the discussion section. 2 Background & Previous Work Datasets created for sentiment analysis have been available for researchers since at least the early 2000s (Mantyl ¨ a et al. 2018). Such datasets generally use a binary or ternary annotation scheme (positive, negative + neutral) (e.g. Blitzer et al. 2007) and have traditionally been based on review data such as, e.g. Amazon product reviews, or movie reviews (Blitzer et al. 2007; Maas et al. 2011; Turney, 2002). Many, if not most, emotion datasets on the other hand use Twitter as a source and individual tweets as level of granularity (Schuff et al. 2017; Abdul-Mageed and Ungar 2017; Mohammad et al. 2018). In the case of emotion datasets, the emotion taxonomies used are often based on Ekman (1971) and Plutchik (1980) (which is partially based on Ekman). 2.1 Existing Emotion Datasets Bostan and Klinger (2018) analyze 14 existing emotion datasets of which only two are multilabel. These are AffectiveText (Strapparava and Mihalcea, 2007) and SSEC (Schuff et al. 2017). Nearly all of these datasets use an annotation scheme based on Ekman (Ekman, 1971; Ekman, 1992) with many adding a few labels often following Plutchik’s theory of emotions (Plutchik 1980). A typical emotion dataset consists of 6-8 categories. The exception Bostan and Klinger (2018) mention is CrowdFlower2 with 14 categories, and those not mentioned in Bostan et al. are e.g.

Improvements for AI systems

textbfMultilingual Emotion Detection for Low-Resource Languages: Deployment of XED Dataset with Projected Annotations (32 languages). This system can perform fine-grained emotion classification across a broad range of languages, including English and Finnish, by leveraging the projected annotations mentioned in Section 3.4.1 to generate resources for under-resourced languages where human annotation is scarce.
textbf Enhanced Sentiment Analysis via Multilabel Classification: Utilizing XED's Fine-Grained Taxonomy (8 core emotions + neutral). The improved system can classify texts with a high degree of nuance, as demonstrated by the evaluation results showing macro f1 scores of 0.54 for our multilabel classification with a fine-tuned BERT model.
textbf Cross-Domain Emotion Transfer: Application of XED to Unseen Domains (e.g., Social Media). The system can be adapted for tasks outside movie subtitles because the source data covers several different genres and, although scripted, represents real human language used in a multitude of situations similar to many social media platforms, allowing it to perform emotion classification tasks and other applications that can benefit from sentiment analysis and emotion detection such as offensive language identification.
textbf Robust Model Selection: Employing BERT for State-of-the-Art Performance. The improved system should utilize a fine-tuned English uncased BERT for classification, achieving a high macro f1 score of 0.536, which is shown to be comparable to other similar datasets when classes are merged (e.g. Demszky et al., 2020).
textbf Context-Aware Annotation Handling: Utilizing Named Entity Recognition for Improved Accuracy in English Data. By using Stanford NER (named entity recognition) (Finkel et al., 2005) to replace names and locations with the tags: [PERSON] and [LOCATION], the system can achieve an accuracy improvement, as shown by English with NER, BERT 0.536 0.544.

Abstract

We introduce XED, a multilingual fine-grained emotion dataset. The dataset consists of human-annotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 additional languages, providing new resources for many low-resource languages. We use Plutchik's core emotions to annotate the dataset with the addition of neutral to create a multilabel multiclass dataset. The dataset is carefully evaluated using language-specific BERT models and SVMs to show that XED performs on par with other similar datasets and is therefore a useful tool for sentiment analysis and emotion detection.