SELF & FEIL: Emotion and Intensity Lexicons for Finnish

arXiv:2104.13691 · cs.CL · Submitted 2021-04-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "SELF & FEIL: Emotion and Intensity Lexicons for Finnish".

Jane: The gist: This paper introduces SELF and FEIL, a Sentiment and Emotion Lexicon for Finnish and a Finnish Emotion Intensity Lexicon, respectively,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper "SELF and FEIL: Emotion and Intensity Lexicons for Finnish." Basically, they’ve developed two lexicons: SELF for sentiment and emotion, and FEIL for emotion intensity. Jane The main idea here is that these resources are built by taking existing ones from the NRC Emotion Lexicon and making them specifically right for the Finnish language.

Lu: They claim this is the first comprehensive sentiment and emotion lexicon ever made for Finnish >

Meng: So, what does that mean practically when you’re trying to build an AI system for Finnish text? Jane It means that instead of relying on a rough translation, developers can use these corrected resources to get a much clearer picture of what people are actually feeling in the language.

Tom: Exactly. The authors say they used annotations projected from the NRC Emotion Lexicon but they had to carefully edit those translations >

Lu: They tackled problems with Google Translate choosing just the most common translation for single words, which causes trouble when you have many synonyms and near-synonyms in a lexicon >

Jane: So, it’s not just about having more words; it’s about fixing how those words translate so they keep their original emotional meaning >

Tom: Right. And they did this by examining things like mistranslations on polysemic words, for example, where a word was translated as a tree when the context suggested an action, like flogging >

Lu: They also dealt with issues where automatic translations were too general for some terms, and they replaced those with words that had similar connotations to the original English word >

Meng: From a production standpoint, how much of a problem does this actually solve? Jane Well, when you’re deploying models to handle Finnish text, having these refined resources means less noise in the sentiment analysis because those tricky translation issues are already handled >

Tom: And they did some comparisons. They evaluated it by looking at the affect and mood of a novel called Rautatie by Juhani Aho >

Jane: They compared their SELF lexicon against the original NRC Emotion Lexicon on that novel, and the results showed that SELF provided a plot analysis that matched the actual story development much more closely >

Lu: The paper also looks at how these lexicons perform when comparing them to other resources, like seeing how many matches they get for words or intensities >

Meng: And they found some numbers to back it up. For instance, for duplicate removal, SELF had a rate of forty-six point one percent compared to forty-one point three percent for the NRC lexicon >

Tom: That’s a solid number showing an improvement in accuracy on that task >

Jane: It also showed that for mistranslation sense, SELF showed a rate of two point zero percent compared to four point nine percent for the original NRC lexicon >

Lu: They also looked at the intensity lexicon, which is different because it doesn't include positive or negative sentiments, but gives a score between zero and one for how intense an emotion is >

Meng: So they found that when comparing intensity distributions using kernel density estimates, there’s more matches with the FEIL lexicon than with the NRC lexicon >

Tom: And the literary expert they used agreed that the plot analysis created using SELF resembled the actual plot development of the novel more closely >

Jane: It really shows that these adjustments, which reduced size by twelve point two percent for word-emotion pairs and ten point five percent for intensities, actually made a more accurate tool overall >

Lu: The paper is available on GitHub and they mentioned plans to update those existing ones based on user feedback, plus create new ones with domain-specific alterations >

Meng: I'm curious about the limitations. What did the authors say this method doesn't do? Jane They pointed out that emotion words are closely linked with the culture they are spoken in, and emotions and feelings are expressed quite differently in different languages, which is a general consideration when using resources from one language with another >

Tom: So, the limitation is that while they fixed translation issues, you still have to be mindful of cultural differences because emotions aren't universal across all languages >

Jane: That’s right. The authors acknowledge that the problem wasn't just the initial annotations, but mostly how Google Translate handles single words out of context >

Lu: So, the paper "SELF and FEIL: Emotion and Intensity Lexicons for Finnish" suggests that refining cross-lingual resources by focusing on context and nuance is a very effective way to improve AI for specific languages >

Meng: I think this is useful because it shows how much precision you can gain just by fixing those translation artifacts before you even start training models on the data >

Tom: It’s about moving from a rough guess to a much more reliable signal for understanding human emotion in Finnish literature and text >

Conclusion: Tom: So, we're wrapping up this discussion on the paper "SELF and FEIL: Emotion and Intensity Lexicons for Finnish," which focuses on creating two specialized resources for emotion and intensity in Finnish. Jane The authors, Mohammad and Turney, took existing data from the NRC Emotion Lexicon—the big one—and they heavily edited those translations to make them accurate for Finnish.

Lu: They specifically addressed how Google Translate messes up when it translates single words out of context, which is a huge problem with emotion lexicons because there are so many synonyms and near-synonyms in the language. Meng So, what's the practical result for deploying an AI model? Tom It means less noise in sentiment analysis when you're working with Finnish data because they cleaned up those translation issues that Google Translate introduced. Jane Exactly. They show that this refinement leads to a smaller but much more accurate lexicon for both emotion and intensity in Finnish. Lalam I’m sure this work will be useful because it tackles the real-world problem of imperfect cross-lingual translation when building these kinds of tools.

Tom: The whole paper is called "SELF and FEIL: Emotion and Intensity Lexicons for Finnish." Lu It’s about taking existing, well-known resources and making them better suited for the specific nuances of the Finnish language. Jane What does that mean practically? It means AI systems trying to understand Finnish text will get a much clearer picture of what people are actually feeling. Lu They showed that this refinement leads to a smaller but more accurate lexicon for both emotion and intensity in Finnish. Meng For us in production, that kind of accuracy boost when dealing with Finnish text is what makes deployment actually possible. Tom It’s about moving from a rough guess to a much more reliable signal for understanding human emotion in Finnish literature and text. Jane And this sets the stage for how better language-specific resources can improve the performance of sentiment analysis models across different languages.

Emily Ohman

University of Helsinki

cs.CL

Submitted: 2021-04-28

Updated: 2021-04-28

Comments: unpublished short paper

DOI: 10.5617/dhnbpub.11320

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist: This paper introduces SELF and FEIL, a Sentiment and Emotion Lexicon for Finnish and a Finnish Emotion Intensity Lexicon, respectively, which are valuable augmentations of existing emotion

Key concepts

SELF Lexicon: SELF is a new Finnish lexicon for sentiment and emotion that was created by re-translating existing NRC emotion data. This process involved manually checking and correcting Google Translate's output to ensure the Finnish words accurately captured the original English meanings and emotional connotations, leading to a more nuanced resource.
FEIL Lexicon: FEIL is a Finnish Emotion Intensity Lexicon. Unlike sentiment lexicons, this one doesn't just mark an emotion as present or absent; it assigns a score from 0 to 1 indicating the intensity of that emotion associated with a word. This provides a more detailed measure of how strongly an emotion is expressed in text.
Machine Translation Issues: The creation process highlighted problems with using automated tools like Google Translate for specialized lexicons. These tools often choose the most common translation instead of context-specific ones, causing Finnish translations to lose synonyms and subtle meanings, which SELF and FEIL were designed to correct.
Affect Categories: Affect categories refer to the basic types of emotions that are considered universal across different cultures. While these categories are generally consistent globally, the paper notes that expressing and interpreting these emotions differs significantly depending on the specific language or culture being used.

Terminology

Sentiment and Emotion Lexicon for Finnish — 2× in this paper
Finnish Emotion Intensity Lexicon — 1× in this paper

Summary

The gist: This paper introduces SELF and FEIL, a Sentiment and Emotion Lexicon for Finnish and a Finnish Emotion Intensity Lexicon, respectively, which are valuable augmentations of existing emotion lexicons by addressing issues with machine translation.

Lexicon Creation Process

The lexicon creation process involved augmenting existing resources that were originally created manually or later automatically translated<ref:2104.13691#pg3>The first version of the NRC Emotion Lexicon (a.k.a., EmoLex) (Mohammad and Turney, 2010) was created using Mechanical Turk for crowdsourcing annotations<ref:2104.13691#pg3>The lexicon was later significantly augmented to its current 14,182 lexical items also using Mechanical Turk (Mohammad and Turney, 2013). The developers of the lexicon discovered this led to better agreement scores<ref:2104.13691#pg3>For the first version, the annotators were asked to annotate for emotions evoked by the words, and for the later version, they were asked to annotate for emotions associated with the words as the developers of the lexicon discovered this led to better agreement scores<ref:2104.13691#pg3>.

Lexicon Editing and Corrections

The process of creating SELF and FEIL involved re-translating automatic Finnish translations of NRC Lexicons using Google Translate, where duplicates were marked and all translations were carefully evaluated to match the English word’s meaning and the associated emotions, specifically for the FEIL lexicon, intensity<ref:2104.13691#pg3>The problem with Google Translate is that it chooses the most common translation, especially when translating individual words rather than words in context, which means that with an emotion dictionary with many synonyms and near-synonyms, Google Translate provides the same common word for all as the default translation<ref:2104.13691#pg3>This leads to issues with representativity as the Finnish lexicon, if left as is, would only represent the most common words in Finnish, disregarding any synonyms or nearsynonyms leading to a loss of nuances and insufficient coverage of emotion words<ref:2104.13691#pg3>Duplicates were carefully examined in order to find alternative translations that matched the original word better in terms of meaning and connotation, both by human experts and synonym dictionaries and thesauruses (see table 1 for an example)<ref:2104.13691#pg3>. The paper also identified other types of issues, including connotative-generalization where the meaning of the original word was over-generalized so that the translation had lost all original connotations<ref:2104.13691#pg3>The corrected form is less common but almost identical in meaning and connotation to the original English<ref:2104.13691#pg3>.

Evaluation Methodology

The lexicon was evaluated by having an expert in Finnish literature evaluate the affect and mood of the novel Rautatie by Juhani Aho<ref:2104.13691#pg3>This qualitative evaluation was then compared with both the original version of the NRC Emotion Lexicon and SELF using syuzhet on Rautatie, downloaded from Project Gutenberg<ref:2104.13691#pg3>The results can be seen in figure 1 where on the left is the NRC Emotion Lexicon and on the right SELF<ref:2104.13691#pg3>The two plots show similar overall patterns, but are decidedly different in detail, with SELF providing more accurate plot when evaluated by the expert<ref:2104.13691#pg3>.

Performance Comparison

When comparing the performance of SELF and FEIL against the original NRC lexicon on Rautatie, several differences were observed<ref:2104.13691#pg3>Table 2 shows that for Duplicate removal, SELF had 46.1% compared to 41.3% for NRC<ref:2104.13691#pg3>For Mistranslation-sense, SELF showed a rate of 2.0% compared to 4.9% for NRC<ref:2104.13691#pg3>The emotion word distributions reveal that in general the SELF lexicon finds more emotion words in the novel, but that anger, disgust, and sadness, and therefore also negative were over-represented in the original lexicon, likely by duplicates<ref:2104.13691#pg3>The intensity word distribution comparison shows that while curves show similar peaks for both lexicons, there are many more matches with FEIL (top) than the NRC lexicon (bottom)<ref:2104.13691#pg3>The literary expert agreed that the plot created using SELF resembled the actual plot development of the novel more closely<ref:2104.13691#pg3>.

Conclusion on Lexicon Utility

The SELF and FEIL lexicons have proven to be valuable augmentations of the NRC Emotion and Intensity Lexicons<ref:2104.13691#pg3>The problems with the use of the NRC lexicons’ automatically translated versions do not seem to be caused by any issues with the original annotations (also attested by the thousands of projects that have successfully used them for English), nor with the annotation projection as such, but mostly by Google Translate’s algorithms which are not optimal for translating single words out of context<ref:2104.13691#pg3>SELF and FEIL will hopefully prove to be useful in advancing Finnish emotion detection and sentiment analysis research. The final distribution of the SELF and FEIL lexicons can be seen in table 4 in the Appendix<ref:2104.13691#pg3>. All the adjustments mentioned in this section resulted in an overall reduction in lexicon size by 12.2% from 14,182 word-emotion association pairs to 12,448 entries in Finnish<ref:2104.13691#pg3>The final distribution of the FEIL lexicon with all the adjustments mentioned in this section resulted in an overall reduction in lexicon size by 10.5% from 8,149 intensities to 7,291 entries for Finnish<ref:2104.13691#pg3>. The emotion word distributions (see table 5 in Appendix) reveal that in general the SELF lexicon finds more emotion words in the novel, but that anger, disgust, and sadness, and therefore also negative were over-represented in the original lexicon, likely by duplicates<ref:2104.13691#pg3>. The intensity word distribution in the NRC Intensity Lexicon and FEIL can be compared in figure 2 where the NRC Emotion Intensity Lexicon and FEIL have been matched with Rautatie presented by Kernel Density Estimates (KDE)<ref:2104.13691#pg3>The main difference is the more linear drop towards high intensity negative items in FEIL compared to a plateau between 0.5 and 0.8 for negative items in the NRC lexicon evaluation of Rautatie<ref:2104.13691#pg3>. The literary expert agreed that the plot created using SELF resembled the actual plot development of the novel more closely<ref:2104.13691#pg3>. The paper is available on GitHub<ref:2104.13691#pg3>The SELF and FEIL lexicons are available on GitHub, with plans to both update the existing ones based on user feedback, as well as create new ones with domain-specific alterations<ref:2104.13691#pg3>. The paper is available on arXiv:2104.13691v1 [cs.CL] 28 Apr 2021<ref:2104.13691#pg3>.

--- Page 7 ---

The NRC Emotion Intensity Lexicon (Mohammad, 2018) to a large extent uses the same data as the emotion lexicon, but the annotations were compiled using best-worst scaling (BWS)<ref:2104.13691#pg3>BWS is an efficient method to collect massive amounts of scaled annotations and has been proven to beat rating scales and other methods in both quality and cost<ref:2104.13691#pg3>The main difference between the intensity lexicon and the emotion lexicon is that the intensity lexicon does not include sentiments (positive, negative), and that instead of a Boolean option for each emotion (0 for not associated with a particular emotion, 1 for being associated with a particular emotion) the intensity lexicon gives a score as to how intense the emotion associated with a word is (a score between 0 and 1, with 0 for no association and therefore no intensity, to 1 for the highest intensity)<ref:2104.13691#pg3>There are many considerations that need to be taken into account when using resources created for one language with another language<ref:2104.13691#pg3>Emotion words are closely linked with the culture they are spoken in and emotions and feelings are expressed quite differently in different cultures and languages<ref:2104.13691#pg3>Although research shows the universality of affect categories (Cowen et al., 2019; Scherer and Wallbott, 1994), Mohammad et al.

Improvements for AI systems

A Finnish emotion detection system can be improved by using SELF and FEIL lexicons to overcome limitations of machine-translated NRC Lexicons, as SELF and FEIL improve the results for Finnish language tasks as compared to the original machine translated NRC Lexicons.
The AI system can generate more nuanced sentiment and emotion analyses by incorporating intensity scores from FEIL, which gives a score as to how intense the emotion associated with a word is (a score between 0 and 1).
Finnish NLP models can be enhanced with culturally aware annotations derived from the lexicon editing process, addressing issues like cultural differences and mistranslation identified in Table 2, leading to more accurate emotional representations.
The system can better model plot movement in Finnish literature by utilizing the Syuzhet tool, which is described as an R package used to create graphs approximating plot movement in novels using emotion dictionaries and other tools.
A sentiment analysis pipeline can be refined to reduce noise from synonym issues, as SELF and FEIL incorporate careful edits for synonyms or near-synonyms leading to a loss of nuances and insufficient coverage of emotion words.

Abstract

This paper introduces a Sentiment and Emotion Lexicon for Finnish (SELF) and a Finnish Emotion Intensity Lexicon (FEIL). We describe the lexicon creation process and evaluate the lexicon using some commonly available tools. The lexicon uses annotations projected from the NRC Emotion Lexicon with carefully edited translations. To our knowledge, this is the first comprehensive sentiment and emotion lexicon for Finnish.