LT@Helsinki at SemEval-2020 Task 12: Multilingual or language-specific BERT?

arXiv:2008.00805 · cs.CL · Submitted 2020-08-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LT@Helsinki at SemEval-2020 Task 12".

Jane: The gist The results show that offensive tweet classification is one of several language-based tasks where BERT can achieve state-of-the-art results,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're wrapping up this look at the LT@Helsinki team's submissions for SemEval two thousand twenty Task twelve which is all about identifying offensive language and finding what those insults are targeting <ref:2008.00805#pg1>.

Jane: It really boils down to whether you use a general multilingual model or a model specifically tuned for one language to get better results.

Tom: Yeah, the paper is titled "LT@Helsinki at SemEval-two thousand twenty Task twelve: Multilingual or language-specific BERT?" and it looks at that exact question.

Lu: The authors are looking at models pre-trained by Google's BERT and seeing how they perform on different languages like Danish, Arabic, Greek, and Turkish.

Meng: They used datasets like OLID and SOLID for training their specific models on sub-tasks A and C.

Jane: They found that language-specific models actually performed better than the multilingual ones when they looked at the Danish data in sub-task A.

Tom: That's interesting because it suggests that for certain languages, a model trained just on that language is way more effective than one trying to handle everything at once.

Lu: And they point out that while general models can do decent work with very little data, language-specific fine-tuning seems to give you an edge when the target language is close to English.

Jane: So what does this mean for us? It suggests that if you're working on a language that isn't super close to English, like Arabic or Turkish, a tailored model might actually be the way to go instead of just using a big multilingual one.

Tom: Exactly. It’s not always about throwing the biggest AI model at the problem; sometimes it’s about picking the right tool for the specific job and language.

Meng: And they did flag some challenges too, like how difficult class imbalance is when you're trying to identify targets in sub-task C.

Jane: Right, so while they showed these performance differences, there are still hurdles in making sure those models work perfectly across all languages and all the different types of offense.

Tom: We'll get into that next, looking at what they suggested for future work on improving those systems.

Lu: The authors suggest that further work should focus on comparing these two types of BERT models more rigorously to figure out exactly when you should choose one over the other.

Jane: It sounds like the future direction for this research is really about establishing clear guidelines for model selection based on language characteristics rather than just picking the biggest model available.

Meng: And they did flag some challenges too, like how difficult class imbalance is when you're trying to identify targets in sub-task C.

Jane: Right, so while they showed these performance differences, there are still hurdles in making sure those models work perfectly across all languages and all the different types of offense.

Tom: We'll get into that next, looking at what they suggested for future work on improving those systems.

Conclusion: Tom: So, we've looked at the details of this paper about SemEval two thousand twenty Task twelve, which is all about identifying offensive language and finding what those insults are targeting. Jane It really comes down to that big question they posed in the title: Multilingual or language-specific BERT?

Tom: Yeah, the authors are taking models pre-trained by Google's BERT and testing how well they do on different languages like Danish, Arabic, Greek, and Turkish. Lu They used datasets like OLID and SOLID for training their specific models on sub-tasks A and C. Meng And they found that language-specific models actually performed better than the multilingual ones when they looked at the Danish data in sub-task A.

Jane: That suggests that for certain languages, a model trained just on that language is way more effective than one trying to handle everything at once. Tom Exactly, it means you can get good results even with less general training data if you tailor the model to your specific language. Lu And they point out that while general models can do decent work with very little data, language-specific fine-tuning seems to give you an edge when the target language is close to English.

Jane: So what does this mean for us? It suggests that if you're working on a language not very close to English, like Arabic or Turkish in this study, you might get better results using a model tailored just for that language instead of relying on the general multilingual one. Tom And they point out in their future work that they should do a more thorough comparison between multilingual and language-specific BERT to see if language-specific models are always superior or if it depends entirely on the specific language. Lu They also considered looking at emotion-laden words in offensive texts as a potential avenue for improvement, but they weren't able to meaningfully use that information because offensive tweets didn't actually contain many emotion words while non-offensive tweets did.

Jane: So the paper shows that while BERT is strong for these tasks across different languages, the choice between a general multilingual model and a language-specific one really depends on how close your target language is to English. Tom And they point out in their future work that they should do a more thorough comparison between multilingual and language-specific BERT to see if language-specific models are always superior or if it depends entirely on the specific language.

Marc Pamies, Emily Ohman, Kaisla Kajava Jorg Tiedemann

University of Helsinki

cs.CL

Submitted: 2020-08-03

Updated: 2020-08-03

Comments: Accepted at SemEval-2020 Task 12. Identical to camera-ready version except where adjustments to fit arXiv requirements were necessary

DOI: 10.18653/v1/2020.semeval-1.205

Code: https://github.com/botxo/nordic_bert

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 76/100

The gist: The gist The results show that offensive tweet classification is one of several language-based tasks where BERT can achieve state-of-the-art results, and that language-specific models can outperform

Key concepts

Bidirectional Encoder Representations from Transformer (BERT): BERT is a powerful deep learning model pre-trained by Google that can understand the context of words in both directions. It was fine-tuned on specific datasets like OLID and SOLID to perform offensive language detection tasks.
Language-Specific Models vs. Multilingual Models: The study compared using a single multilingual BERT model against models specifically trained for certain languages, such as Nordic BERT for Danish. The findings suggest that language-specific models can outperform multilingual ones when dealing with certain languages like Danish.
Class Imbalance in Sub-task C: Sub-task C required identifying the target of offense, which had a highly imbalanced dataset with three classes instead of two. To handle this, the team used data augmentation and over-sampling techniques to create a balanced dataset for training.

Terminology

Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The results show that offensive tweet classification is one of several language-based tasks where BERT can achieve state-of-the-art results, and that language-specific models can outperform multilingual models for certain languages like Danish, while class imbalance presents challenges in multi-class identification.

Background

The study addresses the problem of offensive language detection in social media given the massive volume of tweets generated every second (Page 1). The approach relies on Bidirectional Encoder Representations from Transformer (BERT) models pre-trained by Google and fine-tuned on specific datasets like OLID and SOLID (Page 1). The task is divided into three sub-tasks: A, Offensive Language Identification in several languages; B, Categorization of Offense Types; and C, Offense Target Identification (Page 1).

Data and Models Used

The training data for the classifiers was provided by the OffensEval organizers (Page 2). The datasets used for sub-task A included Arabic, Danish, Greek, and Turkish tweets with specific counts for training, test sets, and total samples (Page 2). For the Danish dataset, Nordic BERT1 was used because it was pretrained on Danish Wikipedia texts and other Danish sources (Page 2). For other languages like Arabic and Turkish in sub-task A, standard BERT-Base models were used with only the provided datasets for each language (Page 2). In sub-task C, the English dataset consisted of a three-level hierarchical model where tweets were labeled as OFF or NOT OFFENSIVE, followed by targeted (TIN) or untargeted insults (UNT), and finally categorized as individual (IND), group (GRP), or other entity (OTH) for the target identification level (Page 2).

System Implementation for Sub-tasks A and C

For sub-task A on the Danish dataset, two baseline methods were compared: random forest and BERT. The random forest implementation involved pre-processing steps such as lower-casing characters, removing irrelevant punctuation marks, reducing the length of characters that appear more than two consecutive times, and converting hashtags to sentences (Page 3). Tokenization was performed using the TweetTokenizer tool from NLTK (Page 3). For sub-task A, the results showed a macro-F1 score of NOT 0.465, Random Forest with TFIDF 0.773, and Multilingual BERT-Base 0.768, with Nordic BERT achieving a higher score of 0.804 (Page 3).

Addressing Class Imbalance in Sub-task C

Sub-task C presented a difficulty due to the highly imbalanced dataset composed of three classes instead of two (Page 4). To combat this imbalance, the team focused on balancing the data by adding instances from non-majority classes (GRP and OTH) and applying an over-sampling technique with replacing, which produced a totally balanced dataset of 11,628 tweets (Page 4). The model used for this task was a BERT-base-uncased model with sequence lengths set to 128 (Page 4).

Performance Results

The performance on the Danish sub-task A, using Nordic BERT for the final submission, yielded an accuracy of 92.38% and an F1-score of 81.18% (Page 5). For other languages, Greek achieved an F1-score of 82.6%, Turkish achieved 77.2%, and Arabic achieved 73.1% (Page 5). In sub-task C, the submission ranked second with an accuracy of 79.74% and an F1-score of 66.99% (Page 5). The low F1-score in sub-task C was attributed to the high numbers of misclassifications of the minority class (OTH) (Page 5).

Conclusion and Future Work

The study concludes that models pre-trained on general corpora can obtain competitive results when there is very little data available, as seen in the Danish sub-task where it obtained the best results (Page 5). The authors suggest that multilingual BERT works best with languages similar to English, implying language-specific models might benefit other languages more than Danish (Page 5). Future work should involve a more thorough comparison between multilingual and language-specific BERT to determine if language-specific models are always superior or dependent on the language (Page 5). The team also considered examining emotion-laden words in offensive texts as a potential avenue for improvement (Page 5).

How it works

The system used BERT models fine-tuned on specific datasets for sub-tasks A and C (Page 1).
For sub-task A, the Danish model utilized Nordic BERT, which was pretrained on Danish Wikipedia texts and other Danish sources (Page 2).
For sub-task C, class imbalance was mitigated by training on augmented data and applying an over-sampling technique with replacing to achieve a balanced dataset (Page 4).
Performance evaluation involved calculating macro-F1 scores for sub-task A and accuracy and F1-scores for sub-task C (Page 3, Page 5).
The team found that language-specific models performed better than multilingual models for the Danish data set in sub-task A (Page 1).

Unfruitful Attempts at Performance Improvement

The team attempted to improve performance by augmenting the data using the NRC Emotion Lexicon to compare normalized word counts per offensive post (Page 3). However, they were unable to meaningfully utilize this information because offensive tweets contained no emotion words and many non-offensive tweets did contain them (Page 3). This indicates that augmentation based on emotion words did not significantly boost performance for this specific task (Page 3).

References

Pete Burnap and Matthew L Williams. 2015. Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making. Policy & Internet, 7(2):223–242 (Page 6).

C¸ agrı C¸ ˘ oltekin. 2020. A Corpus of Turkish Offensive Language on Social Media. In ¨ Proceedings of the 12th International Conference on Language Resources and Evaluation. ELRA (Page 6).

Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM (Page 6).

Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17), pages 86–95 (Page 6).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (Page 6).

Bjorn Gamb ¨ ack and Utpal Kumar Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In ¨ Proceedings of the first workshop on abusive language online, pages 85–90 (Page 6).

Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018a. Benchmarking aggression identification in social media. In TRAC@COLING 2018 (Page 6).

Ritesh Kumar, Atul Kr Ojha, Marcos Zampieri, and Shervin Malmasi. 2018b. Proceedings of the first workshop on trolling, aggression and cyberbullying (trac-2018). In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018) (Page 6).

Els Lefever, Bart Desmet, and Guy De Pauw. 2018. Ta-cos 2018: 2nd workshop on text analytics for cybersecurity and online safety: Proceedings. In TA-COS 2018–2nd Workshop on Text Analytics for Cybersecurity and Online Safety, collocated with LREC 2018, 11th edition of the Language Resources and Evaluation Conference (Page 6).

European Language Resources Association (ELRA). (Page 6).

Ping Liu, Wen Li, and Liang Zou. 2019. Nuli at semeval-2019 task 6: transfer learning for offensive language detection using bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 87–91 (Page 6).

Shervin Malmasi and Marcos Zampieri. 2018. Challenges in discriminating profanity from hate speech. Journal of Experimental & Theoretical Artificial Intelligence, 30(2):187–202 (Page 6).

Binny Mathew, Ritam Dutt, Pawan Goyal, and Animesh Mukherjee. 2019. Spread of hate speech in online social media. In Proceedings of the 10th ACM Conference on Web Science, WebSci ’19, page 173–182 (Page 6).

Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465 (Page 6).

Improvements for AI systems

textbf Language-Specific Model Superiority for Low-Resource Languages in Sub-task A (Danish Focus): Activating language-specific models over multilingual ones maximizes performance when data is scarce, as the paper found that at least for this data, the language-specific model worked better than the multilingual.
textbf Targeted Class Imbalance Mitigation via Data Augmentation in Sub-task C: Employing an over-sampling technique with replacing on minority classes to achieve a totally balanced dataset of 11,628 tweets (3,876 for each class) prevents high misclassification rates caused by skewed distributions.
textbf Contextual Feature Integration for Offensive Identification: Incorporating emotion-laden words via the NRC Emotion Lexicon could enhance detection accuracy, although the paper noted that we were unable to meaningfully utilize this information to improve the performance of our system without context or negation.
textbf Model Selection Based on Pre-training Depth: Prioritizing models with deeper pre-training, such as Nordic BERT 2, over original Base versions significantly boosts performance, especially when training data is limited.

Abstract

This paper presents the different models submitted by the LT@Helsinki team for the SemEval 2020 Shared Task 12. Our team participated in sub-tasks A and C; titled offensive language identification and offense target identification, respectively. In both cases we used the so-called Bidirectional Encoder Representation from Transformer (BERT), a model pre-trained by Google and fine-tuned by us on the OLID and SOLID datasets. The results show that offensive tweet classification is one of several language-based tasks where BERT can achieve state-of-the-art results.