Linear Classifier: An Often-Forgotten Baseline for Text Classification

arXiv:2306.07111 · cs.CL, cs.AI, cs.LG · Submitted 2023-06-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Linear Classifier: An Often-Forgotten Baseline for Text Classification".

Jane: The gist The importance of running simple baselines like linear classifiers alongside advanced models such as BERT is highlighted because they can confirm whether the results of advanced models are…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "Linear Classifier: An Often-Forgotten Baseline for Text Classification." It’s about how people are using these huge pre-trained language models like BERT just by training them for a few short epochs and then deploying them.

Jane: And the authors of this paper, Yu-Chen Lin and colleagues, point out that this direct approach may only sometimes give satisfactory results. They argue that we need to look at something simpler alongside these advanced methods.

Tom: Exactly what they claim is the importance of running a simple baseline like linear classifiers using bag-of-words features along with those advanced models. They say this is important because for many text data, linear methods show competitive performance, high efficiency, and robustness.

Lu: I think that makes sense. The core idea here is that if you want to use something powerful like BERT, you need a way to confirm if the results from BERT are actually acceptable before you invest all that effort into it.

Meng: But what do they suggest we use for this simple baseline? What kind of features are we talking about when they mention bag-of-words?

Jane: Well, the paper says LibMultiLabel first generates uni-gram TF-IDF features according to texts in the training set, and those factors are used to get TF-IDF for the test set. Then they use three classic methods that adopt binary linear SVM and logistic regression for multi-class and multi-label scenarios.

Tom: And they specifically focus on one of those methods, which is called One-vs-rest. They say this method learns a binary linear SVM for each label, so data with or without this label are positive or negative, respectively.

Lalam: I see that in Table one they show the Micro-F1 scores from Chalkidis et al <ref:2306.07111#pg1>. (two thousand twenty-two) and they compare it against BERT results too <ref:2306.07111#pg2>. The paper says their one-vs-rest results are slightly worse than the linear SVM results in Chalkidis et al. (two thousand twenty-two) which also applies the one-vs-rest strategy, but our one-vs-rest is still competitive with those BERT results on the last four problems <ref:2306.07111#pg2>.

Jane: That's interesting, so even though it's a simple method, they still find that it can be competitive in some situations when compared to the advanced models. What about the comparison between linear methods and BERT itself?

Tom: This is where things get really telling for me. Surprisingly, their running of LibMultiLabel’s BERT leads to worse test performance than linear methods on almost all data sets they tested.

Meng: Wait, so even when you use BERT directly from this library, it seems like it fails compared to the simple linear approach? That's a strong finding for production deployment.

Lu: It suggests that BERT can achieve superior results only if it is properly applied, and a direct run of BERT in LibMultiLabel is described as a total failure according to their findings.

Lalam: They illustrate this by showing that the results in Chalkidis et al. (two thousand twenty-two) are reproduced through more sophisticated settings like segmenting documents, which demonstrates that BERT can achieve superior results if properly used <ref:2306.07111#pg2>.

Jane: So the main point is that we need a way to confirm if those advanced model results are actually good, and simple linear methods serve as a good way to benchmark that acceptability.

Tom: And when you look at the trade-offs, they bring up performance gain against cost, including running time and model size. Simple methods are useful here because they can justify the usage of those more complex advanced approaches.

Meng: So if we're looking at deploying something to production, this study suggests that when those large language models may not perform significantly better than simpler methods, traditional methods are much simpler and require fewer resources.

Lu: And regarding long document length, they show that using the hierarchical setting to handle long document length may not always lead to the best performance for BERT. So they suggest a general setting for handling long documents still needs further investigation, and applying hierarchical BERT might be data-dependent.

Jane: It really shows that there is a trade-off between getting high accuracy and managing the computational cost associated with those very large models.

Tom: We've got the whole picture now on this paper "Linear Classifier: An Often-Forgotten Baseline for Text Classification." It’s about using simple linear classifiers as a crucial baseline to check if we really need to use massive pre-trained language models like BERT directly.

Jane: The authors emphasize that simple methods provide an efficient and robust way to confirm the proper use of an advanced approach, even if BERT doesn't always win on every single test set they ran.

Lu: It’s a good reminder for the field that when performance gains from large models aren't significant, simpler methods require fewer resources and are often better.

Meng: I think for anyone shipping models to production, knowing when to stop chasing the biggest model is really important information here.

Lalam: This paper highlights that simple linear methods can serve as efficient and robust baselines to confirm the proper use of an advanced approach in NLP applications.

Conclusion: Tom: So we've been looking at this paper, "Linear Classifier: An Often-Forgotten Baseline for Text Classification," and what we’re seeing is that these simple linear methods aren't just academic; they matter when you’re trying to use huge language models.

Jane: Right. So, the main point here is that even though we have these massive models like BERT, if you just train them for a few short epochs and deploy them without checking anything else, the results might not be actually good enough.

Lu: Exactly. The authors argue that running a simple baseline using bag-of-words features along with these advanced methods is super important because it confirms whether the results from the advanced models are actually acceptable.

Meng: What does that mean for us when we're shipping stuff to production? It means we can’t just blindly trust the state of the large model; we need a sanity check against something simpler.

Tom: That’s right, Meng. The paper shows that simple linear methods show competitive performance, high efficiency, and robustness for many text data sets.

Jane: So it's not about replacing BERT entirely; it's about using the linear methods as a way to justify the usage of those more complex advanced approaches.

Lu: It highlights that advanced models like BERT might only achieve their best results if they are applied in a very specific, proper way.

Tom: And one surprising thing the paper points out is that running LibMultiLabel’s BERT directly sometimes performs worse than these simple linear methods on almost every data set.

Jane: That's a big finding. It suggests that when you use these large models, they can only get superior results if you use more sophisticated settings, like segmenting documents to handle long text lengths better.

Lu: So the authors suggest that a general setting for handling long documents still needs more investigation because applying hierarchical BERT might not always give the best performance depending on the data.

Meng: From my side, it's a reminder about trade-offs: you’ve got performance gain versus the cost of running time and model size. Simple methods are just useful tools to benchmark that trade-off.

Tom: So it boils down to this: for some text sets, those simple linear methods give competitive performance right now, and across the whole NLP area, we still need to figure out exactly when to use these massive language models versus simpler methods that require fewer resources.

Jane: And the authors are really pointing toward that big question of when to use these powerful tools and when traditional methods are just much simpler.

Lu: It's a good reminder that for BERT, using those hierarchical settings might not always be the best path, so we still need to investigate how long documents behave.

Meng: So it’s about finding that sweet spot where the performance gain from a large model actually outweighs all those extra costs.

Tom: And next up, we're going to look at how this finding affects the way we deploy these models in real-world scenarios.

National Taiwan University · ASUS Intelligent Cloud Services · Mohamed bin Zayed University of Artificial Intelligence

cs.CL, cs.AI, cs.LG

Submitted: 2023-06-12

Updated: 2023-06-12

Comments: Accepted by ACL 2023

DOI: 10.18653/v1/2023.acl-short.160

Code: https://github.com/JamesLYC88/text_classification_baseline_code

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 75/100

The gist: The gist The importance of running simple baselines like linear classifiers alongside advanced models such as BERT is highlighted because they can confirm whether the results of advanced models are

Key concepts

Bag-of-Words Features: These are simple features created by counting the frequency of words (unigrams) in a text. The paper uses TF-IDF (Term Frequency-Inverse Document Frequency) on these word counts to represent text documents numerically. This is a foundational, non-deep learning way to turn text into data that traditional linear classifiers can understand.
Linear Classifiers: These are straightforward mathematical models, such as Linear SVM or logistic regression, used for classification. The paper specifically uses these methods on the simple bag-of-words features. They are highlighted because they are fast, require minimal tuning, and provide a robust benchmark against complex neural networks like BERT.
BERT Performance Application: The study found that directly applying BERT often yields worse results than linear methods. This suggests that advanced models need proper application, such as segmenting long documents or using hierarchical settings. The paper concludes that simple baselines are necessary to determine when the complexity of a pre-trained language model is actually justified.

Terminology

Robustness — 1× in this paper · explained in 5 episodes

Summary

The gist The importance of running simple baselines like linear classifiers alongside advanced models such as BERT is highlighted because they can confirm whether the results of advanced models are acceptable

Introduction and Motivation

Large-scale pre-trained language models such as BERT are popular solutions for text classification, but directly training them for a few epochs may only sometimes get satisfactory results. The paper argues the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods. For many text data, linear methods show competitive performance, high efficiency, and robustness. Advanced models such as BERT may only achieve the best results if properly applied.

Linear Methods for Text Classification

To use a linear method, LibMultiLabel first generates uni-gram TF-IDF features (Luhn, 1958; Jones, 1972) according to texts in the training set. The paper advocates that simple uni-gram TF-IDF features trained by linear classifiers can be a useful baseline to start with for text classification. Three classic methods adopt binary linear SVM and logistic regression for multi-class and multi-label scenarios. These methods include One-vs-rest, Thresholding, and Cost-sensitive approaches. The paper notes that these methods basically need no further hyperparameter tuning, so we can directly run them.

Experimental Results Comparison

The experimental results show that linear methods are useful baselines. Our one-vs-rest results are slightly worse than the linear SVM results in Chalkidis et al. (2022), which also applies the one-vs-rest strategy. However, our one-vs-rest is still competitive to BERT results in Chalkidis et al. (2022) on the last four problems. The two extensions of one-vsrest (i.e., thresholding and cost-sensitive) improve almost all situations. All the linear methods are still hundreds of times faster than BERT, and linear methods were run on a CPU while for BERT we need a GPU.

BERT Performance and Proper Application

Surprisingly, our running of LibMultiLabel’s BERT leads to worse test performance than linear methods on almost all data sets. This suggests that BERT can achieve superior results if properly used. The paper illustrates this by showing that a direct run of BERT in LibMultiLabel is a total failure, while the results in Chalkidis et al. (2022) are reproduced through more sophisticated settings like segmenting documents. This demonstrates that BERT can achieve superior results if properly used.

Conclusion and Trade-offs

The study reminds us of the trade-off between performance gain and the cost including running time, model size, etc., and simple methods are useful to benchmark and justify the usage of advanced methods. In summary, the study reminds us of the importance of employing simple baselines in NLP applications. For some text sets evaluated in this work, we have seen that simple linear methods give competitive performance. Across all NLP areas, an important issue now is when to use PLMs and when not. The paper demonstrates that when PLMs may not perform significantly better, traditional methods are much simpler and require fewer resources. For BERT, we showed that using the hierarchical setting to handle long document length may not always lead to the best performance. Thus a general setting for handling long documents still need to be investigated. The result of applying hierarchical BERT may be data-dependent. The paper shows that linear methods can serve as efficient and robust baselines to confirm the proper use of an advanced approach.

Limitations

In this work, we do not propose any new methods because, as an opinion paper, we focus on raising the problems and making vivid demonstrations to readers. The experiments are limited to linear SVM and BERT on data sets in the benchmark LexGLUE.

Ethics Statement

We ensure that our work complies with the ACL Ethics Policy.

Acknowledgements

This work was supported by NSTC of Taiwan grant 110-2221-E-002-115-MY3 and ASUS Intelligent Cloud Services. The authors thank Ming-Wei Chang and reviewers for constructive comments.

References

Bernhard E. Boser, Isabelle Guyon, and Vladimir Vapnik. 1992 A training algorithm for optimal margin classifiers In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pages 144–152 ACM Press.

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022 LexGLUE A benchmark dataset for legal language understanding in English In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics pages 4310–4330.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 BERT pre-training of deep bidirectional transformers for language understanding In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT pages 4171–4186.

Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, XiangRui Wang, and Chih-Jen Lin. 2008 LIBLINEAR a library for large linear classification Journal of Machine Learning Research, 9:1871–1874.

Rong-En Fan and Chih-Jen Lin. 2007 A study on threshold selection for multi-label classification Technical report, Department of Computer Science, National Taiwan University.

Christian Gomes, Marcos André Gonçalves, Leonardo Rocha, and Sérgio D. Canuto. 2021 On the costeffectiveness of stacking of neural and non-neural methods for text classification Scenarios and performance prediction In Findings of the Association for Computational Linguistics: ACL/IJCNLP pages 4003–4014.

Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. 2016 Deep Learning The MIT Press.

Karen S. Jones. 1972 A statistical interpretation of term specificity and its application in retrieval Journal of Documentation, 28(1):11–20.

Ken Lang. 1995 Newsweeder Learning to filter netnews In Proceedings of the Twelfth International Conference on Machine Learning pages 331–339.

David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004 RCV1 A new benchmark collection for text categorization research Journal of Machine Learning Research, 5:361–397.

Li-Chung Lin, Cheng-Hung Liu, Chih-Ming Chen, KaiChin Hsu, I-Feng Wu, Ming-Feng Tsai, and Chih-Jen Lin. 2022 On the use of unrealistic predictions in hundreds of papers evaluating graph representations In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI) pages 4310–4330 <ref:2306.

Improvements for AI systems

Improve model selection by implementing a comparative baseline check to justify advanced model usage, as Simple baselines help to confirm whether the results of advanced models are acceptable. This allows practitioners to determine when PLMs may not perform significantly better, traditional methods are much simpler and require fewer resources.
Enhance text classification robustness by employing thresholding or cost-sensitive linear methods when BERT performance is uncertain, as these techniques improve almost all situations and can help reduce the gap between linear and advanced models on certain datasets.
Optimize long document processing in BERT systems by implementing a hierarchical architecture, as the paper demonstrates that for long documents, using the hierarchical setting to handle long document length may not always lead to the best performance.
Develop a comprehensive benchmarking framework that systematically compares linear classifiers (using uni-gram TF-IDF features) against BERT variants across different classification scenarios (multi-class vs. multi-label), as this confirms whether linear methods show competitive performance, high efficiency, and robustness.