Transformer based self-supervised pre-trained models have transformed the concept of Transfer learning in Natural language processing (NLP) using Deep learning approach. Self-attention mechanism made transformers more popular in transfer learning across a broad range of NLP tasks. Among such tasks, Sentiment analysis helps to identify people`s opinions towards a topic, product or service. In this project we analyse the performance of self-supervised models for Multi-class Sentiment analysis on a Non benchmarking dataset. We used BERT, RoBERTa, and ALBERT models for this study. These models are different in design but have the same objective of leveraging a huge amount of text data to build a general language understanding model. We fine-tuned these models on Sentiment analysis with a proposed architecture. We used f1-score and AUC (Area under ROC curve) score for evaluating model performance. We found the BERT model with proposed architecture performed well with the highest f1-score of 0.85 followed by RoBERTa (f1-score=0.80), and ALBERT (f1-score=0.78). This analysis reveals that the BERT model with proposed architecture is best for multi-class sentiment on a Non-benchmarking dataset.

NLP, Transfer learning, Sentiment analysis, BERT, RoBERTa, ALBERT

[1] Ashish Vashwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ?ukasz Kaiser, Illia Polosukhin, “Attention is all you need”, 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA pp.5998-6008, 2017.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding”, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186.
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, “Roberta: A robustly optimized bert pre training approaches”, 2019, arXiv preprint arXiv:1907.11692.
[4] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel,Piyush Sharma, Radu Soricut, “Albert: A lite bert for self-supervised learning of language representations”, 2019, arXiv preprint arXiv:1909.11942.
[5] Matthias Aßenmacher, Christian Heumann, “On the comparability of pre-trained language models”, CEUR Workshop Proceedings, Vol.2624.
[6] Cristóbal Colón-Ruiz, Isabel Segura-Bedmar, “Comparing deep learning architectures for sentiment analysis on drug reviews", Journal of Biomedical Informatics, Volume 110, 2020, 103539, ISSN 1532-0464.
[7] Thanapapas Horsuwan, Kasidis Kanwatchara, Peerapon Vateekul, Bonser Kijsirikul, "A Comparative Study of Pretrained Language Models on Thai Social Text Categorization", 2019, arXiv:1912.01580v1.
[8] Carlos Aspillaga, Andres Carvallo, Vladimir Araujo,"Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks", Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), European Language Resources Association (ELRA), Marseille, pp. 1882–1894, 2020.
[9] Vishal Shirsat, Rajkumar Jagdale, Kanchan Shende, Sachin N. Deshmukh, Sunil Kawale, “Sentence Level Sentiment Analysis from News Articles and Blogs using Machine Learning Techniques”, International Journal of Computer Sciences and Engineering, Vol.7, Issue.5, 2019.
[10] Avinash Kumar1, Savita Sharma, Dinesh Singh, "Sentiment Analysis on Twitter Data using a Hybrid Approach", International Journal of Computer Sciences and Engineering, Vol.-7, Issue-5, May 2019.