Customer Sentiment Analysis for E-Commerce: A Hybrid approach using CNNs & BERT

Abstract

As the online shopping industry continues to grow at a staggering rate, e-commerce channels have also begun the use of automated systems for sentiment analysis so as to be able to assess comprehension of user reviews. Over the past few years, machine learning (ML) has come of age with the introduction of deep learning models such as Convolutional Neural Networks (CNNs) and Bidirectional Encoder Representations from Transformers (BERT) which have greatly improved the levels of accuracy achieved in sentiment analysis. This paper aims to address this problem by proposing a hybrid model, based on networks of convolutional neural networks (CNNs) and BERT, for the purposes of e-commerce sentiment classification. Posting more plausible situations, academic findings, and perspectives for future research, we focus more on practical issues. When we compared traditional approaches with our hybrid, it became clear that the model performs better in many dimensions: accuracy, precision, and progressively recall.

Introduction

I. INTRODUCTION

The proliferation of e-commerce has increased the volume of customer feedback, including but not limited to product reviews, mentions on social media, and more direct contact with customers. Due to this, sentiment analysis—the automatic retrieval of opinions and emotions embedded in the text, has turned out to be very significant in as far as understanding consumer satisfaction, trends, and how to manage the brand's activities effectively. In consideration of the increasing levels of conducting business activities through the internet, the level of user-generated content available has become too much for manual analysis, hence the need for automated users’ feedback analysis.

This is especially the case because the types of feedback received are very diverse, ranging from straightforward star ratings to complex and detailed reviews. Unfortunately, classic sentiment analysis techniques like Naïve Bayes and Support Vector Machines (SVM) have performed poorly on these datasets, which are extensive and unstructured.

Their classic views, however, prove ineffective on unstructured data in the modern age that is dominated by features like social media. These older models are often dependent on the use of feature engineering and basic writing modes, which include Bag of Words (BoW) models or Term Frequency-Inverse Document Frequency (TF-IDF) models, which do capture language features like sarcasm or ambiguities.

Therefore, due to the reliance on traditional approaches, most of the customer feedback is counterproductive because businesses are unable to tap into certain vital information. Due to that existing reality, the evolution of new advanced applications is necessitated to fit the ever-changing data of such markets, and therefore deep learning algorithms such as CNNs and BERT had to be introduced. [1], [2], [3].

II. MACHINE LEARNING MODELS FOR CUSTOMER SENTIMENT ANALYSIS

A. Traditional Models

Conventional approaches such as Naïve Bayes, Decision Trees, or Support Vector Machines (SVM) have been effective in previously conducted works related to sentiment analysis.

These systems have allowed the construction or retrieval of extracted features from a text with a representation like a bag of words (BoW) or Term Frequency-Inverse Document Frequency (TF-IDF). While these approaches have been instrumental in the evolution of text mining and analysis, they struggle [1], [2] with intricate language forms and their contextual use. These sentiment analysis tools also have a weakness in the analysis of dependent sentiment, such as mocking with words, using figures of speech, or even words with opposite meaning; these phrases are often seen in product reviews found online [3], [4].

Fig. 1  CNN Architecture

B. Deep Learning Models

With respect to the foregoing, Deep learning models designed for instance, by means of CNNs prove to be a major advancement considering that they are able to learn features directly from text on their own. These convolutional neural networks, in other words, came from computer vision because they help in the data analysis of huge volumes of data and, in the process, are able to identify features in the text without the use of manual feature engineering. These models, for instance, are very good at detecting certain localized aspects of text, such as particular words or phrases associated with either the positive or the negative polarity. Their limitation, however, is that they do not have the ability to understand the overarching picture very well, especially in cases when the text consists of long reviews and the reader is expected to comprehend the linkage among sentences [5], [6]. CNNs do an excellent job of analyzing brief and to the point reviews but fail to interpret the elaborate crux of longer customer feedback.

C. Transformer Models- BERT

The BERT architecture, which is a transformer-based model, surpasses the limitations of the convolutional neural network (CNN) language models in that it understands the relationship between words in a whole sentence or even a paragraph. BERT is bidirectional in that it determines the meaning of a word based on its context and the words that come before and after it; this allows BERT to recognize a deeper level of meaning. This is why BERT works well for long reviews that possess a lot of context since comprehension of the overall sentiment of such reviews depends on the text’s other constituents. Such a clause can turn a seemingly positive statement upside down, a clause that those modeling CNNs fail to deal with. Since CNNs are good as feature extractors and BERT is good in providing context, our model is a hybrid of the two in order to achieve the best of both worlds.

Fig. 2 BERT Architecture

D. Proposed Hybrid Solution

In this study, we present a novel hybrid approach for e-commerce reviews sentiment analysis that employs both CNN and BERT frameworks. The CNN part extracts information from short text features, while the BERT understands the overall sentiment from the complex, long structure of data. By combining such two models, we are able to appreciate the sentiment nuances involved in statements like “fast delivery” while also appreciating detailed reviews that can be spread across a couple of sentences or even paragraphs. For instance, whereas CNN can only highlight the positive sentiment words, BERT can classify the sentiments of a review accurately based on the entire content provided.[2], [4], [5]

III. MODEL ARCHITECTURE

The design of our suggested hybrid model incorporates a CNN layer to begin with, which scans the input text for certain key phrases that portray strong emotions. These local features are then forwarded to a pre-trained BERT model, which takes care of localized features in relation to the whole text rather than individual features. Finally, the integrated features are processed in a classification layer whose output is the sentiment category (positive, negative, or neutral). Not only does this combination lead to better accuracy, it also eliminates some of the disadvantages that come with independently utilizing either model [5], [6].

Fig. 3 Proposed Hybrid Architecture

IV. EXPERIMENTAL RESULTS

In order to ascertain the performance of the hybrid model, we ran a few experiments using two sets of data: the IMDB movie database review dataset and a Twitter Sentiment dataset. It was noted that the hybrid model gave better results than CNN and BERT when used solely, with general accuracy of 93% achieved. This is a notable improvement over existing methods such as Naïve Bayes and SVM, as well as single deep learning approaches like LSTMs [1] [3]. In addition to this, the model showed an increase in recall and precision, making it even more effective when it comes to searching for any sentiment in various types of reviews [5] [6].

Fig. 4  Performance Metrics- Model Accuracy & Model Recall   

Fig. 5  Performance Metrics- Model Precision & F1-Score

Fig. 6  Performance Metrics- Model Precision & F1-Score

V. APPLICATION IN E-COMMERCE

Customer reviews are in no short supply on e-commerce websites; Amazon and Flipkart, for example, have lots of reviews that vary from simple thumbs up or thumbs down to long and complicated reviews. [3], [4]. When we applied our observation model to real-world e-commerce data, that hybrid model further boosted the accuracy of sentiment detection in comparison to the other methods. For the more nuanced cases, such as “The product works great, but the consumer service was horrible,” BERT enabled them to be categorized as a mixed review with more precision, primarily because of its contextual understanding that is from the sentence level to the document level. Meanwhile, CNN's capacity to handle short and impactful phrases like ‘great quality’ was particularly useful as it helped the model to perform well regardless of the review lengths and complexities. [2], [4], [5].

VI. ETHICAL CONSIDERATIONS

Although sentiment analysis has its advantages, it also brings a number of ethical issues to the fore. The principal focus is privacy, as models do not always seek permission before analyzing personal data in bulk. In order to abide by laws such as the General Data Protection Regulation (GDPR), all the data that a company wants to use has to be anonymized [3], [5]. Further, sentiment analysis systems may be biased based on the training data used for building such systems. For instance, a model could inappropriately tag positive sentiments when evaluated on datasets with a disproportionate representation of negative sentiments from the population. This implies that continuous fairness assessment and model retraining are imperative [6], [7].

Conclusion

In this research article, the authors propose a hybrid approach to sentiment analysis that makes use of Convolutional Neural Networks and BERT. Blending the local feature extraction capability of CNNs with BERT’s analysis enhances the performance metrics of the model in terms of accuracy, precision, and recall. With the ever-growing growth of e-commerce, this hybrid model and its applications in sentiment analysis will be in demand for those companies that wish to satisfy and meet the needs of their customer base. Future work will be directed towards enhancing the model’s performance in terms of cost and exploring other fields where Deeplearning system can be applied [1], [2], [5].

References

[1] Prabhu, R., & Nashappa, C. S. (2023). A dynamic weight function based BERT auto encoder for sentiment analysis. International Journal of Applied Science and Engineering, 21(1), 182-192. https://doi.org/10.6703/IJASE.202403_21(1).006 [2] Hourrane, O., Sane, M., & Ezzeddine, H. (2018). Sentiment analysis: An empirical study of machine learning techniques. International Journal of Engineering & Technology, 7(4), 5726-5731. [3] Loukili, M., Maanane, A., & Azzimani, M. (2023). Sentiment analysis of product reviews for e-commerce recommendation based on machine learning. International Journal of Advances in Soft Computing and its Applications, 15(1), 1-12. [4] Abighail, B. M., Sukhvinder, S., & Abiola, A. (2023). Sentiment analysis on e-commerce review. 8th International Conference on Computer Science and Computational Intelligence. Elsevier. [5] Anju, A., Thomas, M., & Santhosh, R. (2022). Sentiment analysis for e-commerce website. 10th IEEE International Conference on Emerging Trends in Engineering & Technology-Signal and Information Processing. [6] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171-4186). [7] Zhang, Y., Zhao, J., & LeCun, Y. (2018). Deep learning for sentiment analysis: A survey. ACM Computing Surveys, 54(5), 1-36. [8] Chen, Y., & Sun, H. (2020). Deep learning for sentiment analysis: A survey. IEEE Transactions on Neural Networks and Learning Systems, 31(9), 3107-3120. [9] Liu, B. (2012). Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers. [10] Alhazmi, A., & Alzahrani, A. (2022). Sentiment analysis on e-commerce reviews using deep learning. Journal of King Saud University-Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2021.10.002 [11] Hossain, M. S., & Muhammad, K. (2020). A survey of sentiment analysis methods for online reviews. Journal of Network and Computer Applications, 168, 102748. [12] Yadav, V., & Ranjan, P. (2019). A comprehensive survey on sentiment analysis for e-commerce reviews. International Journal of Information Management, 46, 130-148. [13] Kumar, A., & Sharma, P. (2021). Sentiment analysis of online reviews: A systematic literature review. Journal of Retailing and Consumer Services, 58, 102353. [14] Arora, A., & Bhattacharyya, D. (2020). Sentiment analysis in social media: A review of methods, techniques, and trends. Journal of King Saud University-Computer and Information Sciences, 32(1), 24-32.

Copyright

Copyright © 2024 Rupesh Mangalam, Sahil Sharma, Akash Mishra, Anuradha Devi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.