Emotion Detection for Hotel Industry Feedback System using Machine Learning

Abstract

This study investigates the extent to which hotel employees are conscious of their abilities to identify facial expressions and emotions conveyed by customers while interacting with those customers. Facial expressions and emotions may have significant implications when interpreted as a sign of satisfaction and the quality of the service that was provided. According to the findings, a sizeable portion of hotel staff members do not have an accurate understanding of their abilities to recognise facial expressions and emotions, and a large number of them have a tendency to exaggerate their capabilities. This study has significant ramifications in regard to employee efforts put into tasks (such as effort and concentration), employee self-development and training, and employee willingness to take risks in their respective service encounters.

Introduction

I. INTRODUCTION

Feedback is a vital part of the process of self-improvement in any service sector. This study computes the hotel and restaurant service sector. In the conventional model, feedback is collected by means of questionnaires (whether manual or digital) and through online screens or kiosks. The model displays various delicacy messages related to the customer's overall experience of the service.

This proposed application can be technologically advanced by incorporating digital video cameras to

capture the mood or emotion of the customers at various key strategic places within the vicinity of the particular hotel or restaurant outlet and by using artificial intelligence and machine learning to extract frames from the continuous video stream to record the exact emotion expressed by the customer with respect to and in reaction to the various services offered by the soliciting organization.

These frames, which include emotions displayed by the consumer, are passed through various established human emotion detection algorithms to extract the exact feeling felt by the customer in response to the efficiency of the service provided.

For instance, when a user passes in front of such a camera-and-display device, the device may display a message such as "Please rate the ambience of the hotel with respect to your experience" or "Please indicate through a gesture how you felt about the food". When the response is recorded, based on the intensity of the feeling in the user, the algorithm may classify the response as "excellent ", "outstanding" "happy", "not satisfactory, "etc.

This system thus delivers the customer feedback to the service-provider organization, which can then use the information to take corrective measures to improve the quality of service to the end customer.

II. LITERATURE SURVEY

1. A model for the analysis of sentiment based on images is proposed by this system. In order to predict the emotions conveyed by pictures, three pretrained models (VGG-19, DenseNet121, and ResNet50V2) are compared and contrasted. When the initial layers of the model are frozen, the performance of the model is improved, while other parts of the model are left unfrozen. The overfitting effect has been reduced thanks to the addition of layers such as dropout, batch normalisation, and weight regularisation layers, which have in turn helped to predict sentiments taken from the database. When performing image sentiment analysis with the DenseNet121 model, an accuracy of 0.89 was achieved.
2. This research work uses social media platforms, which are the easiest way to express emotions pertaining to any occasion, and the easiest way to express their feelings through social media is by uploading pictures. Based on the emotions in the picture, they are classified as positive, negative, and neutral. This classification is used in the automatic tagging available on social media based on the available image, through which the emotions and sentiments of a person can be found by natural language processing (NLP). Two pretrained models and one custom model are used for comparison and to predict the emotions. Better results are shown with a custom model and by using a web camera.

3. This research study is based on the facial expression of the person; there emotions and sentiment are found, which is conducted with the available database using Matlab. Gaussian filtering is used to remove the noise in the image, canny is used to find out heavy edges, and geometric ratios remove errors, which increases the performance of image processing. The expressions such as happy, sad, angry, confused, etc. are classified based on sentiment recognition.

4. In this study, convolutional neural networks are used to determine different emotions like fear, happiness, anger, sadness, surprise, etc. This type of classification helps in applications where an automatic tag predictor is used, usually in social media, where the sentiment of a person is understood based on their emotions. Two pretrained models are compared, and the accuracy of image sentiment is determined. First, a sentiment is determined, and then emotions are predicted, which describe happy, sad, fear, surprise, no emotion, etc.

5. This research intends to improve the signal-to-noise ratio of seismic data by employing median filtering and mean filtering, both of which have previously resulted in high signal-to-noise ratios and putting forward a convolutional neural network noise reduction framework. This article presents a deep learning convolutional neural network that achieves high performance by employing fault noise suppression as its primary training method. The input layer, the convolution layer, the activation layer, the normalisation layer, and the output layer are all components of the network. The utilisation of residual learning, the RELU function, and batch normalisation has resulted in an improvement in the accuracy of the FD-Net model, as well as the preservation of the original datasets and a high signal-to-noise ratio, and it keeps the fault information that is associated with the image.

6. This project reports on a sewing defect detection method that is based on CNN with a per-trained VGG-16. The goal of this method is to find broken stitches in a picture of a sewing operation. The set of rotated, normal, and sewing images were used to arrive at a conclusion regarding the efficacy, which was found to be 92.3%. Computing devices and deep learning libraries were investigated so that the amount of time spent computing could be reduced. The outcome of the test demonstrates that the manufacturing technology is useful to produce garments.

7. The primary objective of this work is to develop an alternative method to the traditional pooling method, which is represented here by the combpool layer, which has three different architectures. It has been observed that the combination of calculating the arithmetic mean and the maximum value produces more accurate results. The integration of Sugeno into the integral combination led to an improvement in performance. When used in conjunction with other functions, the DV Sugeno integral produces satisfactory results. Since the generalisation of the Sugeno integral produced positive results, the performance of models that include the Choquet integral and generalisations as a combinatorial layer was tested. Combpool is also utilised in contemporary architectural designs.

8. This work presents the detection of food spoilage from the production stage to the consumption stage. The freshness of fruit is detected by a computer vision-based technique that uses deep learning with CNN. Later, the model is analysed with the available dataset of fresh and rotten fruit images from Kaggle.

9. This paper presents a larger amount of analysis on facial responses. Where in eyebrow raises, sad, happy, disgusting, positive, and negative expressions were measured frame by frame and mapped with ad liking and brand purchase, and effectiveness was measured. A model has been built to predict the emotional responses, and the results showed high liking scores. And also, one more model has been created for predicting changes in intent based on automatic facial responses; the results show good effectiveness of the model. The work tested in this paper is for short-term purchasing decisions like chocolate. This approach can be combined with content analysis for audio and visual ads.

10. This study demonstrates a few difficulties associated with sentiment analysis. The approaches that are used can be broken down into three categories: lexicon-based, machine learning, and hybrid. Techniques for conducting supervised, unsupervised, and deep forms of machine learning are the three categories into which supervised machine learning falls. The posts and reviews on social networks, which take the form of textual data, are analysed during the decision-making process based on the individuals' emotional behaviour, and the results of this analysis are classified as pre-processing, feature extraction, and classification or clustering. Comparing the results of using all three methods for sentiment analysis reveals that the results have a high rate of accuracy.

11. This study demonstrates that the functionality of image pre-processing and augmentation techniques that are already present in HistoClean can be improved in the field of deep learning by utilising CNN. In order to implement image preprocessing techniques, the open-source software HistoClean is utilised. As a result, time is saved, and both transparency and data integrity are maintained.

HistoClean gives users a quick, powerful, and reproducible experience without requiring them to know how to code. In addition to this, it reduces the amount of time spent running and rerunning scripts. The programme was designed to simplify both the steps in the procedures and the steps themselves, making it as user-friendly as possible.

12. In this study, a comparison of different sentiment analysis classifiers is proposed using three distinct methods: machine learning, deep learning, and an evolutionary strategy referred to as EvoMSA. The comparison is carried out by developing two different corpora of expressions within the programming domain. This determines the emotional state of the student in relation to teachers, exams, homework, and academic projects. A corpus known as sentiTEXT contains polarity, that is, both positive and negative labels, whereas a corpus known as eduSERE contains both positive and negative learning-centred emotions, including labels such as happy, sad, exhausted, excited, bored, and frustrated. After examining the three approaches, it was determined that the evolutionary algorithm (EvoMSA) produced the best results for the corpus sentiTEXT dataset, which had an accuracy of 93%, and the corpus eduSERE dataset, which had an accuracy of 84%.

13. This work presents a comprehensive study on various methods of face recognition that are based on deep learning. Deep learning techniques are completely applied to face recognition, and they have played a vital role in avoiding challenges in FR, pose variations, illumination, age, facial expression, and heterogeneous face matching. Deep learning techniques are completely applied to face recognition. The use of methods based on deep learning has led to improved results in the processing of RGB-D, video, and heterogeneous face data.

14. The primary objective of this research is to on this project, a method is developed for implementing a fully connected deep neural network and a convolutional neural network inference system on a field-programmable gate array (FPGA). CNN makes use of both the systolic array architecture present and the parallel processing potential. The minimum amount of memory necessary for algorithmic analysis is required for fixed-point trained parameters. According to the findings, selecting block memory over distributed memory saves approximately 62% of the lookup tables for the DNN, while selecting distributed memory over block memory saves approximately 30% of the BRAM for the LeNet-5 CNN unit. This study provides some insight into the process of developing digital systems based on FPGAs for applications that require DNN and CNN.

15. Sentiment analysis makes use of a wide variety of machine learning and depth learning methods, such as SVM, NB, Haar cascade, LBPH, CNN, and so on. This allows guests in hotels to express their opinion on the food, ambience, and other aspects of the experience, which can be a helpful resource to obtain feedback from customers. The proposed work determines sentiment analysis pictures pertaining to users along with their faces from the hotel reviews, revealing that it is highly efficient in locating and categorising emotions from images.

16. This work discusses the analysis of textual sentiment in addition to the analysis of visual sentiment for the image that was provided, classifying the image as either positive, negative, or neutral. Utilising deep learning techniques such as CNN, visual sentiment analysis can be recast as image classification. However, the mood conveyed by the captured image is influenced by three different factors: the image factor, the user factor, and the item factor. both focused on the products and the people using them. CNN was designed so that image characteristics can be swapped out for different user expressions at any given time. The findings indicate that pictures taken in restaurants were more successful than others in categorising respondents' feelings.

17. This paper makes the argument that sentiment analysis should be classified according to facial expressions and visual analysis to determine whether an image is optimistic or pessimistic based on the emotions conveyed by it. CNN is used to construct a model for the prediction of the sentiment of images. Additionally, the opinions are categorised effectively, which increases the accuracy of the hotel picture that is posted on social media. When compared to naive bayes, the results produce the best performance analysis of reviews from images when using the ML technique.

18. The concepts of image and sentiment are Adjective noun pairs, also known as automatically finding the tags of a picture, are beneficial in terms of locating the emotions. Since deep learning techniques are able to effectively learn the polarity of images, they are used for sentiment analysis. This allows for more accurate results. Deep neural networks, convolutional neural networks, region-based convolutional neural networks, and fast recurrent neural networks are all proposed in this paper with the intention of being applicable to specific applications in image sentiment analysis.

19. The purpose of this research is to determine whether CNN can accurately forecast a range of feelings, including happiness, surprise, sadness, fear, excitement, exhaustion, and neutrality. There are a number of applications that are capable of automatically tagging predictions for visual data on social media and for comprehending feelings. They used the VGG-16 model, the ResNet model, and a customised CNN framework with seven layers in order to be able to predict the emotion and sentiment of the image that was given to them. The prediction was both more accurate and more efficient as a result of using CNN.

20. The trained model was used to find the emotion in a live photo by loading a captured image using a GUI.

21.  This project discusses sentiment analysis using facial recognition, utilising the Viola-Jones algorithm to find an image and local binary pattern for expression recognition. The classification of expression is accomplished with the help of support vector machines. You can tell from a person's expression whether they are happy, sad, or excited just by looking at their face. This framework initially locates and reads the face of a person; then, after finding the face, it computes a variety of facial expressions; finally, after finding the expressions, it categorises them as happy, sad, exhausted, and so on.

III. COMPARISON TABLE

Conclusion

This paper presents a solution to getting proper feedback from the customers, which helps in the proper development of the business. Here, the issue is taking reviews or feedback from the customers who visit the place (hotel). Those who are interested in giving feedback regularly do so, but a major percentage of people think it is a waste of time, so they will not give the review or feedback. This project helps us get proper feedback from the customers. In simple terms, we have developed a technique in which customers need not spend more time holding a pen and paper and filling out forms according to their wishes. We just need a small glimpse of an image at which the customer will look, and the camera automatically captures the expression and updates it in the database accordingly. Meaning the question about the particular property, e.g., service, cost, food, is taken and analysed using the technology, then automatically updated in the database as good, excellent, or bad based on the expression the customer gives for the question asked by our server.

References

[1] Chandrasekaran G, Antoanela N, Andrei G, Monica C, & HemanthJ. “Visual Sentiment Analysis Using Deep Learning Models with Social Media Data”Applied Sciences(Switzerland) -2022. [2] Swapnil Sanjay Nagvenkar, Ruchika Prakash Nagekar, Tanmay Naik, Amey Kerkar, “Emotion detection based on facial expression” (IJCRT)-2022. [3] Aynur Sevinç, Buket Kaya, Ayten Geçmez, “A Sentiment Analysis Study on Recognition of Facial Expressions: Gauss and Canny Methods” (IEEEI)-2020. [4] Doshi U, Barot V, & Gavhane S, “Emotion detection and sentiment analysis of static images” (IEEE)-2020. [5] Gherkar Y, Gujar P, Gaziyani A, & Kadu S, “Sentiment Analysis of Images using Machine Learning Techniques Yash” (ICCDW)-(2022). [6] Kim H, Jung W K, Park Y C, Lee J W, & Ahn S H, “Broken stitch detection method for sewing operation using CNN feature map and image-processing techniques. Expert Systems with Applications” (IEEE)-2021. [7] Rodriguez Martinez I, Lafuente J, Santiago R H N, Dimuro G P, Herrera F, & Bustince H,”Replacing pooling functions in Convolutional Neural Networks by linear combinations of increasing functions Neural Networks”. (IEEE)2022. [8] Valentino F, Cenggoro T W, & Pardamean B, “A Design of Deep Learning Experimentation for Fruit Freshness Detection. IOP Conference Series: Earth and Environmental Science” (IEEE)-2021. [9] D McDuff, R El Kaliouby, J F Cohn and P Picard, Predicting ad liking and purcha se intent: Large-scale analysis of facial responses to ads” (IEEE)- 2015. [10] Shaha Al-Otaibi and Amal Al-Rasheed, “A Review and Comparative Analysis of Sentiment Analysis Techniques” (Informatica)-2022. [11] Kris DnMcCombe, Stephanie G Craig, Amélie Viratham Pulsawatdi, Javier I Quezada-Marín, “Open- source software for histological image pre-processing and augmentation to improve development of robust convolutional neural networks” (IEEE)-2021. [12] Estradam MLB, Cabada R Z, Bustillos R O & Graff M,“Opinion mining and emotion recognition applied to learning environments”(Expert Systems with Applications)-2020. [13] Guo G, & Zhang N, “A survey on deep learning-based face recognition”. (Computer Vision and Image Understanding)-2019. [14] Mukhopadhyay A K, Majumder S, & Chakrabarti I. “Systematic realization of a fully connected deep and convolutional neural network architecture on a field programmable gate array” (IEEE)-2022. [15] Yash Gherkar, Parth Gujar, Siddhi Kadu, “Sentiment Analysis of Images using Machine Learning Techniques” (itmconf)-2022. [16] QuocTuan, Truong, HadyW, Lauw, “Visual Sentiment Analysis for Review Images with Item Oriented and User-Oriented CNN”(Corpus)-2022. [17] S Shivakumar,V R sanjay, “Visual Sentiment Classification of Restaurant Review Images using Deep Convolutional Neural Networks” (Corpus)-2022. [18] Namita Mittal, Divya Sharma, M. Joshi, \"Image Sentiment Analysis Using Deep Learning\" (Corpus) -2018. [19] Udit Doshi, Vaibhav Barot, Sachin Gavhane, \"Emotion Detection and Sentiment\" (IJESC)-2017. [20] R.N Devendra Kumar, Dr. C. Arvind, \"Facial Expression Recognition System \"Sentiment Analysis\" (Jour of Adv Research in Dynamical & Control Systems)-2017. [21] Chandrappa S, Dharmanna L, Basavaraj Anami, \" A Novel Approach for Early Detection of Neovascular Glaucoma Using Fractal Geometry\", International Journal of Image, Graphics and Signal Processing (IJIGSP), Vol.14, No.1, pp. 26- 39, 2022.DOI: 10.5815/ijigsp.2022.01.03 [22] Dharmanna L, Chandrappa S, T. C. Manjunath, Pavithra G,\"A Novel Approach for Diagnosis of Glaucoma through Optic Nerve Head (ONH) Analysis using Fractal Dimension Technique\", International Journal of Modern Education and Computer Science (IJMECS), Vol.8, No.1, pp.55-61, 2016.DOI: 10.5815/ijmecs.2016.01.08 [23] Guru Prasad, M.S., Naveen Kumar, H.N., Raju, K. et al. Glaucoma Detection Using Clustering and Segmentation of the Optic Disc Region from Retinal Fundus Images. SN COMPUT. SCI. 4, 192 (2023). https://doi.org/10.1007/s42979-022-01592-1 [24] Chandrappa S, Guruprasad, M.S., Kumar, H.N.N. et al. An IOT-Based Automotive and Intelligent Toll Gate Using RFID. SN COMPUT. SCI. 4, 154 (2023). https://doi.org/10.1007/s42979-022-01569-0 [25] Chandrappa S, Dharmanna L and K. I. R.Neetha, \"Automatic Elimination of Noises and Enhancement of Medical Eye Images through Image Processing Techniques for better glaucoma diagnosis,\" 2019 1st International Conference on Advances in Information Technology (ICAIT), Chikmagalur, India, 2019,pp.551-557,doi: 10.1109/ICAIT47043.2019.8987312.\\ [26] Chandrappa S, Dharmanna L, Shyama Srivatsa Bhatta U V, Sudeeksha Chiploonkar M, Suraksha M N, Thrupthi S, \"Design and Development of IoT Device to Measure Quality of Water\", International Journal of Modern Computer Science (IJMECS), Vol.9,No.4,pp.50-56,2017.DOI: 10.5815/ijmecs.2017.04.06 [27] Chandrappa S, Dharmanna Lamani, Shubhada Vital Poojary, Meghana N U,\"Automatic Control of Railway Gates and Destination Notification System using Internet of Things (IoT)\", International Journal of Education and Management Engineering (IJEME), Vol.7, No.5, pp.45-55, 2017.DOI: 10.5815/ijeme.2017.05.05

Copyright

Copyright © 2023 Mr. Chandrappa S, Meghana G, Bhoomika H J, Pooja P, Sindhu K. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.