Music Recommendation System based on User’s Facial Expression

Abstract

Music is a fantastic way for people to express themselves as well as a good source of enjoyment for music lovers and listeners. Besides, relaxing music is an effective manner for eliciting strong feelings and starting a quiet communication. With technological advancements, the number of artists, their music, and music listeners is increasing, which brings up the consequence of manually searching and picking music. This study offers a system that uses facial expressions at real time of a user to assess the his/her mood( Emotion detection Model), product of which is also combined with mapped music from the music dataset to generate a user-specific music playlist( music recommendation model). Convolutional neural network is used to classify the users sentiments in 7 different kinds with an accurateness rate of 94 percent, therefore satisfying the effective aim of the study.

Introduction

I. INTRODUCTION

Music is always being known to change moods of the people. Capturing and recognizing emotions of the person and displaying relevant songs that matches the person’s mood can gradually calm their mind. It will end up giving a pleasing effect. This project helps us to capture the emotion that are expressed by a person through his/her facial expressions. A music player is designed in such a way that it captures human emotions with the help of the web camera interface that is available on almost every computing systems. The image of the user is captured by the computing systems and then with the help of the techniques like image segmentation and image processing, the features from the face of a target human being are extracted and the emotion that the person is trying to express is detected. The project also aims to refresh the mood of the user, with the help of songs that match the requirements of the user by capturing the image of the user and playing it. Even from the ancient times the best form of expression analysis that is known to humankind is facial expression recognition. The best possible way in which people tend to analyze or conclude the emotion or the feeling or the thoughts that another person is trying to express is by facial expression. In some of the cases, changed moods of people can also help in overcoming situations like sadness and depression. With the aid of expression analysis, many health risks can be avoided, and also there can be steps taken that help brings the mood of a user to a better stage.

Renuka R Londhe etal.[1] proposed a paper which fo- cused on the study ofchanges in the curves of the face and the intensities of the corresponding pixels. The author used Artificial Neural Networks( ANN), which was used to classify the emotions/sentiments. The author also proposed varied approaches for a playlist. Zheng etal.[2] proposed two significant classes for facial feature extraction, which included Appearance- based feature extraction and geometric predicated feature extraction, which included extraction of some essential points of the face similar as mouth, eyes, and eyebrows. Nikhil etal.[3]. determines the mindset of the user by using facial expression Humans frequently express their feeling by their expressions, hand gestures, and by raising the voice of tone but substantially humans express their emotions by their face. Emotion- based music player requires less time complexity of the user. Generally, large number of songs are already added to people’s playlist. Playing songs aimlessly doesn’t satisfy the mood of the user. This system allows the us to play songs according to our mood automatically by the expressions captured. The web camera captures the image of the user, and then the images are saved. Initially, the images are converted to binary format from RGB format. This process of representing the data is called a feature- point detection method. This process can also be done by using Haar Cascade technology handed by Open CV. The music player which is developed by using a java program manages the database. The song according to the mood of the user is played by the system.Z. Zeng etal.[5] delved varied advances in human affect recognition. He concentrated on varied approaches that can handle audio and/ or visual recordings of displays of affective countries. The paper provides a detailed review of  978-1-7281-4635-5/20/$31.00 ©2020 IEEE audio/ visual computing styles.

The effect is described as a prototype of emotion classes which include happiness, fear, sadness, anger, surprise and disgust. For the development of automatic, spontaneous affect recognizer, this paper mainly focused on agitating the challenges in computing styles, which helped in emotion detection. It also linked some problems that have been missed or avoided inunimodal posed emotion recognition. Parul Tambe etal.[7] proposed an idea which automated the relations between the users and music player, which learned all the preferences, feelings and activities of a user and gave song selection as a result. The varied facial expressions of users were recorded by the device to determine the emotion of the user to prognosticate the genre of the music. Jayshree Jha etal.[11]. proposed an emotion- based music player using image processing This showed how varied algorithms and techniques that were suggested by different authors in their research could be used for connecting the music player along with human feelings. It has therefore helped in reducing the efforts of user in creating and managing playlist and delivering an excellent experience to the music listeners by bringing them the most suitable song according to his/ her current expression. Anukritine etal.[18] came up with an algorithm that gives a list of songs from the user’s playlist in accordance with the user’s sentiment. The algorithm which was designed was focused on having lower computational time and also therefore reduces the cost included in using various hardware. The main idea was to separate the feelings into five divisions i.e., Joy, sad, angriness, surprise and fear also provided a highly accurate audio information retrieval approach that extracted applicable information from an audio signal in lesser time. Aditya etal.[19] developed an android application which acts as a customized music player for a stoner using image processing to dissect and present songs to stoner according to the user’s mood. The application was developed using Eclipse and also OpenCV to apply facial recognition algorithms. This paper also showed comparison between various algorithms applied in facial detection. The pictures of the user were captured using the front camera of the mobile device. It aimed to give satisfaction to music lovers by extracting their feelings.A. Habibzad etal.[20] proposed a new algorithm is proposed to recognize the facial expression, which included three stagespre-processing, feature extraction, and classification. The first part de- scribes various stages in image processing include preprocessing, filtering used to extract variegated facial features. The other part optimized the eye and lip ellipse characteristics, and in the third part, the eye and lip optimal parameters were used to classify the feelings. The attained results showed that the speed of facial recognition was far better than other usual approaches.Prof. Nutan Deshmukh etal.[21] concentrated on creating a system that fetches the emotion of the user using a camera and also automates the result using the emotion detection algorithm. This algorithm captures the mood of the user after every figured interval of time as the mood of the user may not be the same after some time; it may or may not alter. The proposed algorithm on an average calculated estimation takes around0.95-1.05 sec to induce an emotionbased music system, which was better than former existing algorithms and reduces the cost of designing. Chang Liuet al[24] described a system that makes use of Brain- Computer Interfaces, also called as BCI. BCI makes use of devices to transfer signals to the processing systems. The person’s cognitive state of mind is monitored by EEG hardware. The disadvantage of the scheme is that they need the input from the user’s brain continuously to perform the classification. An algorithm based on MID is used to continuously monitor and reprocess the signals received from the brain of the user and use these signals to actively cover and yield feelings that the user is right now experiencing. Swati Vaid etal[23] reviewed EEG- Electroencephalography( EEG) is a form of medical science that records the electrical activity from the neurons of brain cells. The electrical activity of the neurons from within the cells of the brain is registered. On the basis of the recorded activities of the neurons an approximation is made, and the emotion of the person is estimated from that analysis. This method mentioned above, although serves the purpose of getting the activity of brain cells but fails to serve the purpose of portability and economics.

II. EXISTING SYSTEM

In the existing system author have designed, implemented and analyzed a song recommendation system. Author used Million Song Dataset provided by Kaggle to find correlations between users and songs and to learn from the previous listening history of users to provide recommendations for songs which users would prefer to listen most.They have got best results for memory based collaborative filtering algorithm. They believe that content-based model would have worked bet- ter if we would have enough memory and computational power to use the whole available metadata and training dataset. t is the most basic and simple algorithm. They find the popularity of each song by looking into the training set and calculating the number of users who had listened to this song. Songs are then displayed in the sorted manner in descending order based on their popularity. For each user, They recommend top most popular songs except those already in his profile. This method is not customized and some songs may never be listened in future.

B. Emotion Detection Module

1. Face Detection: One of the applications which is considered under computer vision technology is Face detection. It is the process in which algorithms are developed and trained to properly locate faces or objects in object detection or related system in images. This detection can be real-time with the help of a video frame or images captured. Face detection uses such classifiers, which are algorithms that detect what’s either a face (1) or not a face (0) in an image. To detect faces using numbers of images classifiers are trained in order to get more accuracy. Two sorts of classifiers are used OpenCV, LBP (Local Binary Pattern) and Haar Cascades. For face detection where the classifier is trained with predefined varying face data A Haar classifier is used. It enables it to detect different faces accurately.

2. Feature Extraction: While performing feature extraction, we treat the pre-trained network that’s a succeeding model as an arbitrary feature extractor. Allowing the input image to pass on it forward, stopping at the pre-specified level, and taking the outputs of that level as our features. Starting layers of a convolutional network excerpt high- degree features from the taken image, so use only a countable filters. As we make more deeper layers, we increase the number of the filters to double or thrice the dimension of the filter of the former layer. Pollutants of the deeper layers gain further features but are computationally very deep. Doing this we used the robust, discriminational features learned by the Convolution neural network( 10). The outputs of the model are going to be feature maps, which are an intermediate representation for all layers after the actually first layer. Load the input image for which we want to view the point chart to know which features were prominent to classify the image. Feature map visualization will give perceptivity into the interior representations for specific input for each of the Convolutional layers within the model.

3. Emotion Detection: Convolution neural network framework applies filters or feature detectors to the input image to get the feature maps or activation maps using the Relu activation function( 11). Feature detectors or filters help in relating varied features pre- transferred in the image similar as edges, perpendicular lines, horizontal lines, bends, etc. For invariance to translation, later on pooling is applied over the feature maps. Pooling is prognosticated on the conception that formerly we change the input by a touch amount, the pooled outputs do n’t change. We can use any of the pooling from minimum, average, or maximum. But maximum- pooling provides better performance than minimum or average pooling. Flatten all the input and giving these smoothed inputs to a deep neural network which are outputs to the class of the object. The class of the image will be binary, or it’ll be a multi-class classification for identifying digits or separating varied vesture items. So mostly, we give an input image also the CNN model returns the results( 10). When we take the real- time image by a user also that image was transferred to the pre-trained CNN model, besides predicting the emotion and adding the tag to the image.

C. Music Recomendation Module

1. Song Database: We created a database for BTS English songs. It consists of 100 songs per emotion As we all know music is un-doubtedly involved in enhancing our mood. So, suppose a user is sad then the system will recommend such a music playlist which motivates him or her and by this automatic mood will be delighted.

IV. RESULTS AND DISCUSSIONS

After impementation our system will show the whole list of songs on the basis of result of emotion detection in percentage and it will definately delight the users mood . If the detected emotion is happy then it will suggest Happy song else suggesting song base on emotion of the user.

V. ACKNOWLEDGMENT

I would like to express my profound gratitude to Dr.Sanjay Shitole, Head Of Department of Information Technology de- partment, and Dr. Shikha Nema, Principal of UMIT for their contributions to the Approval of Our project titled Music Recommendation System Based on User’s Facial Expression. I would like to express my special thanks to our mentor Prof.Sujata Kullur for her time and efforts she provided throughout the semester. Your useful advice and suggestions were really helpful to us during the project’s completion. In this aspect, I am eternally grateful to you. I would like to acknowledge that this project was completed entirely by our group and not by someone else.

Conclusion

A detailed review of the literature tells that there are nu- merous ways to execute Music Recommender System. On the basis of findings, we fix the objectives of our system. As the power and advantages of AI- powered operations are trending, our design will be a state- of- the- art trending technology application. In this system, we deliver an overview of how music can affect the user’s mood and how to handpick the right music tracks to enrich the user’s moods. The user’s feelings can be detected by the enforced system. Happy, sad, neutral, surprised or angry were the emotion detected by our system. After determining the user’s emotion, the propounded system gave the user with a playlist that contains music matches that detected the mood. Our music recommendation system based on facial emotion recognition will lessen the efforts of users in the creation and management of playlists.

References

[1] Ramya Ramanathan, Radha Kumaran, Ram Rohan R, Rajat Gupta, and Vishalakshi Prabhu, an intelligent music player based on emotion recognition, 2nd IEEE International Conference on Computational Systems and Information Technology for Sustainable Solutions 2017. https://doi.org/10.1109/CSITSS.2017.8447743 [2] Shlok Gilda, Husain Zafar, Chintan Soni, Kshi- tija Waghurdekar, Smart music player integrating fa- cial emotion recognition and music mood recommenda- tion, Department of Computer Engineering, Pune Insti- tute of Computer Technology, Pune, India, (IEEE),2017. https://doi.org/10.1109/WiSPNET.2017.8299738 [3] Deger Ayata, Yusuf Yaslan, and Mustafa E. Ka- masak, Emotion-based music recommendation system us- ing wearable physiological sensors, IEEE transactions on consumer electronics, vol. 14, no. 8, May 2018. https://doi.org/10.1109/TCE.2018.2844736 [4] Ahlam Alrihail, Alaa Alsaedi, Kholood Albalawi, Liyakathunisa Syed, Music recommender system for users based on emotion detection through facial features, Depart- ment of Computer Science Taibah University, (DeSE), 2019. https://doi.org/10.1109/DeSE.2019.00188 [5] Research Prediction Competition, Challenges in represen- tation learning: facial expression recognition challenges, Learn facial expression from an image, (KAGGLE). [6] Preema J.S, Rajashree, Sahana M, Savitri H, Review on facial expression-based music player, International Journal of Engineering Research Technology (IJERT), ISSN-2278-0181, Volume 6, Issue 15, 2018. [7] AYUSH Guidel, Birat Sapkota, Krishna Sapkota, Music recommendation by facial analysis, February 17, 2020. [8] CH. sadhvika, Gutta.Abigna, P. Srinivas reddy, Emotion- based music recommendation system, Sreenidhi Institute of Science and Technology, Yamnampet, Hyderabad; Interna- tional Journal of Emerging Technologies and Innovative Re- search (JETIR) Volume 7, Issue 4, April 2020. [9] Vincent Tabora, Face detection using OpenCV with Haar Cascade Classifiers, Becominghuman.ai,2019. [10] Zhuwei Qin, Fuxun Yu, Chenchen Liu, Xiang Chen. How convolutional neural networks see the world - A survey of convolutional neural network visualization methods. Mathe- matical Foundations of Computing, May 2018. [11] Ahmed Hamdy AlDeeb, Emotion- Based Music Player Emotion Detection from Live Camera, ResearchGate, June 2019. [12] Frans Norden and Filip von Reis Marlevi, A Comparative Analysis of Machine Learning Algorithms in Binary Facial Expression Recognition, TRITA-EECS-EX 2019:143. [13] P. Singhal, P. Singh and A. Vidyarthi (2020) Interpretation and localization of Thorax diseases using DCNN in Chest X- Ray. Journal of Informatics Electrical and Elecrtonics Engi- neering,1(1), 1, 1-7 [14] M. Vinny, P. Singh (2020) Review on the Artificial Brain Technology: BlueBrain. Journal of Informatics Electrical and Electronics Engineering,1(1), 3, 1-11. [15] A. Singh and P. Singh (2021) Object Detection. Journal of Management and Service Science, 1(2), 3, pp. 1-20. [16] A. Singh, P. Singh (2020) Image Classification: A Survey. Journal of Informatics Electrical and Electronics Engineer- ing,1(2), 2, 1-9. [17] A. Singh and P. Singh (2021) License Plate Recognition. Journal of Management and Service Science, 1(2), 1, pp. 1-14

Copyright

Copyright © 2023 Vaishnavi Gaikar, Astha Dhengle, Namrata Mhatre, Prof. Sujata Kullur. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.