Survey on Facial Emotion Recognition using Deep Learning

Abstract

The facial emotion recognition that extracts the expression of human which conveys a lot of information visually. This plays a crucial role in the human-machine interaction with which other applications are implemented in several fields of technology. Facial emotion recognition is performed in four stages consisting of pre-processing, face detection, feature extraction and expression classification. This concept includes various deep learning methods (convolutional neural network) to identify the seven key emotions namely: happy, sad, angry, fear, surprise, disgust and neutral. Recognising the facial expression by computer with high recognition rate is still a challenging job in this current industry. Some of the applications using these recognition systems are beyond the implementation stage that may include human behaviour understanding, detection of mental disorders, and synthetic human expressions. This research uses FER2013 dataset which helps to clearly classify the seven expressions that is proven to be effective for human emotion recognition.

Introduction

I. INTRODUCTION

In the modern technology our desires went huge and it has no bounds. The research mainly going on through the digital image and image processing fields. The way of progress and implementation is drastically increasing exponentially. The processing of image is a vast area of research now-a-days along with its applications.

Image processing helps to output the expression of image along with the specification of emotion that the image considers as input. Our emotion is revealed by the expressions in our face. Facial Expressions plays an important role in analysing the application or situation. Facial expression is a non-verbal scientific gesture which gets expressed in our face as per our emotions. Automatic recognition of facial expression plays an important role in artificial intelligence and robotics and thus it is a need of the generation. Some application related to this includes security, law enforcement and many more for tracking purpose.

This project mainly works to develop the facial emotion recognition automatically using the considered emotions from the dataset that is implemented into the input and classified it into seven different expression classes such as:

1. Neutral
2. Angry
3. Disgust
4. Fear
5. Happy
6. Sadness
7. Surprise  

Support Vector Machine (SVM) is a supervised classification algorithm where a line is drawn between two different categories to differentiate between them. This model is a powerful tool to identify predictive model or classifiers. Existing system shows accuracy more than 90% when SVM (Support Vector Machine) classifier is used.

The technologies used for human computer interaction are increasing day by day. Facial emotion recognition is one among the trending technologies that are implemented here. Facial expressions are fastest means of communication while conveying any type of information. Recently, devices interact with human along with the performance of coding. But, if machine can extract real time data of the human it is interacting with, it would help to improvise human machine interaction. FER, if implemented, may lead to a natural Human Machine Interface (HMI).

Also, Human emotions helps in the implementation of:

• Training AI agents
• To make better decisions for employees.
• To improve focus and performance.

II. LITERATURE REVIEW

Harisu Abdullahi Shehu at el. [1] Face expression recognition has become on of the important area of research in research years. Some of the miscellaneous regularization methods have been utilized in order help combat overfitting, to reduce training time and to generalize their model. Haar-cascade classifier is feature-based object detection algorithm to detect objects from images. This Haar-cascade classifier is used to crop the faces and focuses on region of interest, it hypothesizes that it would attain a fast convergence without using the whole image to analyse facial expressions. HC is used to identify the faces in the input image or the images are analysed through the dataset. The detected faces are cropped and resized into particular pixel i.e., 128x128 pixels. The cropped images or faces are then fed into the model for training and emotion analysis of input image. The labels are given to the image and analyse its effect on databases. Facial emotion recognition is a trending area of research that has gained attention in the past few decades due to increase in the usage of cameras.   

Noel Jaymon at el.[2] This paper has proposed methodology which can be split into two stages i.e., Model Training and Real time detection. The dataset is imported into pandas data-frame using pandas library through which all pre-processing open was conducted. The image representation is done through the pixel value. The images and also live video inputs are adapted for input which is saved, and the input is converted into grayscale images, that helps to fasten the process.

K.S Chandraprabha at el.[3] Real time Employee Emotion Detection System (RtEED) which uses machine learning to detect the emotions of the employees, it is a real-time detection using image processing with human-friendly machine interaction. RtEED system helps the employer to depict the emotions and produce information on the personalities and thoughts of the employees. RtEED system is designed to identify the basic six emotions that are extracted from the dataset such as: happy, sad, surprise, anger, fear and disgust. These helps to output the accurate results.

Wafa Mellouka at el.[4] Facial expression is mainly implemented in several areas such as safety health and human machine interaction. Researchers in this area are interested to interpret different techniques, to code facial expressions and extract these features for having better predictions by computer as an end result. Different architectures are exploited in this field in order to examine the better model which is able to execute the accurate result. The researchers concentrated on automatic facial emotion recognition i.e., FER via deep learning for better performance of the considered architecture. The purpose of this is to serve and guide researchers by reviewing recent works and providing insights to make improvements.

         Table-1: Comparison of datasets

Nithya Roopa S at el. [5] Facial emotion recognition is a part of expression detection which is gaining lots of importance in this current world. The need for it and usage of facial expression have been tremendously increasing day by day. There may be many methods to identify expressions using machine learning and artificial intelligence techniques, these help in the classification of image which helps in recognition and classification of expressions according to the images that are given as input to the system. Various datasets are verified and investigated for training expression recognition models are explained in this paper. Inception Net is used for emotion recognition with Kaggle and Karolinska Directed Emotional Faces datasets.

Table-2: Summary of survey

Conclusion

Classification of human expressions along with static facial images using deep learning techniques which is considered as one of the difficult tasks to identify emotions on diversified face expression from different angles. This system works with live capturing of images and captured series of images with which the classification of expressions or emotions are neatly observed. SVM is a supervised classification algorithm where a line is drawn between two different categories to differentiate between them. SVM classifier used in the system gives the expected output properly. So, this paper concentrates on future learning, face and facial component detection and feature extraction with expression classification by using deep learning algorithm.

References

[1] M. Hossain and G. Muhammad, \"Emotion Recognition Using Deep Learning Approach from Audio-Visual Emotional Big Data,\" Information Fusion, vol. 49, pp. 69- 78, September 2019. [2] Nithya Roopa. S, \"Emotion Recognition From Facial Expression Using Deep Learning\", Vol.8, pp.91-95, August 2019. [3] Wafa Mellouka, Wahida Handouzia, \"Facial emotion recognition using deep learning\", pp.690-697, August 2020. [4] Md. Forhad Ali, Mehenag Khatun, Nakib Aman Turzo, \"Facial Emotion Detection Using Neural Network\", International Journal of Scientific & Engineering Research Volume 11ISSN 2229-5518, 2020. [5] Samadhi P. K. Wickrama Arachchilage and Ebroul Izquierdo, \"Deep-learned faces: a survey\", Arachchilage and Izquierdo EURASIP Journal on Image and Video Processing, PP.2-33, 2020 [6] Ninad Mehendale, ?Face Emotion Recognition Using Convolutional Neural Network?, Feb 2020 [7] Anna Pecchinenda, Vivian Ciaramitaro, Jun Moriya, Daniel Fitousi, \"Stereotypical Processing of Emotional Faces:Perceptual and Decisional Components\", Article in Frontiers in Psychology,Volume 12, Article 733432, pp. 1-25., October 2021 [8] Harisu Abdullahi Shehu, Md. Haidar Sharif and Sahin Uyaver, \"Facial Expression Recognition Using Deep Learning\", AIP Conference Proceedings 2334, 070003 (2021), 02 March 2021 [9] Md.Rabiul Islam, Md.Ali Moni, Md.Rashed-AI-Mahfuz,?Emotion Recognition From EEG Signal Focusing on Deep Learning and Shallow Learning Techniques?, June 2021. [10] Noel Jaymon, Sushma Nagdeote, Ayush Yadav, Ryan Rodrigues, ?Real Time Emotion Detection Using Deep Learning?, Feb 2021. [11] K.S Chandraprabha, A.N Shwetha, M.Kavitha, R.Sumathi, ?Real time-Employee Emotion Detection System (RtEED) Using Machine Learning? ,March 2021. [12] Rashid Jahangir, Ying Wah Teh, Faiqa Hanif, Ghulam Mujtaba,?Deep Learning approaches for Speech Emotion Recognition: State of the art and research challenges? July 2021. [13] Ke Zhang, Yuanqing Li, Jingyu Wang, Erik Cambria, “Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge? April 2021.

Copyright

Copyright © 2023 Rachana G Raikar, Sushma R. B, Rakshitha C, Shreegowri S, V Indushri. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.