Human Emotion Detection Using Deep Learning

Abstract

Human emotions are the mental state of feelings and are spontaneous. There is no clear connection between emotions and facial expressions and there is significant variability making facial recognition a challenging research area. Emotion recognition becomes a popular topic of deep learning and provides more application fields in our daily life. The primary idea of our project is to process the input images of human facial emotion to train the pre-trained models on datasets. In recent years, Machine Learning (ML) and Neural Networks (NNs) have been used for emotion recognition. In this project, a Convolutional Neural Network (CNN) is used to extract features from images to detect emotions. The numerical result of the algorithm will show a probabilistic result of each labeled class.

Introduction

I. INTRODUCTION

Facial emotions are important factors in human communication that help to understand the intentions of others. In general, people infer the emotional state of other people, such as joy, sadness and anger, using facial expressions. Facial expressions are one of the main information channels in interpersonal communication. Therefore, it is natural that facial emotion research has gained a lot of attention over the past decade with applications in perceptual and cognitive sciences . Interest in automatic Facial Emotion Recognition (FER) has also been increasing recently with the rapid development of Artificial Intelligent (AI) techniques. They are now used in many applications and their exposure to humans is increasing. To improve Human Computer Interaction (HCI) and make it more natural, machines must be provided with the capability to understand the surrounding environment, especially the intentions of humans. Machines can capture their environment state through cameras and sensors. In recent years, Deep Learning (DL) algorithms have proven to be very successful in capturing environment states. Emotion  detection is necessary for machines to better serve their purpose since they deliver information about the inner state of humans. A machine can use a sequence of facial images with DL techniques to determine human emotions.

II. LITERATURE SURVEY

1. Facial Emotion Detection Using Deep Learning

Arpankumar Patel, Ansel Joseph, ShubhamSurvase, Rohini Nair

Abstract: Human Emotion detection from image is one of the most powerful and challenging research task in social communication. Deep learning (DL) based emotion detection gives performance better than traditional methods with image processing. This paper presents the design of an artificial intelligence (AI) system capable of emotion detection through facial expressions. It discusses about the procedure of emotion detection, which includes basically three main steps: face detection, features extraction, and emotion classification. This paper proposed a convolutional neural networks (CNN) based deep learning architecture for emotion detection from images. The performance of the proposed method is evaluated using two datasets Facial emotion recognition challenge (FERC-2013) and Japaness female facial emotion (JAFFE). The accuracies achieved with proposed model are 70.14 and 98.65 percentage for FERC2013 and JAFFE datasets respectively.

2. A Lightweight Convolutional Neural Network For Real-Time Facial Expression Detection

Abstract: In this paper our group proposes and designs a lightweight convolutional neural network (CNN) for detecting facial emotions in real-time and in bulk to achieve a better classification effect. We verify whether our model is effective by creating a real-time vision system. This system employs multi-task cascaded convolutional networks (MTCNN) to complete face detection and transmit the obtained face coordinates to the facial emotions classification model we designed firstly. Then it accomplishes the task of emotion classification.

Multi-task cascaded convolutional networks have a cascade detection feature, one of which can be used alone, thereby reducing the occupation of memory resources. Our expression classification model employs Global Average Pooling to replace the fully connected layer in the traditional deep convolution neural network model. Each channel of the feature map is associated with the corresponding category, eliminating the black box characteristics of the fully connected layer to a certain extent.

At the same time, our model marries the residual modules and depth-wise separable convolutions, reducing large quantities of parameters and making the model more portable. Finally, our model is tested on the FER-2013 dataset. It only takes 3.1% of the 16GB memory, that is, only 0.496GB memory is needed to complete the task of classifying facial expressions. Not only can our model be stored in an 872.9 kilobytes file, but also its accuracy has reached 67% on the FER-2013 dataset. And it has good detection and recognition effects on those figures which are out of the dataset

III. OBJECTIVES

 In this project, we use a deep learning technique called Convolutional Neural Networks(CNNs) to establish a classification model that combines feature extraction with classification to detect the facial emotions. To develop a facial expression recognition system. To experiment machine learning algorithm in computer vision fields. To detect emotion thus facilitating Intelligent Human-Computer Interaction.

IV. METHODOLOGY

1. Problem Statement
2. Proposed Systems

With the goal to improve the process of facial sentiment analysis systems, a classification mechanism is proposed using a CNN architecture. The goals of this project are to establish a model that can classify 7 basic emotions: happy, sad, surprise, angry, disgust, neutral, and fear..

3. Advantage of Proposed System

a. High picture quality improves the effectiveness  of  facial recognition

b. Even low resolution photographs are usable with the suggested method

c. Higher accuracy while being more computationally efficient.

V. MODULES

A. Face Capturing Module

Image Acquisition: Images used for facial expression recognition are static images or image sequences. Images of face can be captured using camera. Face detection: Face Detection is useful in detection of facial image. Face Detection is carried out in training dataset using Haar classifier called Voila-Jones face detector and implemented through Opencv. Haar like features encodes the difference in average intensity in different parts of the image and consists of black and white connected rectangles in which the value of the feature is the difference of sum of pixel values in black and white regions.

B. Pre-Processing Module

Image pre-processing includes the removal of noise and normalization against the variation of pixel position or brightness.

1. Color Normalization
2. Histogram Normalization

The amount of training data directly affects the final recognition effect. The larger the amount of data, the better the effect.

C. Training Module                                                                                                                               

Deep learning training is when a deep neural network (DNN) “learns” how to analyze a predetermined set of data and make predictions about what it means. It involves a lot of trial and error until the network is able to accurately draw conclusions based on the desired outcome.

D. Face Recognition Module

A Convolutional neural network (CNN)is a type of artificial neural network that has one or more convolution layers and are used mainly for image processing, classification, segmentation and also for other auto correlated data. Deep learning is a machine learning based artificial neural network that recognize objects in image by progressively extracting features from data through higher layers.As shown in figure in order to recognize face in an image we have to train the CNN with human faces.

The benefit of using CNNs is their ability to develop an internal representation of a two- dimensional image. This allows the model to learn position and scale of faces in an image. After train the CNN it can able to recognize face in an image One can effectively use Convolutional Neural Network for Image data. CNN that extracts features in an image.

E. Expression Detection  Module

General expression recognition methods include image reprocessing, facial feature extraction, and expression recognition. The reprocessing stage of image expression recognition performs face detection to obtain facial region images. The recognition of expressions in low-pixel facial images also requires image enhancement or image super resolution during reprocessing. Image enhancement is to enhance the existing information of the image information by adding pixel. We use CNN for image recognition

VI. FLOWDIAGRAM

VII. BLOCKDIAGRAM

VIII. ALGORITHM USED

Convolutional Neural Networks (CNN, or ConvNet) are a type of multi-layer neural network that is meant to discern visual patterns from pixel images. In CNN, ‘convolution’ is referred to as the mathematical function. It’s a type of linear operation in which you can multiply two functions to create a third function that expresses how one function’s shape can be changed by the other. In simple terms, two images that are represented in the form of two matrices, are multiplied to provide an output that is used to extract information from the image. CNN is similar to other neural networks, but because they use a sequence of convolutional layers, they add a layer of complexity to the equation. CNN cannot function without convolutional layers. In a variety of computer vision tasks, CNN artificial neural networks have risen to the top. It has picked people’s interest in a variety of fields. A convolutional neural network is made up of numerous layers, such as convolution layers, pooling layers, and fully connected layers, and it uses a back propagation algorithm to learn spatial hierarchies of data automatically and adaptively. You will learn more about these terms in the following section

Conclusion

This project proposes an approach for recognizing the category of facial expressions. Face Detection and Extraction of expressions from facial images is useful in many applications, such as robotics vision, video surveillance, digital cameras, security and human-computer interaction. This project’s objective was to develop a facial expression recognition system implementing the computer visions and enhancing the advanced feature extraction and classification in face expression recognition. In this project, seven different facial expressions of different persons’ images from different datasets have been analyzed. This project involves facial expression preprocessing of captured facial images followed by feature extraction using feature extraction using Local Binary Patterns and classification of facial expressions based on training of datasets of facial images based on CNN.

References

[1] Chu, William Wei-Jen Tsai, Hui-Chuan, YuhMin Chen and Min-Ju Liao. Facial expression recognition with transition detection for students with high-functioning autism in adaptive e- learning.” Soft Computing: 1-27, 2017. [2] Tzirakis, George Trigeorgis, Mihalis A. Nicolaou, Panagiotis, Bjrn W. Schuller, and Stefanos Zafeiriou. ”End-to-end multi-modal emotion recousing neural networks.” IEEE Journal of Topics in Signal Processing 11, no. 8: 13011309, 2017 [3] Gampala, V., Kumar, M.S., Sushama, C. and Raj, E.F.I., 2020. Deep learning based image processing approaches for image deblurring. Materials Today: Proceedings. [4] Aneja, Gary Faigin, Deepali, Alex Colburn, Barbara Mones, and Linda Shapiro. ”Modeling stylized character expressions via deep learning.”In Asian Conference on Computer Vision, pp. 136-153. Springer, 2016 [5] Natarajan, V.A., Kumar, M.S., Patan, R., Kallam, S. and Mohamed, M.Y.N., 2020, September. Segmentation of Nuclei in Histopathology images using Fully Convolutional Deep Neural Architecture. In 2020 International Conference on Computing and Information Technology (ICCIT1441) [6] J. Nicholson, K. Takahashi, and R. Nakatsu. Emotion recognition in speech using neural networks. Neural computing applications, 9(4): 290-296, 2000. [7] B. Fasel and J. Luettin. Automatic facial expression analysis: a survey. Pattern recognition, 36(1):259-275, 2003. [8] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images, 2009. [9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009. [10] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural infor- mation processing systems, pages 1097-1105, 2012. [11] Y. Lv, Z. Feng, and C. Xu. Facial expression recognition via deep learning. In Smart Computing (SMARTCOMP), 2014 International Conference on, pages 303-308. IEEE, 2014. [12] TFlearn. Tflearn: Deep learning library featuring a higher-level api for tensorflow. URL http://tflearn.org/. [13] Open Source Computer Vision Face detection using haar cascades. URL:http://docs.opencv.org/master/d7/d8b/tutorialpyfacedetection.html

Copyright

Copyright © 2023 Mr. Pandarinathan, S. Velan, S. Deepak. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.