Facial Landmark Detection with Sentiment Recognition

Abstract

Facial landmark detection is gaining more importance and popularity as its requirement and applications increase tremendously. Facial landmark detection has always been an exciting field of research cause of many potential applications encompassing facial recognition, sentiment recognition, and facial expression analysis. Innumerous datasets are extracted, cleaned, and explored for training Facial Landmark detection at various angles and setups, resulting in successful working in occult faces and multiple facial mapping in real-time. Facial Landmark detection, coupled with one of the robust applications of sentiment recognition, becomes an excellent tool for the user with the potential to add many applications further in the future.

Introduction

I. INTRODUCTION

The Facial Landmark Detection tracks 468 unique points on a human face to ensure its mapping and working accuracy. The pandemic has made all of us wonder a lot of things, including the question of what is the post-pandemic       world going to be like? Applications of facial landmark detection technology fall under the umbrella of two categories – 1) Facial Recognition and 2) Face Expression Analysis. In today’s world, where people are returning to stores, ATMs, shopping malls, and Airports, we need a robust Facial Landmark detection software to keep track of customers, identify shoplifters, and so much more. Almost everything is going back offline, and facial recognition can take care of security concerns. Facial Landmark detection, coupled with one of the most robust applications of sentiment recognition, becomes an excellent tool for the user to help determine a person's current mood and mental state. The potential to add a multitude of applications further in the future, from detecting a person's age to predicting any possible mental illness in a person based on micro expressions. The libraries used in making the projects are OpenCV, time, MediaPipe, and many more from Python and Machine Learning frameworks.

II. RELATED WORKS

During our literature review, we found various existing systems using different techniques and tools to serve somewhat the same purposes. We came across methods that can be classified into two main categories – 1) Generative methods and 2) Discriminative methods. The generative model attempts to fit a generative shape over the given facial structure over the 3-D space of the model's parameter. The discriminative method tries to search through possible candidates in predefined facial landmarks of the human face and then combines the result for the configuration of face landmarks. Another model named the Boosted Appearance model created by Liu works with a boosting-based classifier, which is better for optimizing appearances. Another model proposed by Liang suggested a component-based extension to this approach. Kim and Lee have used multi-linear analysis to have a solid fitting. Kanade and Gu have developed a three-layer generative model for landmarks and geometric transformation candidates. Discriminative models such as one proposed by Valtsar proposed Support Vector Regression to help search landmarks.

III. METHODOLOGY

We have used CNN technology to recognize and map facial landmarks and sentiment recognition.

Procedure

A. We saw different approaches to CNN-based Facial Landmark detection models. We concluded by choosing the heatmap approach to it from the possible options of Regression (Two types, including Direct and Cascade Regression). Pixel-Wise Classification model as well as Distribution models. We found that real-time heatmaps are the most effective models to train our CNN network for facial landmark detection.

The following Procedure explains the details of our Heatmap approach -

It detects the landmarks by mapping them landmarks on a 2-D heatmap. Our model of Heatmap and its structure is inspired by Fully Convolutional Network (FCN), which generates semantic features to set the Heatmap. We can divide heatmaps into categories based on heatmap characteristics and loss functions.

1. Distributed Model: It indicates the position of the landmark by using multivariate distribution. The Center of distribution is to be found at the coordinates of the landmarks. We used 2-D Gaussian distribution is utilized to figure out landmark coordinates.

B. Libraries and Packages

Following are the various python libraries and packages used to build the system:

1. Pandas: Pandas is a Data Sets library used in Python. It is used in dissecting, cleaning, investigating, and controlling data. It permits us to examine big data and make ends dependent on measurable speculations. It can clean muddled data sets and make them discernible and vital.
2. NumPy: Numpy Library is used to work with arrays in Python. It has the capability to work in the space of linear algebra, Fourier transform, and matrices. In Python, records fill the need for arrays, yet they are delayed to measure. NumPy is used to give Python records to an array object up to 50x quicker than normal speed.
3. Beautiful Soup: It is used to get data out of HTML and XML dialects. It assists you with pulling specific substances from a page, eliminating the HTML markup, and saving the data. Beautiful Soup is used for web scraping and allows you tidy up and parse the documents you have pulled down from the web.
4. OpenCV: The main aim of OpenCV is to make way for real-time computer vision to enable the user to track activities in video or image format in real-time.
5. Scikit Learn: Scikit Learn  highlights different arrangement, relapse, and grouping algorithms, including support vector machines, irregular woods, inclination boosting, k- means, and DBSCAN. It is intended to work with the Python mathematical and logical libraries such as NumPy and SciPy

IV. WORKING

The system involves techniques and steps, each equally important to get the desired output. The steps are elaborated as follows:

A. Modes

We have three different modes of working with the program. The program will run the Facial Landmark detection module in the first mode. In the second mode, the program will run the sentiment recognition module. In the third mode, all program functionalities will be visible and available to the user.

1. Mode 1: Facial Landmark Detection – In this mode, a camera window will pop up and scan for the human face. After detecting the human face, it will map the facial features and portray the landmarks on the face. It will smoothly follow the face wherever it displaces in the camera field and keep mapping and detecting the facial landmarks. It will even detect and map occult faces whenever the face is tilted away from the camera.

2. Mode 2: Sentiment Recognition – In the second mode of our program, the program will use landmarks to detect the user's sentiment based on their facial expression and face landmark positioning. The program to detect sentiments is based on CNN, which has been trained with human faces with different expressions.

3 . Mode 3: In this mode, we combine modes 1 and 2 to create a window that will show the facial landmarks as well as the sentiments of the user on adjacent screens on the same window.

V. LIST OF FEATURES

Following is a list of concrete features that the project offers to the users:

1. Face-mapping - This feature allows us to capture and detect the face on camera.
2. Face-Mesh – Our project creates a face mesh to keep track of the facial image.
3. Face Landmark – Tracks and Detects Facial landmarks using 468 unique coordinates on a human face.
4. Real-time – Every feature mentioned above happens in real-time, making our project practical and efficient.
5.  Occult Faces – Even if a person is facing sideways, our program can track and map the facial landmarks on the face without difficulty.
6. Multiple Face Detection – The program can detect multiple human faces in real-time, making it another practical project application.
7. Smooth face tracking – As the face moves in real-time, it can easily keep up and smoothly translate all the landmarks.
8. Sentiment Recognition – The program can detect a person’s sentiments based on their face expression analysis.

Conclusion

This paper addresses and implements a valuable method of Face Landmark Detection. We have tried to use this method to make a Face Landmark Detection system; with Python & Deep Learning, we can recognize the Landmarks of Someone\'s Face, which can be used in face recognition. This system can be beneficial in creating a base for Face Recognition Software which has multiple benefits of its own. Face Landmark Detection has numerous benefits and is one of the main underlying concepts behind face recognition. It can be used for Security Purposes, Face Recognition Attendance systems, Filter & Beauty Enhancement, Emotion Detection Systems, and Depression & Disease Detection systems.

References

[1] Kostiantyn Khabarlak and Larysa Koriashkina, “Fast Facial Landmark Detection and Applications: A Survey” - https://arxiv.org/abs/2101.10808. [2] How To Apply Machine Learning For Face Detection [3] Face Detection by Sukhmeet Brar. [4] Handbook of Face Recognition by Anil Kumar Jain, Stan Z. Li. [5] Our Biometric Future: Facial Recognition Technology and the Culture of Surveillance by Kelly Gates. [6] Advances in Face Detection and Facial Image Analysis by B. Smolka, Michal Kawulok, M. Emre Celebi. [7] Deep Learning Book by Aaron Courville, Ian Goodfellow, and Yoshua Bengio [8] Pattern Recognition and Machine Learning Book by Christopher Bishop. [9] https://www.youtube.com/watch?v=GwIo3gDZCVQ

Copyright

Copyright © 2022 Divyeh Sharma, Vanshaj Jain, Sakshi Khandelwal, Abhishek Das. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.