Stress Detection using Deep Learning Techniques

Abstract

Stress is a typical component of daily life that has an impact on people in different circumstances. However, sustained or acute stress can negatively impact our health and interfere with our daily activities. Early recognition of mental stress is essential for good management and the avoidance of future health problems brought on by chronic stress. Understanding the connection between facial expressions and the accompanying emotional experiences of individuals is a topic of great interest. According to research, facial expressions and indications might offer important clues for the study and classification of stress. Notably, changes in the mouth and eyebrows are important signs of stress on the human face. This technology records live video and applies conventional conversion to capture stress levels. In order to analyse the user\'s stress levels, this system records live video and uses conventional conversion and image processing techniques. The technology provides more precise and effective results in stress prediction by utilising machine learning algorithms that concentrate on brow and lip motions.

Introduction

I. INTRODUCTION

Stress is a prevalent issue in today's society, affecting both physical and mental health. It is a natural response of the body to pressure and can create a feeling of being overwhelmed. Stress can be brought on by a variety of circumstances or life events, and various people deal with it in different ways. The capacity to handle stress is influenced by a variety of elements, including personality qualities, socioeconomic status, early experiences, and genetic predispositions. On the contrary hand, excessive stress can be harmful and can hasten the onset of depression. According to the World Health Organization (WHO), stress is considered a psychological problem that affects approximately one in four individuals. It can lead to mental and socio-economic challenges, including difficulties at work, strained relationships, depression, and, in severe cases, even suicidal tendencies. Long-term stress can also result in a constant state of worry, fear, and dissatisfaction, negatively impacting both physical and mental health. Consequently, despite material prosperity, many people still struggle to find happiness. Different things, such as emotional, physical, and mental sources, can lead to stress. On the other hand, persistent stress is a problem that can be harmful, particularly when it develops into a chronic condition. Chronic stress may occasionally have a genetic or inherited component.  It is crucial to recognize and manage stress before it escalates, as excessive stress can have severe consequences, even leading to loss of life.

The first step in living a stress-free and contented life is to acknowledge and accept stress. Early stress detection and management are crucial, and there are cost-effective solutions available for this. There is a lot of literature on stress detection, and a considerable amount of research has been done to detect and identify stress in people. Both traditional and scientific methods exist for this purpose. One approach is the use of questionnaires, where psychiatrists provide a comprehensive set of questions to assess if an individual is experiencing stress. However, this method has its limitations and drawbacks, as the accuracy of the answers is dependent on the person's honesty, and some questions may not be suitable or applicable in every situation.

Another method involves the use of sensors to measure physiological responses and indicators of stress. Although this methodology can offer more objective data, it can also be time-consuming and expensive when compared to other methods.

Assessing and reducing the effects of stress on our socioeconomic well-being require effective stress management strategies. Helping people who are under stress manage effectively requires offering them counseling support. While stress cannot be totally eliminated, it can be managed and its effects lessened by taking preventive measures. Currently, only specialists in medicine and psychology are qualified to assess whether a person is depressed or stressed. Traditional methods, such as self-reporting, are often used to identify stress levels. However, with the advancement of IT industries and the introduction of new technologies and products, there is potential for innovative approaches to stress detection and management.

Since it is believed that regardless of race or culture, facial expressions represent concurrent emotional experiences, stress can be seen to have an impact on the human face. Recent research has demonstrated that facial cues and emotions can offer insightful information for classifying and assessing stress. Key indicators of stress on the human face include eyebrow and lip movements. In this project, a method is proposed for stress recognition using high-dimensional features extracted from face images captured by a standard camera. Facial landmarks are utilized to enhance feature extraction, as they exhibit significant changes when a person is stressed. Machine learning, a branch of artificial intelligence, empowers the system to autonomously learn and improve from its experiences without explicit programming (AI).

Machine Learning enables computer programs to access and analyse data, learning from it autonomously. Explicit programming can create mathematical models that render conclusions or predictions using training data. visual mining is used to find hidden information, create relationships between visual data, and find patterns that might not be immediately obvious. This field includes datasets, image processing, data mining, machine learning, and datasets as essential elements. According to cautious estimates in the medical literature, stress is thought to be the underlying cause of 50–80% of physical disorders. It is frequently cited as a major contributor to cardiovascular disease. Various illnesses like diabetes, ulcers, asthma, migraine headaches, skin diseases, epilepsy, and sexual dysfunction have also been connected to stress. These ailments, along with many others, are thought to be psychosomatic in nature, which means that mental factors either cause them or make them worse.

There are three main effects of stress.

• Subjective Symptoms: Guilt, shame, anxiety, wrath, and frustration are just examples of subjective symptoms of stress.
• Physical Symptoms: People may feel worn out, tense, anxious, irritable, depressed, or lonely. Stress can cause noticeable behavioural changes in a person, such as an increase in accidents, drug usage, and alcohol use.
• Cognitive Symptoms: Stress can affect cognitive functioning, resulting in poor judgement, rash decisions, forgetfulness, and increased sensitive to criticism.

A. Objective

The main objective of stress detection using deep learning techniques is to:

1. Predict a person's level of stress based on monitored  movements of their lips and brows
2. To assess the person's stress levels.
3. To determine based on the calculated stress level whether the person is experiencing high or low stress.

B. Aim of the Project

The project's objective is to measure stress using the space between the lips and eye brows in a live video. Stress label is identified and stress level is forecasted by monitoring brow and lip movements.

II. LITERATURE SURVEY

R. Riaz, N. Naz, M. Javed, F. naz and H. Toor, “ Effect of Mental Workload Related Stress on Physiological Signals,” 2020 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), 2021. In a study changes in four physiological signals were examined in response to mental workload-related stress. The study involved 26 participants, and the four physiological signals analysed were electrodermal activity (EDA), breathing rate (BR), skin temperature (ST), and blood pressure (BP). Stress was induced using methods previously employed in the literature, such as public speaking, exposure to different odors, and playing stressful games. The results indicated that EDA exhibited rapid changes with varying stress levels, particularly during public speaking, where the average change in EDA reached up to 8 micro siemens. Additionally, skin temperature showed an increase as stress levels rose, with an average increase of approximately 0.8 degrees Celsius. Notably, once the skin temperature rose, it took some time to cool down even as stress levels decreased. On the other hand, BP and BR did not demonstrate a consistent pattern or trend in response to changing stress levels. [1].

L. Liakopoulos, N. Stagakis, E. I. Zacharaki and K. Moustakas, “CNN-based stress and emotion recognition in ambulatory settings,” 2021 12th International Conference on Information, Intelligence, Systems Applications (IISA), 2021. This paper presents a facial expression recognition system that utilizes Convolutional Neural Networks (CNN) for appearance-based analysis. The Local Binary Pattern (LBP) is employed to extract appearance features, and the CNN is trained to categorize facial expressions into four basic emotions: anger, fear, unhappiness, and non-stressed expressions. The system is tested using the Indian and Cohn-Kanade databases.

The goal of the study is to develop safeguards for psychosocial hazards by concentrating on monitoring emotional states utilising wearable technology and discrete sensors. Signal processing techniques and cutting-edge machine learning algorithms are used to analyse data from numerous sensing modalities, such as heart rate signals, electrophysiological signals, facial expression features, and body posture, in order to covertly identify stress and unpleasant emotions. Different classifiers (SVM, KNN, Random Forest, Neural Networks) and data fusion strategies for the sensors are compared in the study. Furthermore, the research includes a second phase assessment that involves emotion recognition through facial expressions using images captured by a smartphone camera. When paired with stress indicators, the CNN's implementation on the Android platform provides close sswhich makes it easier to investigate the link between stress and depressive states. [2].

S. Chickerur and A. M. Hunashimore, “A Study on Detecting Stress using Facial Expressions, Emotions and Body Parameters,” 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), 2020. This paper titled "A Study on Detecting Stress using Facial Expressions, Emotions, and Body Parameters" presents a study that examines the detection of mental stress through changes in heart rate and facial expressions resulting from excessive internet usage. The study involved a setup comprising a heart rate monitoring device and the capture of facial expressions. A group of 100 volunteers (80 boys and 20 girls) participated in the study, which included two different situations: the first involved using digital systems without internet access, while the second involved using digital systems with internet access. The results demonstrated significantly higher stress levels during the second situation compared to the first. While clinical research indicates that heart rate variability is a reliable indicator of stress levels, the study also captured facial expressions along with heart rate to enhance detection efficiency. It is important to note that stress can be caused by various factors, and this discussion focuses on negative or "bad" stress, such as work-related mental pressure, rather than positive or "good" stress, such as achieving good grades in an exam. [3].

The paper titled "Text Based Stress Detection Techniques Analysis Using Social Media" (S. Jadhav, A. Machale, P. Mharnur, P. Munot, and S. Math) explores stress detection methods utilizing textual data, facial expressions, videos, and audio. The study provides an overview of research approaches employed in stress detection using social media. Specifically, the paper focuses on stress detection techniques using textual data derived from sources like tweets, comments, and chats. The objective of this work is to analyze the methods utilized for stress detection through textual data. [4].

III. SYSTEM ANALYSIS & DESIGN

A. Proposed System

The existing stress detection system suffers from limitations in real-time detection, as it relies on images or pre-recorded videos. Moreover, it only considers the movement of the left and right eyebrows as stress indicators, overlooking the importance of lip movements in stress detection. To address these drawbacks, a proposed system has been developed to offer a user-friendly application for stress detection in real-time. This system utilizes the OpenCV library, enabling live video capture of human faces and stress detection based on both eyebrow and lip movements. It further classifies the stress level as high or low based on the calculated stress level. By incorporating four key points for stress detection, the proposed system aims to provide improved accuracy.

B. Architecture Design

3)  OpenCV - Python

When a person is using a computer, a camera is utilised to capture their immediate frontal vision. The captured video is split into equal-length segments, and a collection of frames with a similar number are retrieved from each segment and evaluated using some image processing algorithms. OpenCV's integration with Python provides a powerful toolkit for image processing and computer vision tasks, addressing the unique requirements of image classification pipelines. Its functionalities for data pre-processing, augmentation, feature extraction, model evaluation, and real-time classification enhance the accuracy, robustness, and performance of image classification models, ultimately contributing to more effective and efficient image analysis and classification tasks.

OpenCV plays a significant role in video classification tasks by providing a comprehensive set of functionalities for video input, frame processing, feature extraction, temporal analysis, and model integration. It enables reading videos from different sources, processing individual frames, and extracting relevant features for modelling and classification.

cv2.imread() and cv2.imshow() are two important functions provided by OpenCV in Python for reading and displaying images, respectively. The cv2.imread() function is used to read an image from a file. It takes the path to the image file as a parameter and returns a NumPy array representing the image. The cv2.imshow() function is used to display an image in a window. It takes the window name and the image as parameters. It will create a new window with the specified name and display the image within it. The window will automatically adjust its size to fit the image dimensions.

4)  Pixel Transformation

In the field of image processing, pixel transformation is a  frequently employed technique for altering pixel values.Its primary objective is to increase the variability and versatility of an image. One common application of pixel transformation is the conversion of a color image into a grayscale image, where the colors are replaced with shades of gray. Additionally, pixel transformation can involve the determination of a threshold value, by means of which the grayscale image is transformed into binary form.In this process, each pixel is compared to the threshold value, and if the pixel value exceeds the threshold, it is set to 1; otherwise, it is set to 0.

5)  Facial Landmark

Face landmark detection is a method used to identify specific points of interest on a person's face in an image. It has been utilized for various purposes, including emotion detection, gaze estimation, face swapping, facial augmentation, and controlling virtual characters.

The coordinates of the lips and eyebrows are found by the stress detection module by analysing a binary image starting in the upper left corner. By calculating the displacement of these facial features from their average positions, we can determine the degree of stress. The eyebrow coordinates are examined to assess the movement of the eyebrows, while the lip coordinates are examined to evaluate lip movements. Emotions such as fear, sadness, and anger are classified as indicators of stress, while other emotions are considered non-stressed. To quantify the stress level, mathematical calculations like normalization are applied to the distances between the eyebrows and lips. By considering the collective decisions of individual frames, the overall stress level is determined. If the stress level exceeds a specified threshold, it is labelled as high stress along with the corresponding level; otherwise, it is classified as low stress along with its level.

IV. MODULES

A. Training Dataset

The dataset undergoes training using Convolutional Neural Networks (CNNs). To accomplish this, a Sequential Model is employed, consisting of five convolutional layers. A 64-person batch size is used during the training procedure, which spans 100 epochs. The Adam optimizer is used to compile the model. Finally, a JSON file containing the trained model is saved.

B. Live Video Capturing

Python provides a variety of libraries for image and video processing, including OpenCV. OpenCV is a comprehensive library that offers numerous functions for working with images and videos. With OpenCV, you can capture videos from a camera. By creating a video capture object using the cv2.VideoCapture() function, you can easily capture videos through a webcam and perform desired operations on them. The process involves setting up an infinite while loop, using the read() method to read frames from the video using the video capture object, and displaying the frames using the cv2.imshow() method. The loop can be terminated when a specific key is pressed by the user.

C. Emotion Classifier

The model is trained using Fer2013 (Facial Expression Recognition) which is the dataset used for facial expression detection and classifying the emotions as stressed and not stressed. Emotions are classified into angry, sad, surprised,  happy, disgust, fear and neutral. Based on these emotions a person is classified into stressed or not stressed. Further, classified stressed emotions are monitored for low stress and high stress based on distance between eyebrow and lip movements.

D. Detecting Eyebrow Movements and Lip Movements

Face landmark detection is a computer vision task where we want to detect and track key points from a human face. This task applies to many problems.

Face Landmark Detection with Dlib:

Dlib is a versatile library that combines machine learning and computer vision techniques. Although originally implemented in C++, it can be utilized in Python as well. Among the many capabilities of Dlib, one of the valuable applications is face landmark detection. By using a pretrained model, we can initialize the face landmark detector. This model is based on ensemble regression trees, as it aims to predict continuous values. Additionally, Dlib provides a convenient function called get_frontal_face_detector() which is already integrated within the library, eliminating the need to include the main model file.

To begin the face landmark detection process, we first pre-process the image by converting it into a grayscale format. This step allows for faster processing. We then utilize the built-in face detector in Dlib to identify the bounding box around the face. The output of the face landmark detector is a shape object containing 68 (x, y) coordinates representing various facial landmark regions. To visualize these landmarks on the image, we convert the shape object into a NumPy array using the face_utils.shape_to_np utility function. This conversion enables us to easily overlay the discovered face landmarks onto the image.

E. Calibrating the Distance

After acquiring the facial landmarks using FACIAL LANDMARKS IDXS, we calculate the Euclidean distance between the endpoints of the left eyebrow and the starting points of the right eyebrow. Similarly, we calculate the Euclidean distance between the left and right corners of the lips. These distances are then normalized. The purpose of normalization is to standardize the values of numeric columns in the dataset to a common scale without distorting the relative differences between the values. In machine learning, normalization is necessary when features have varying ranges. For instance, if we have a dataset with features like age and income, where age ranges from 0 to 100 and income ranges from 0 to 100,000 and beyond, the income values are significantly larger than the age values. Without normalization, the larger income values can dominate the analysis and potentially bias the results. Therefore, by normalizing the data, we bring all variables to the same range, ensuring that each feature contributes equally to the analysis.

F. Predicting Stress using Eyebrows

Stress is calculated based on distance between two eyebrows alone by normalizing the distance between ending point of the left eyebrow and starting point of the right eyebrow based on dlib facial landmark points.

G. Predicting Stress using Eyebrows and lips Together

Stress is calculated based on distance between two eye brows and distance between two joining points of lips. Predicting stress using lips and eyebrows together is a new addition which gives more efficient results when compared to predicting stress using eyebrows alone. when a person smiles, the distance between the joining points of their lips usually increases, it is not always a definitive indicator of happiness. Smiling is a complex facial expression that can have multiple meanings therefore eyebrows and lips together helps to find efficient stress levels..

If the average normalized value of distance between two eye brows and lip points is greater than or equal to 0.65 then that person is classified as having high stress else the person is classified as having low stress and their respective stress level is calculated using the mentioned formula (normalized_distance * 100) and along with it stress label is displayed on the frame.

H. Plotting the Stress Levels

After Monitoring  the stress levels of a person for a while a graph is plotted between number of frames on y axis and  the count of frames in which the person is stressed and not stressed on x -axis.

Conclusion

Stress Detection System helps the end user to detect stress based on facial expressions in an easier way and early diagnosis prevents them from having high risk.

References

[1] R. Riaz, N. Naz, M. Javed, F. naz and H. Toor, “Effect of Mental Workload Related Stress on Physiological Signals”, 2020 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), 2021. [2] L. Liakopoulos, N. Stagakis, E. I. Zacharaki and K. Moustakas, “ CNN based Stress and Emotion Recognition in a]Ambulatory Settings”, 2021 12th International Conference on Information, Intelligence, Systems Applications (IISA), 2021 [3] S. Chickerur and A. M. Hunashimore, “A Study on Detecting Stress using Facial Expressions, Emotions and Body Parametersl”, 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), 2020. [4] S. Jadhav, A. Machale, P. Mharnur, P. Munot and S. Math, “Text Based Stress Detection Techniques Analysis Using Social Medial”, 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), 2019. [5] L. Pepa, A. Sabatelli, L. Ciabattoni, A. Monteriu, F. Lamberti and L. Morra, “Stress Detection in Computer Users From Keyboard and Mouse Dynamics”, in IEEE Transactions on Consumer Electronics, Feb. 2021. [6] N. Chaowadee, P. Lertsiriyothin, T. Phuangkhemkhao and T. Chanwimalueang, “Reinforced Learning in Children through a Stress Warning Unit”, 2021 2nd Information Communication Technologies Conference (ICTC), 2021. [7] Bhattacharyya. R, Basu. S, “Eye Movement Analysis for Depression Detection”, The Economic Times, ACT Australia Nat Univ, Canberra, 2018. [8] U. S. Reddy, A. V. Thota and A. Dharun, “Machine Learning Techniques for Stress Prediction in Working Employees”, 2018 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), 2018. [9] G. Giannakakis, D. Manousos, F. Chiarugi, “Stress and Anxiety Detection Using Facial Cues from Videos”, “Biomedical Signal Processing and Control”, January 2017.

Copyright

Copyright © 2023 Ms. A. Aruna Jyothi, G. Spandana , S. Sai Harshitha , G. Sravya, Ch. Bhuvana . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.