Mental Health Status Detection through Handwriting Analysis

Abstract

Around the turn of 20th century, scientists began stirring up what would ultimately become a world changing revolution. This is where Machine learning and Artificial intelligence came into being. We have come so far that now we can use the same technologies to check the mental health status of a person. We have come so far into technological advancement that we are able to use certain aspects of handwriting to predict the mental status of a person. The implementation of it includes checking the slanting of the pen, thickness of the wording and various other features. The project is said to use five classes depression, anxiety, stress, panic and normal. This project will prove to be beneficial to those who are in critical need of therapy but don\'t have enough money to fund it and the project will help diagnose the mental illness. Now many scientists believe that we are on the cusp of a second revolution as machine learning as grown so big. They predict it will transform the society yet again, this time through the vital importance of changing the human intelligence to machine intelligence.

Introduction

I. INTRODUCTION

Because mental health is equally as essential as physical health, it is critical to monitor a person's mental health, which includes sadness, anxiety, stress, and panic, which are all frequent among people. All behaviours, including writing, originate in the brain. Writing, like other human behaviours, is controlled by the central nervous system. Our brain sends information to our hands via nerve impulses,    which    cause   a    motor   motion.

[8] Our brain leads what we type on paper, but our muscles are the result of two-way communication between the brain and the muscles. Although handwriting is driven by pen activity, it is controlled by the central nervous system, which may be an unconscious but highly revealing process. Because of the close relationship between brain impulses and handwriting, handwriting may be used to correctly predict an individual's physical, emotional, and mental health[10]. Handwriting analysis is performed to find any handwriting anomalies in the topic.

According to recent statistics, 20% of Indians would be diagnosed with different mental diseases in their lifetime by the end of December 2022. A person's mental health can be determined in a variety of ways, including by examining the symptoms or having it diagnosed by a psychologist. A person's participation in various environmental settings also plays a role.[1-3] Academic anxiety, for example, have wide-ranging consequences on test performance and topic choice, resulting in a reduction in cognitive ability[15].[11] Handwriting analysis employs CNN, which stands for Convolution neural network, a strong deep learning technology because to its capacity to handle vast volumes of data. [6][12][14] It is employed in image data, classification prediction problems, and regression prediction issues, in addition to handwriting analysis. Recognizing unpleasant emotions is critical, and handwriting analysis, which is the examination of handwriting in an attempt to establish someone's mental health condition, serves this function.

The suggested technique is called "Mental Health Status Detection through Handwriting Analysis." First, a handwritten sample will be submitted to the system as  input. The  incoming  image is then pre- processed,  which   includes image scaling and normalization. The convolutional neural network will then be used to train the dataset before testing. Finally, a fresh input sample will be supplied to predict the emotion based on the trained model, which will test this input sample and provide a result indicating sadness, anxiety, stress, panic, or normal.

II. RELATED WORKS

A. “Artificial Neural Network for human behavior prediction through handwriting analysis”

The authors of the research [3] employed a technique to estimate a person's personality based on their handwriting's baseline, pen-pressure, or individual letters. These parameters are then used as inputs to the artificial neural network, which generates the writer's personality attribute. The 30 various outputs   produced  by ANN correspond to the  30  different  personality  qualities  of the writer, depending on how they handle pressure  and  what  their writing  standard is. In  order  to  forecast  the writer's conduct, a sizable amount of data from 30 possible combinations of attributes is evaluated.

B “Automatic emotion recognition through    handwriting analysis”

The scanned handwritten sample of a writer is used as the input in this study [4], which then performs image preprocessing (thresholding, noise removal, and skew correction), feature extraction (using the features slant, baseline, pen pressure, size, margin, and zone), and classification to identify emotions. This technology recognizes emotions like depression,   fear,  anxiety, excitement, etc. that pinpoint  the  writer's  precise  mental  state. Finding out if the writer is stressed out would be beneficial; if so,  psychological  treatment or  other  forms  of  assistance  might  be  given with the goal of helping them recover.

C. “Handwriting analysis for mental disorder detection”

This method [8] uses a machine learning approach, such as a convolution neural network and a support vector machine, to extract information from a person's handwriting and predict their personality traits accurately. The handwriting samples' attributes will be extracted into feature vectors, which can then be compared to the dataset that was initially trained before being mapped to categories with accompanying personality traits. Diagnoses for conditions like Alzheimer's, Autism, Mild Cognitive Impairment, Dysgraphia, Schizophrenia, and Parkinson's are made using this technique. Review of the digital handwriting analysis supported by mental illness. It has been discovered that characteristics related to motion, time, and pressure can be used to identify mental illnesses using handwriting analysis.

D. “Mental health analysis using handwriting analysis by generating writing prompts”

The size of the letters, the slant of the writing, the baseline, pen pressure, the spacing between letters, the spacing between words, and the top margin of a document are the seven handwriting characteristics used in this study [9] to determine whether or not the writer has stress, depression, or anxiety. The handwritten sample's retrieved characteristics can be scaled, standardized, and used to infer information about a person's mental state. Additionally, four SVMs will be utilized to categorize the outcomes of the techniques mentioned in the study. The Support Vector Machine model's accuracy over the test dataset is 98.58 percent.

III. PROPOSED SYSTEM

A. Dataset

The dataset used in the proposed system is divided into five classes. They are, depression, anxiety, stress, panic and normal. Depression class contains 1039 scanned handwriting images, Anxiety class contains 983 images, Stress class has 934 images, Panic class has 1034 images and at last Normal class has 909 images. All these images of five different classes are used for training purpose.

B. Methodology

The flow of the proposed system is- first an image of handwritten text whose mental status has to be found is uploaded using upload function, then the image goes through image pre-processing, then in the next step the image is given as input to model file to predict one of the five results, they are- Depression, Anxiety, Stress, Panic or Normal.

1. Training Model: The dataset must first be loaded into the training model before it can be split. The dataset is divided into test data and train data in the ratio 1:4. The divided train data is then used as input to the CNN algorithm, which aids in training, in the subsequent data modelling process. By providing the algorithm with test data, the trained image is assessed. The CNN algorithm also incorporates feature extraction. After data modelling and accuracy calculations, the create model function is performed. After the data has been trained and is displaying a high accuracy rate, the model file has to be built.
2. Testing Model: First handwriting image of a person whose mental status has to predict is uploaded to the system. The image is pre-processed, which involves image resizing and normalization. After image pre-processing step, the image is given to convolution neural network model file to predict one of the five classes- Depression, Anxiety, Stress, Panic or Normal, to which the image belongs to.
3. Algorithm: This method predicts a person's mental state from a sample of handwritten text using a convolution neural network technique. Convolution layer, Pooling layer, and Fully Connected layer make up the three layers of the CNN architecture. [13].

a. Convolution Layer: A bias b is given if an input image I has dimensions I1 X I2 and a kernel (filter) has dimensions k1 X k2. The result will be a feature map F with the dimensions i X j. The below equation provides the convolution operation.

Feature map size is given by-

F = [(n-w+2p)/s] + 1, where

n-image size, w-filter size, p-padding and s-stride

Non-linearity is introduced after each convolution operation, this is performed by Rectified Linear Unit (ReLU). This is used to change the negative pixel values to zero in the convolution layer output. It is given by the equation-

F(x) = max(x,0)

b. Pooling Layer: Each feature map in the output of the convolution layer is reduced in dimension by this layer. There are numerous types of pooling; in this system, max pooling has been used. Depending on the desired size, it selects the largest element from the matrix of the rectified feature map.

c. Fully Connected Layer: After combining the Convolution layer and Pooling layer, next is, Fully Connected layer. This layer uses high level features from Convolution layer/Pooling layer to predict the output.

C. Advantages

1. There are many other ways to identify a person’s mental health, but they are high-priced and are not easily available to common people. The proposed system can be implemented as a web application. It is less expensive and easily accessible by common people compared to other mental status prediction methods.
2. In the system we have used Convolution Neural Network to build the model file. The CNN algorithm provides organized dense neural networks which is used to perform identification or prediction etc.
3. One more advantage of using Convolution Neural Network algorithm is, feature extraction is a part  of  CNN.  Therefore,  no   need to perform feature extraction externally. It performs features dimensionality reduction and parameter sharing without human intervention.
4. Handwriting of a person is a powerful tool to predict his/her emotional state. Handwriting is unique to each person and also changes with the current emotion of the person. For example, if a person is nervous then his/her then there is a change in pressure of the pen, each letter’s size, base line, slanting of each letter, spacing between words and letters or zones etc.

IV. IMPLEMENTATION

The goal of the implementation phase is to translate the proposed system into a coded form using programming languages. Here, we have used Hyper Text Markup Language and Cascading Style Sheet for front end, python for building and modeling CNN model, MySql for database and python web framework Flask is also used in the project.

The flow of the system is like this, a new user must register through email and setting password, the users who have already registered can directly login by giving the credentials that they had used to register to the application. Once logged in there are two options, one is to train the model and another one is for demo. If a user wants to check his/her mental status, he/she has to upload an image of handwriting sample and click predict emotion button. The system predicts the result corresponding to the image.

V. RESULT

4899 images representing the classes of depression, anxiety, stress, panic, and normal make up the dataset. There are 3919 images in the training dataset and 980 images in the testing dataset. The CNN model is created using the Python frameworks Keras and Tensorflow. One of the five outcomes—depression, anxiety, stress, panic, or normal—is what the model predicts.

VI. FUTURE SCOPE

The following are some of the future enhancements that can be done on the system:

1. To obtain a greater accuracy, fine-tuning of current network parameters can be done.
2. Expanding the system to operate on offline as well as online samples.
3. Extending the system to work as a language independent assistant.

Conclusion

A person\'s handwriting may be taken into consideration as a significant indicator of their mental health. Convolutional Neural Network approach is used to extract characteristics from the handwriting samples and categorize them to the proper class in machine learning to accomplish the same goal. The number of training samples rises when cross-validation is utilized, improving the system\'s accuracy. During training, the cross-validation models\' top weights are saved. This paper presents the methodology and implementation of ‘Mental health status detection through handwriting analysis. This system implementation is expected to reduce the training time and increase efficiency by using CNN model to predict the mental health status of a person.

References

[1] Gunes, Hatice, and Massimo Piccardi. \"Bi-modal emotion recognition from expressive face and body gestures.\" Journal of Network and Computer Applications 30, no. 4 (2007). [2] Ginevra Castellano, Loic Kessous, Amaryllis Raouzaiou, Caridakis, George, Lori Malatesta, Stelios Asteriadis, and Kostas Karpouzis. \"Multimodal emotion recognition from expressive faces, body gestures and speech.\" In IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 375-388. Springer, Boston, MA, 2007. [3] Champa, H. N., and K. R. AnandaKumar. \"Artificial neural network for human behaviour prediction through handwriting analysis.\" International Journal of Computer Applications, Volume (2010). [4] D. S. Bormane, Aaditi Dhadwal, Shiwali Alone, S.V. Kedar and Rashi Agarwal. “Automatic Emotion Recognition through Handwriting Analysis: A Review.” International Conference on Computing Communication Control and Automation, 2015. [5] Marcos Faundez-Zanuy, Likforman-Sulem, Laurence, Anna Esposito, St´ephan Cl´emenc¸on and Gennaro Cordasco. “EMOTHAW: A novel database for emotional state recognition from handwriting and drawing.” IEEE Transactions on Human-Machine Systems 47, no. 2 (2017). [6] Zhao, Huijuan, Ning Ye, and Ruchuan Wang. \"A Survey on Automatic Emotion Recognition Using Audio Big Data and Deep Learning Architectures.\" In 2018 IEEE 4th International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing, (HPSC) and IEEE International Conference on Intelligent Data and Security (IDS), pp. 139-142. IEEE, 2018. [7] Jiawen Han, George Chernyshov, Dingding Zheng, Peizhong Gao, Takuji Narumi, Katrin Wolf, Kai Kunze. “Sentiment Pen: Recognizing Emotional Context Based on Handwriting Features.” ACM, 2019. [8] Shreeyash Magar, Bhumika Marathe,Sunny Khambayat, Prem Mamadge. “Handwriting Analysis for Mental Disorder Detection.” International Journal of Engineering Research & Technology (IJERT), 2021. [9] Manogna Pallapothu, Pragati Shinde, Vinish Marito. “Mental Health Analysis Using Handwriting by Generating Writing Prompts.” Publication, year: International Journal of creative research thoughts(IJCRT), 2021. [10] Giannini, Marco, Pietro Pellegrini, Alessio Gori, and Yura Loscalzo. \"Is Graphology Useful in Assessing Major Depression?.\" Psychological reports (2018). [11] Crawford, John R., and Julie D. Henry. “The Depression Anxiety Stress Scales (DASS): Normative data and latent structure in a large non-clinical sample.” British journal of clinical psychology 42, no. 2 (2003). [12] Balci, Batuhan, Dan Saadati, and Dan Shiferaw. “Handwritten Text Recognition Using Deep Learning.” CS231n: Convolutional Neural Networks for Visual Recognition, Stanford University, Course Project Report, Spring (2017). [13] Zeiler, Matthew D., and Rob Fergus. \"Visualizing and understanding convolutional networks.\" In European conference on computer vision, pp. 818-833. Springer, Cham, 2014. [14] Simard, Patrice Y., David Steinkraus, and John C. Platt. \"Best practices for convolutional neural networks applied to visual document analysis.\" In Icdar, vol. 3, 2003. [15] Ko?akowska, Agata, Agnieszka Landowska, Mariusz Szwoch, Wioleta Szwoch, and Michal R. Wrobel. \"Emotion recognition and its applications.\" In Human-Computer Systems Interaction: Backgrounds and Applications 3, pp. 51-62. Springer, Cham, 2014.

Copyright

Copyright © 2022 Prof. Vinay M G, Anaga A, Anagha S, Khushi Kumar, Monica S. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.