Sign Language Translator and Teaching System

Abstract

Sign Language is a non-verbal form of communication that is used to interact with the specially abled. These people have communication barriers when it comes to daily interactions with others who don’t understand sign language. Technological advancements and progress in Machine Learning algorithms have made gesture recognition possible. Sign language is one of the oldest and most natural forms of language for communication, but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real-time method using neural networks for fingerspelling based on American sign language. In our method, the hand is first passed through a filter and after the filter is applied, the hand is passed through a CNN module which predicts the class of the hand gestures. This literature survey analyses studies based on sign language translator and teaching applications. Conclusions drawn from this literature review could help in creating the perfect application that can make daily communication easy for the specially aided. We plan on providing two modules, one wherein the sign language gestures are converted into text and displayed. The second module is made wherein the text entered is converted into sign language and displayed on the User interface.

Introduction

I INTRODUCTION

Communication is how people talk, explain what they feel, and interact with others. It is mankind’s mode of conversation to express what goes on in the human brain. Communication can be done via speech, sound, and written language. Speech is the most common one among these.  A person who is specially-abled faces communication barriers in every stage of their daily routines. May it be while shopping or for something as simple as asking for change. How are they supposed to get their thoughts forward and express themselves? This issue led to the growth of sign language.

Sign Language is used by the deaf and dumb to converse with people on a day-to-day level. Sometimes this could be an issue because the other person might not know this language. He or she might not understand what the person is trying to convey.

It is to overcome these issues that we wish to build a Sign language teaching and learning assistant. Our application would highly benefit society and help make these unpleasant barriers redundant. One of the biggest challenges today is raising awareness about the importance of sign language and providing the right support for people to learn it. Our application would make people want to study with its easy-to-use, interactive User Interface.

The application also helps bridge the gap between written language and sign language, making it easier for people to communicate with each other.  Finally, the system can be used to improve the lives of those who use it, by providing them with a way to effectively communicate with each other. The options in the market for these learning applications are very few and not everybody knows about them. Our application aims at being everyone’s go-to when it comes to using sign language to communicate.

We plan on giving users login and signup options to save their progress if they wish to learn using our content. Users would be prompted with two options: one where sign language would get converted to text/audio and one where text/audio input gets converted to text. Our images were taken by the application, first, go through feature extraction. In this process, the images from the training data go through convolution and max pooling layers.

This acts as the input for Neural Networks. The Neural network is divided into two dense layers; one is the hidden layer and the other is the output layer. The hidden layer comprises the activation function ReLu whereas the output layer comprises the SoftMax activation function as we use classification in this module. This module predicts the required outputs for their respective inputs taken from the camera. When a user performs a sign, the webcam records it and saves this. This captured image then goes through Convolution Neural Networks and the module created. This module then converts the message into text or audio as per the user’s preference.

For Sign Language learning, the application will capture the signs you perform and show you what is wrong with them. It will help rectify errors and improve the user’s fluency. Our application aims to help society grow and be an open environment where the specially abled don’t feel embarrassed or face difficulties in carrying out their daily chores.

II. LITERATURE SURVEY

Kartik Shenoy, Tejas Dastane, Varun Rao, Devendra Vyavaharkar, “Real-time Indian Sign Language (ISL) Recognition”. Orientation histogram was used to extract features and classification was done using KNN and Euclidean distance. Only a set of words was taken here and so was not fully accurate. [1]

J. Rekha, J. Bhattacharya and S. Majumder, "Shape, texture and local movement hand gesture features for Indian Sign Language recognition". Principal Curvature Based region with wavelet packet decomposition extraction and classification was done using Multiclass SVM. Though Static and Dynamic gestures were only used and not words. [2]

Hand Talk Translator Mobile Application converts text into sign language gestures and animates it for a person to follow. Voice input can also be given instead of text and this application will show signs of it. Unfortunately, it does not check for errors when a person is using sign language. [3]

Lingvano Mobile Application The application has a variety of dialogue clips showing people signing with corresponding subtitles in English. The application also has a sign language mirror option that allows users to copy a person doing a sign while projecting their image for feedback. Even though it helps learn, it does not convert signs to text or audio. [4]

Hands-On ASL It is a visual-only application that uses 3D model hands instead of videos or still images. You can zoom into the hands and rotate them to view them from different angles. Only for learning purposes, not communication. [5]

Adithya, V. & Vinod, P.R. & Gopalakrishnan, Usha. (2013). Artificial neural network-based method for Indian sign language recognition. The Segmentation using Otsu’s algorithm is used. SIFT and HOG are used for feature extraction and classification using ANN. Finger spellings are used so may not be recognized everywhere. [6]

Sharma, S., Singh, S. Recognition of Indian Sign Language (ISL) Using Deep Learning Model. The number of fingertips and their distances from the centroid together with PCA is used as a feature descriptor. Alphabets and numbers were used. Sentence formation issues. [7]

Tavari, Neha V., Prof. A. V. Deorankar and Dr. Prashant N. Chatur. “Hand Gesture Recognition of Indian Sign Language to aid Physically impaired People.” HOG feature extractor with Neural Network Classifier. Double Hand 18 ISL alphabets were used. The issue for those who cannot use both hands. [8]

Kagalkar, R. M., & Gumaste, S. V. (2017). ANFIS Based Methodology for Sign Language Recognition and Translating to Number in Kannada Language. International Journal of Recent Contributions from Engineering. Centre of gestures, a distance of measures to boundary and degree measure as feature extraction and ANFIS as a classifier. Only makes use of alphabets. Would create problems when communicating numbers. [9]

A. S. Nikam and A. G. Ambekar, "Sign language recognition using image-based hand gesture recognition techniques," Maximum curvature points as fey frames or gesture shape identification. Only makes use of alphabets. Would create problems when communicating numbers. [10]

III. PROBLEM STATEMENT

The communication barrier between a specially aided person and those around him or her can be a hassle. We may not understand what they are communicating to us. For this, we have decided to create a sign-language teaching and learning application that would not only translate the signs a person does but also can teach someone to use sign language to communicate.

IV.  PROPOSED SYSTEM

The system which we are going to make contains devices like a mobile phone, camera, and speaker.

An existing application called Hand Talk Translator was instrumental in giving us the perfect start. We reviewed all existing applications and listed down the additional features we wanted to add.

The following are the features provided by our system:

1. When a user opens the application, he would be prompted with 2 options: Sign-To-Text or Text-To-Sign.
2. Sign-To-Text takes input using a camera and converts signs into text/audio.
3. Text-To-Sign takes audio or text input and shows signs to follow and perform.
4. An option to learn sign language using videos
5. Camera records and captures images while learning and explains errors and ways to rectify them.

V. METHODOLOGY

The linear working of our project can be explained in the following manner:

1. Step 1: The user opens the web application
2. Step 2: The user selects either Translate
3. Step 3: The user clicks on the sign language by region. eg: ASL, BSL, IS
4. Step 4: Translate: The user selects the mode of translation: text to sign language or vice versa
5. Step 5: A big window where the camera is on along with a white scanning box on the top right of the camera section. The white scanning box is where users can place their hands and do sign language gestures which will get detected and translated to text.
6. Step 6: The user logs in or creates an account to track the progress of the learning.
7. Step 7: For learning, the user selects a language.
8. Step 8: The user selects the module
9. Step 9: The module opens with a white screen for recognizing hand gestures.
10. Step 10: The webcam captures the images done by the user and checks the accuracy of the gesture with the video file.

VI. DESIGN

The above DFD (Data flow diagram) is where the user makes a sign language hand gesture in real time. The video is captured using OpenCV and then transformed frame by frame. The CNN model is then fed with an extracted hand gesture. Then the gesture is predicted. [12]

VII. ACKNOWLEDGEMENT

We would like to express our special thanks of gratitude to our guide Prof. Deepali Shrikhande who gave us the golden opportunity to do this wonderful project on the topic Sign Language Translator and Teaching System which also helped us in doing a lot of Research and we came to know about so many new things we are thankful to her. Secondly, we would also like to thank our professors of review panel in finalizing and improving this project within the limited time frame. This project helped us in understanding the various parameters which are involved in the development of web application, various machine learning models and the working and integration of front end along with the back end.

Conclusion

Communication is the process of exchanging thoughts and messages in many ways, such as using diverse gestures to express one\'s views with others. Gestures are non-verbally transmitted messages that are understood through eyesight. Sign language refers to deaf and dumb people\'s nonverbal communication. Sign language is a language that communicates meaning using gestures rather than words, such as hand forms, orientation, movement of the hands, arms, or body, facial expressions, and lip patterns. Our project is aimed at being a Web application and a Mobile application. We have already built a CNN model till now. In this, we took a dataset from Kaggle and trained it. 3 convolutional layers and 3 max-pooling layers were used in the feature extraction process. This result was flattened and used as input for the neural network. The hidden layer is used alongside the activation function as ReLu. Finally, the Soft-Max activation function is used.

References

[1] Kartik Shenoy, Tejas Dastane, Varun Rao, Devendra Vyavaharkar, “Real-time Indian Sign Language (ISL) Recognition”. [2] J. Rekha, J. Bhattacharya and S. Majumder, \"Shape, texture and local movement hand gesture features for Indian Sign Language recognition,\" 3rd International Conference on Trendz in Information Sciences & Computing (TISC2011), 2011, pp. 30-35, doi: 10.1109/TISC.2011.6169079. [3] Hand Talk 2013. Hand Talk Translator (version 3.4.12). [Mobile app]. [Accessed 20 October 2022] [4] Fischer Matthias u Mitges-Lingvano GesbR 2021. Lingvano (version 2.6.7). [Mobile app]. [Accessed 15 October 2022] [5] Play Nice LLC 2019. Hands-on ASL (version 1.0). [Mobile app]. [Accessed 20 October 2022] [6] Adithya, V. & Vinod, P.R. & Gopalakrishnan, Usha. (2013). Artificial neural network based method for Indian sign language recognition. 1080-1085. 10.1109/CICT.2013.6558259. [7] Sharma, S., Singh, S. Recognition of Indian Sign Language (ISL) Using Deep Learning Model. Wireless Pers Commun 123, 671–692 (2022). [8] Tavari, Neha V., Prof. A. V. Deorankar and Dr. Prashant N. Chatur. “Hand Gesture Recognition of Indian Sign Language to aid Physically impaired People.” (2014). [9] kagalkar, R. M., & Gumaste, S. V. (2017). ANFIS Based Methodology for Sign Language Recognition and Translating to Number in Kannada Language. International Journal of Recent Contributions from Engineering, Science & IT (iJES), 5(1), pp. 54–66. [10] A. S. Nikam and A. G. Ambekar, \"Sign language recognition using image based hand gesture recognition techniques,\" 2016 Online International Conference on Green Engineering and Technologies (IC-GET), 2016, pp. 1-5, doi: 10.1109/GET.2016.7916786. [11] Hyung-Gun Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, Karthik Ramani, \"InfoGCN: Representation Learning for Human Skeleton-based Action Recognition\", 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.20154-20164, 2022. [12] Roshnee Matlani, Roshan Dadlani, Sharv Dumbre, Shruti Mishra, Mrs. Abha Tewari, \"Real-time Sign Language Recognition using Machine Learning and Neural Network\", 2022 International Conference on Electronics and Renewable Systems (ICEARS), pp.1381-1386, 2022. [13] Qiang Fu, Jiajun Fu, Jian Guo, Shuxiang Guo, Xun Li, \"Gesture Recognition based on BP Neural Network and Data Glove\", 2020 IEEE International Conference on Mechatronics and Automation (ICMA), pp.1918-1922, 2020. [14] Sheikh Monirul Hasan, Md. Saiful Islam, Md. Ashaduzzaman, Muhammad Aminur Rahaman, \"Automated Software Testing Cases Generation Framework to Ensure the Efficiency of the Gesture Recognition Systems\", 2019 22nd International Conference on Computer and Information Technology (ICCIT), pp.1-6, 2019. [15] Ramla Saif, Muhammad Ahmad, Syed Zohaib Hasan Naqvi, Sumair Aziz, Muhammad Umar Khan, Muhammad Faraz, \"Multi-Channel EMG Signal analysis for Italian Sign Language Interpretation\", 2022 International Conference on Emerging Trends in Smart Technologies (ICETST), pp.1-5, 2022. [16] K. Chang, H. Li, Y. Tan, P. L. K. Ding and B. Li, \"A Two-Stage Convolutional Neural Network for Joint Demosaicking and Super-Resolution,\" in IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 7, pp. 4238-4254, July 2022, doi: 10.1109/TCSVT.2021.3129201. [17] I.A. Adeyanju, O.O. Bello, M.A. Adegboye, Machine learning methods for sign language recognition: A critical review and analysis, Intelligent Systems with Applications, Volume 12, 2021, 200056, ISSN 2667-3053, https://doi.org/10.1016/j.iswa.2021.200056.

Copyright

Copyright © 2023 Aditee Kore, Hrishika Samani, Dharmil Karia, Deepali Shrikhande. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.