Sign Language Detection with CNN

Abstract

Sign language is one of the oldest and most natural forms of language for com- munication, but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real time method using neural net- works for fingerspelling based American sign language. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. Our method provides 95.7% accuracy for the 26 letters of the alphabet.

Introduction

I. INTRODUCTION

Sign language is a predominant sign language Since the only disability Deaf and Dumb (hereby referred to as DM) people have is communication related and since they cannot use spoken languages, the only way for them to communicate is through sign language. Communication is the process of exchange of thoughts and messages in various ways such as speech, signals, behavior and visuals. DM people make use of their hands to express different gestures to express their ideas with other people. Gestures are the non-verbally exchanged messages and these ges- tures are understood with vision. This nonverbal communication of deaf and dumb people is called sign language. A sign language is a language which uses gestures instead of sound to convey meaning combining hand-shapes, orientation and move- ment of the hands, arms or body, facial expressions and lip-patterns. Contrary to popular belief, sign language is not international. These vary from region to region.

Minimizing the verbal exchange gap among DM and non-DM people turns into a want to make certain effective conversation among all. Sign language translation is among one of the most growing lines of research and it enables the maximum nat- ural manner of communication for those with hearing impairments. A hand gesture recognition system offers an opportunity for deaf people to talk with vocal humans without the need of an interpreter. The system is built for the automated conversion of ASL into textual content and speech. In our project we primarily focus on pro- ducing a model which can recognize Fingerspelling based hand gestures in order to form a complete word by combining each gesture.

A. Motivation

For interaction between normal people and DM people a language barrier is cre- ated as sign language structure since it is different from normal text. So, they depend on vision-based communication for interaction.

If there is a common interface that converts the sign language to text, then the gestures can be easily understood by non-DM people. So, research has been made for a vision-based interface system where DM people can enjoy communication without really knowing each other’s language.

The aim is to develop a user-friendly Human Computer Interface (HcI) where the computer understands the human sign language. There are various sign languages all over the world, namely american Sign Language (ASL), French Sign Language, British Sign Language (BSL), Indian Sign language, Japanese Sign Language and work has been done on other languages all around the world.

II. LITERATURE SURVEY

A. H Muthu Mariappan, V Gomathi ”Real-Time Recognition of Indian Sign Language” in ISBN:978-1-5386-9471-8

Abstract: The real-time sign language recognition system is developed for recog- nising the gestures of Indian Sign Language (ISL). Generally, sign languages con- sist of hand gestures and facial expressions. For recognising the signs, the Regions of Interest (ROI) are identified and tracked using the skin segmentation feature of OpenCV. The training and prediction of hand gestures are performed by applying fuzzy c-means clustering machine learning algorithm. The gesture recognition has many applications such as gesture controlled robots and automated homes, game control, Human-Computer Interaction (HCI) and sign language interpretation. The proposed system is used to recognize the real-time signs. Hence it is very much useful for hearing and speech impaired people to communicate with normal people.

B. Mohammed Safeel, Tejas Sukumar, Shashank K S ,Arman M D, Shashid- har R, Puneeth S B, ”Sign Language Recognition Techniques- A Review”,

Abstract: Sign language reduces the barrier for communicating with the humans having impaired of speech and hearing, on the other hand Sign language cannot be easily understood by common people. Therefore, a platform is necessary that is built using an algorithm to recognize various signs it is called as Sign Language Recogni- tion (SLR). It is a technique that simplifies the communication between speech and hearing impaired people with normal people, the main aim of SLR is to overcome the aforementioned drawback. In this manuscript it is aimed to review various tech- niques that have been employed in the recent past for SLR that are employed at vari- ous stages of recognition. Adding to the above, various image based with or without the glove employed for detection, their advantages and difficulties encountered dur- ing this process. Also, segmentation, feature extraction, methods used for feature vector quantization and reduction techniques are discussed in detail. Along with these, during classification it involves training, testing that employs various train- ing models, including Hidden Markov Model based approaches and Deep learning methods such as CNN, also techniques like k-NN, ANN, SVM and others. Then finally discussion on results and observations from several techniques are compared. The approaches that are being reviewed are so flexible that they can be employed for major sign detections with applications in various domains.

Keywords: Assistive technology, Gesture recognition, Feature extraction, Image segmentation, Training, Image color analysis, Face recognition

Drawbacks of above systems:

a. Less accuracy due to poor image processing system.

b. No auto correct feature.

c. No 3D Object support.

III. PROPOSED SYSTEM

A. Architecture

In proposed system, when a user will reach to our platform firstly he has to show a particular sign respective to what he wants to be converted in text. The input will be taken in form of image and further processed by using various algorithms like CNN. Then the word formation will be done after that suggestions for the respective word which user want to say will be shown on screen to reduce the communication delay and improve user experience. Along with words a whole sentence can also be shown on the screen. Where the output will be in form of Text and will be displayed on screen.

Implementation Details (Modules)

1. Creation of Training and Testing Dataset: In this module we will create Dataset manually and using resources from Internet. Which will be divided in ratio of 7:3 training to testing.
2. Training of Machine Learning Model: As we are using CNN, we will be using a library named Keras to train the neural network using training dataset to recognize hand gestures.
3. Capturing The Image with Web cam OR Other Camera Device: In this module, we will capture images from user device we will be using OpenCV to do so.
4. Pre-processing the Image: The aim of pre-processing is an improvement of the image data that suppresses unwilling distortions or enhances some image features important for further processing, although geometric transformations of images.
5. Prediction of Alphabet from the Model: Based on the image scanned if the sign is valid (represents an alphabet) then that result will be shown on user inter- face in form of text.
6. Suggestions and Creation of Sentences: Word suggestions will be shown on screen (just like suggestions section on google keyboard) and along with that whole sentence which user want to say through sign language will be shown in form of text.

Conclusion

It has been a great pleasure for me to work on this exciting and challenging project. This project proved good for me as it provided practical knowledge of Python and opencv. It also provides knowledge about the latest technology used in developing web enabled application and client server technology that will be great demand in future. This will provide better Opportunities and guidance in future in developing projects independently. We are trying to present a system for recognizing real-time Sign Language recog- nition models which portrays an impressive role in enhancing casual communication among people with hearing and speech disabilities and normal persons. We can therefore conclude, that the platform can provide immediate assistance to everyone at any place with and device with required specification, make communication between normal and differentially abled person easier through this sign language translation application and, in addition it also provides word suggestions just like google keyboard so that the user can communicate easily and delay in communication should not be there.

References

[1] Sign Language Recognition Techniques- A Review, Publisher: IEEE Cite This Mohammed Safeel; Tejas Sukumar; Shashank K S, 2020 [2] Real-Time Recognition of Indian Sign Language, Publisher: IEEE, Cite This H Muthu Mariappan; V Gomathi, 2019 [3] Mahesh Kumar N B, ”Conversion of Sign Language into Text”, in Interna- tional Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Num- ber 9 (2018) pp. 7154-7161 [4] Chen, Zhi-Hua and Kim, Jung-Tae and Liang, Jianning and Zhang, Jing and Yuan, Yu-Bo. (2014). Real-Time Hand Gesture Recognition Using Finger Segmen- tation.TheScientificWorldJournal. 2014.267872. 10.1155/2014/267872. [5] Dutta, Kusumika and Bellary, Sunny. (2017). Machine Learning Tech- niques for Indian Sign Language Recognition. 333-336.10.1109/CTCEEC.2017.8454988 [6] Raheja JL, Mishra A, Chaudary A. Indian Sign Language Recognition Us- ing SVM 1. Pattern Recognition and Image Analysis. 2016 September;26(2). [7] Goyal,Sakshi, Sharma,Ishita , Sharma, Shanu. Sign Language Recognition System For Deaf And Dumb People. International Journal of Engineering Research Technology (IJERT). 2013 April; 2(4). [8] Gurjal, P., Kunnur, K. Real Time Hand Gesture Recognition Using SIFT. International Journal of Electrics and Electrical. 2012; 2(3). [9] P.K. Athira, C.J. Sruthi and A. Lijiya ,”A Signer Independent Sign Lan- guage Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario”,Journal of King Saud University -Computer and Information Sciences. [10] Koller, O., Zargaran, O., Ney, H., and Bowden, R. (2016). Deep Sign:Hybrid CNN-HMM for Continuous Sign Language Recognition. In Proceedings of the “British Machine Vision Conference 2016. NewYork, UK

Copyright

Copyright © 2022 Suyog Bhamare, Abhishek Shinde, Rahul Palve, A. N. Gharu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.