A Survey on Sign Language Recognition with Efficient Hand Gesture Representation

Abstract

Image classification is one amongst classical issue of concern in image processing. There are various techniques for solving this issue. Sign languages are natural language that want to communicate with deaf and mute people. There\\\'s much different sign language within the world. But the most focused of system is on Sign language (SL) which is on the way of standardization there in the system will focused on hand gestures only. Hand gesture is extremely important a part of the body for exchange ideas, messages, and thoughts among deaf and dumb people. The proposed system will recognize the number 0 to 9 and alphabets from American language. It\\\'ll divide into three parts i.e., pre-processing, feature extraction, classification. It\\\'ll initially identify the gestures from American Sign language. Finally, the system processes that gesture to recognize number with the assistance of classification using CNN. Additionally, we\\\'ll play the speech of that identified alphabets. Keywords: Hybrid Approach, American Sign Language, Number Gesture Recognition. Feature Extraction.

Introduction

I. INTRODUCTION

Sign Language may be a best way of communication between deaf, dumb people and normal people. A sign language is visual language that uses system of manual, facial and body movements. Sign language isn't universal language and different sign languages are utilized in different countries, some countries like UK, USA and India having over one Sign Language. Many Sign Languages are using in World, American language, Indian signing, Japanese language, Turkish Sign Language these are some samples of Sign Language. American Sign Language is that  a part of Sign Language which is usually utilized by deaf and dumb people in India, Signs are depending upon the regional language so in India there's nobody standard Sign Language.

Artificial Neural Network (ANN), a brain-style computational model, has been used for several applications. Researchers have developed various ANN’s structure in accordant with their problem. After the network is trained, it may be used for image classification. The Support Vector Machine may be a theoretically superior machine learning methodology with great leads to classification of high dimensional datasets and has been found competitive with the simplest machine learning algorithms. within the past, CNNs are tested and evaluated only as pixel-based image classifiers. Moving from pixel-based techniques towards object-based representation, the size of remote sensing imagery feature space increases significantly. These results increasing complexity of the classification process, and causes problems to traditional sample-based classification schemes. In this study, we've developed a unique approach of recognizing hand number gestures by recognizing or labelling hand parts exhaustive images. Our proposed approach consists of two main processes: hand parts recognition by random forests (RFs) classifier and rule based hand number gestures recognition. the most advantage of our proposed approach is that the state of every finger is directly identified through the recognize hand parts and so number gestures are recognized supported the state of every finger.

II. RELATED WORK

This work includes the feature extraction, gesture recognition process for that they have proposed a novel approach for recognizing hand number gestures using the recognized hand parts during a depth image. In this paper they proposed a system in that approach they divided that approach into two stages: (i) Hand parts recognition by random forests (RFs) and (ii) rule-based hand number gestures recognition. This contains pairs of depth maps and their Corresponding hand pats labelled maps was generated after that they used DB to training RFs. within the second stage, a depth image was first captured from a depth camera so a hand depth silhouette was extracted by removing the background. Next, the hand parts of a depth silhouette were recognized using the trained RFs. Next, a collection of features was extracted from the labelled on the extracted features, labelled hand parts finally they used rule-based approach to acknowledge number gesture (2014) [1].                                                                                                     This work includes the techniques like thresholding, feature extraction; on the help of this feature, they proposed a straight forward method for recognition of numbers. They used thresholding value for recognition of numbers. They divide that method into three stages i.e. first they captured image by using web camera after capturing image they applied threshold value on it and using that threshold value they recognized the numbers (2011) [2].                                                                                                                        This work contains the features like image-preprocessing, HOG, feature extraction, gesture recognition on the basis of that they need propose a system for decomposition of gestures into single handed or double handed gesture. They classifying gesture into these subcategories simplify the method of gesture recognition within the ISL due to presence of lesser number of gestures in each sub category. They used Histogram of Gradients (HOG) features and geometric descriptors using KNN and CNN classifiers were tried on a dataset consisting of images of 26 English alphabets present within the ISL under variable background. The HOG features when classified with Support Vector Machine were found to be the most efficient approach results of this method in term of accuracy is 94.23 [3].                                                                                                                              

In this proposed system they need made application for people who those vocal and hearing disabilities. It discusses an improved method for sign language recognition and conversion of speech to signs. The algorithm devised is capable of extracting signs from video sequences under minimally cluttered and dynamic background using skin colour segmentation. It distinguishes between static and dynamic gestures and extracts the suitable feature vector. These are classified using Support Vector Machines. Speech recognition is built upon standard module Sphinx. Experimental results show satisfactory segmentation of signs under diverse backgrounds and comparatively high accuracy in gesture and speech recognition (2016) [4].

This work presented a dynamic hand gesture recognition system for household appliance control using only the depth camera. The dynamic hand gesture is recognized using static hand postures and hand trajectory. The proposed system can recognize seven commonly used dynamic hand gestures. Experimental results show that the system is effective for household appliance control [5].  

This paper introduces a novel hand gesture recognition scheme supported depth data. The hand is firstly extracted from the acquired depth maps with the help also of color information from the associated views. Then the hand is segmented into palm and finger regions. Next, two different set of feature descriptors are extracted, one supported the distances of the fingertips from the hand center and also the other on the curvature of the hand contour, finally, a multiclass CNN classifier is utilized to recognize the performed gestures [6].                                                                                                                                                     

In this context, the geometric moments and therefore the orthogonal moments namely the Zernike, Tchebichef and Krawtchouk moments are explored. The proposed system detects the hand region through skin color identification and obtains the binary silhouette. These images are normalized for rotation and scale changes. the instant features of the normalized hand gestures are classified using a minimum distance classifier [7].                                                                                                                             This work focuses on building a robust part-based hand gesture recognition system using Kinect sensor. To handle the noisy hand shapes obtained from the Kinect sensor, we propose a unique distance metric, Finger-Earth Mover’s Distance (FEMD), to measure the dissimilarity between hand shapes. because it only matches the finger parts while not the entire hand, it can better distinguish the hand gestures of slight differences [8].                                                                                                                       

Propose method is to recognize the image-based numbers of Persian linguistic communication (PSL) using thinning method on segmented image. during this approach, after cleaning thinned image, the real endpoints are used for recognition. the method is qualified to produce real-time recognition and isn't affected from hand rotation and scaling [9].                                                                  This work discusses a simple recognition algorithm that recognizes the numbers from 0 to 10 using thresholding. the general algorithm has three main steps: image capture, apply threshold and recognizing the number. The idea is created that user must wear color hand gloves [10]

III. PROBLEM STATEMENT

The first problem is Lack of normal datasets for Sign Language. Another problem is that the variance in sign language with locality and usage of various symbols for the identical alphabet by the identical person.

 IV. OBJECTIVES

A. To overcome existing drawback of sign language recognition system

B. To recognize number as well as alphabets.

C. To develop Number Recognition System using gesture.

D. To develop Alphabet Recognition System using gesture.

E. To develop low-cost system with high performance using thinning and CNN approach.

                                                                                      V. EXISTING RESULT

VI ALGORITHM

The propose work is to target the hybrid approach of algorithms for classification of numbers and alphabets. Proposed model incorporates four phases i.e. Preprocessing, Feature Extraction, Classification, Recognition. In the proposed model the method of thing and CNN algorithm going to use, CNN for classification of feature extraction to classify the numbers and alphabets. The primary section of the proposed model is consisting of Image capturing or acquisition and preprocessing of the image. The very first step is capture image through a camera or through video. After getting image resize that image and extract hand part from that image. Remove the noise if present and converted into Binary Image.

The second section is accommodating Feature Extraction, there are many features available for gesture recognition but this technique will be concentrating on figure tip and active-inactive fingers using thinning techniques.

After getting features it'll provides it to CNN. CNN classified the features. CNN classified that feature for the higher accuracy and compare with training dataset and gives the output. Additionally, we'll play the speech of that identified alphabets.

VII. PROPOSED SYSTEM ARCHITECTURE

Conclusion

Hand gesture recognition for real-life applications is extremely challenging due to its requirements on the robustness, accuracy and efficiency. In this work good comparative study has been adopted supported the methodology proposed using thinning algorithm and support vector machine. This can help to improve the recognition accuracy of number gestures and also recognize the active finger’s names and positions. In our proposed system, we process the input gesture in preprocessing phase, thinning method for feature extraction and finally use CNN for recognizing the exact gesture name, alphabets and number. Additionally, we’ll convert these alphabets into speech. The effectiveness of the design in a real-world situation has been demonstrated by a physical implementation of the system. This method work will be completely done by using python and it’ll be successfully executed.

References

[1] Munir Oudah, Ali Al-Naji, Javaan Chahl, “Hand Gesture Recognition Based on Computer Vision: A Review of Techniques”,2020 [2] Ali Moin, Andy Zhou, Alisha Menon, Simone Benatti, George Alexandrov, Senam Tamakloe, Jonathan Ting, Natasha Yamamoto, Yasser Khan, Fres Burghardt, Luca Benini, Ana C Arias, Jan M Rabaey “A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition”,2021 [3] Abdullah Mujahid, Mazhar Javed Awan, Awais Yasin, Mazin Abed Mohammed, Robertas Damasevicius, Rytis Maskeliunas, Karrar Hameed Adbdulkareem, “Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model”,2021 [4] Yong Soon Tan, Kian Ming Lim, Chin Poo Lee, “Hand gesture recognition via enhanced densely connected convolutional neural network”,2021 [5] Priyanka Parvathy, Kamalraj Subramaniam, G.K.D. Prasanna Venkatesan,P.Karthikaikumar, Justin Varghese, T. Jayasankar, “Development of hand gesture recognition system using machine learning”, 2020 [6] Yong Soon Tan, Kian Ming Lim, Connie Tee, Chin Poo Lee, Chenn Yaw Low, “Convolutional neural network with spatial pyramid pooling for hand gesture recognition”,2020 [7] Manisha Kowdiki, Arti Kharpade, “Automatic hand gesture recognition using hybrid metaheuristic-based feature selection and classification with Dynamic Time Warping”,2021 [8] Gibran Benitez-Garcia, Lidia Prudente-Tixteco, Luis Carlos Castro-Madrid, Rocio ToscanoMedina, Jesus Olivares-Mercado, Gabriel Sanchez-Perez, Luis Javier Garcia Villalba, “Improving Real-Time Hand Gesture Recognition with Semantic Segmentation”, 2021 [9] Wen Qi, Salih Ertug Ovur, Zhijun Li, Aldo, Marzullo, Rong Song,” Multi-Sensor Guided Hand Gesture Recognition for a Teleoperated Robot Using a Recurrent Neural Network”,2021 [10] Fangtai Guo, Zaixing He, Shuyou Zhang, Xinyue Zhao, Jinhui Fang, Jianrong Tan, “Normalized edge convolutional networks for skeleton-based hand gesture recognition”,2021 [11] Sakshi Sharma, Sukhwinder Singh “Vision-based hand gesture recognition using deep learning for the interpretation of sign language”,2021 [12] Hira Ansar, Ahmad Jalal, Munkhjargal Gochoo, Kimbum Kim , “Hand Gesture Recognition Based on Auto-Landmark Localization and Reweighted Genetic Algorithm for Healthcare Muscle Activities”, 2021 [13] Janmenjoy Nayak, Binghnaraj Naik, Pandit Byomkesha Dash, Alireza Souri, Vimal Shanmuganathan,” Hyper-parameter tuned light gradient boosting machine using memetic firefly algorithm for hand gesture recognition”,2021 [14] Yinfeng Fang, Xuguang Zhang, Dalin Zhou, Honghai Liu,” Improve Inter-day Hand Gesture Recognition Via Convolutional Neural Network-based Feature Fusion”, 2021 [15] Shweta Saboo, Joeeta Singha, Rabul Hussain Laskar, “Dynamic hand gesture recognition using a combination of two-level tracker and trajectory-guided features”, 2021 [16] Qing Gao, Yongquan Chen, Zhaojie Ju, Yi Liang, “Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction”, 2021

Copyright

Copyright © 2022 Priyanka Gaikwad, Kaustubh Trivedi, Mahalaxmi Soma, Komal Bhore, Prof. Richa Agarwal. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.