A Review on Dynamic Hand Gesture Recognition Techniques

Abstract

Body language is one of the nonverbal methods of communication, and it comprises hand gestures, arm movements, posturing, and gestures and facial expressions. One way to communicate information through the movement of the body is through gestures. HGR is a smart, intuitive, and easy method of human-computer interaction (HCI). HGR systems have two key applications: SLR and GBC. To help the deaf communicate with the hearing community, SLR tries to automatically interpret SLs via a computer. The idea that SL is a highly ordered and primarily symbolic collection of human gestures is what led to the development of universal gesture-based HCI.

Introduction

I. INTRODUCTION

Typically, gestures entail moving the hands, face, or other body parts. It is a type of non- verbal communication in which specific directly observable actions are used in place of or in addition to words to convey certain information.

In fact, according to some researchers, communication in Homo sapiens originated from a manual gesture-based form of communication in the past. Gestural Theory, which has its origins in the works of the priest and philosopher Abbé de Condillac from the 18th century, was resurrected in 1973 as a part of a discussion on the genesis of language. Gestures can be static or dynamic. An appealing technique for enabling realistic human-computer interaction is direct hand input (HCI). While overcoming this restriction, vision-based solutions still have to deal with other issues caused by the user's body being partially obscured. Vision-based techniques can vary from one another depending on a variety of factors, including the quantity of sensors used, their delay and responsiveness, the organization of the surroundings, any user needs, the low-level characteristics used, and whether two-Dimensional or three- Dimensional representation is used.

A. Types of Gestures

Hand and Arm gestures: SL, HGR, entertainment applications. Head and Face gestures: Shaking/nodding one's head, eye gaze direction, arching one's brow, opening one's lips to speak, winking, flared nostrils, and expressions of surprise, as well as feelings like happiness, fear, disgust, wrath, sorrow, and disdain.

Body gestures: engagement of complete body movement, like recording the outside interactions of two individuals, assessing the dancer's motions to create music and images that fit, and identifying human gaits for athletic and medical training.

1. Sign Language: Deaf people frequently communicate through SL, a form of manual communication. The movements or symbols are organized linguistically in SL. A sign is a gesture that exists on its own. The hand shape, hand placement, and hand movement are the three distinguishing components of each sign. Conversation between people is transformed into HCI through a SLR system. The SL recognition system aims to provide deaf and hearing persons with a precise and efficient method of understanding text and voice making it easier for the deaf and hearing people to communicate. There is no globally recognized SL for the deaf. Furthermore, just like spoken languages, SLs vary from location to region and are not universal. It's interesting to note that most nations with identical spoken languages do not necessarily share the same SL. For instance, there are three types of English:ASL, BSL, and ASL. Despite the widespread assumption that gestures are not "genuine languages," they are just as linguistically complex and rich as any spoken language.

Various SLs have been examined by linguists in the field, and they have discovered that they include dual patterning and recursion in common with other languages. Languages are made up of fewer, useless components that may be merged to form bigger, meaningful ones. This is known as the duality of patterning. Recursion refers to the fact that languages have grammatical rules, and that a rule's output may also serve as its input.

II. LITERATURE SURVEY

With the help of motion-captured data, they hope to provide Labanotation scores. They start off by offering a reliable feature to obtain a helpful depiction of dance motions from the motion data after processing. The dancing movement segments are then used as input to instruct an HMM to match the Labanotation reference symbols for each type of lower leg motions. In particular, they suggest using a highly random tree technique to extract arm movements from upper limb data. In order to determine the notations in both the support columns and the arm columns, they employ the motion data. Finally, a dance piece's Labanotation score can be produced. Studies reveal that when compared to earlier methods, their system generates symbols with greater accuracy. This could lighten the workload. [1] The analysis proceeds the evolution of a DL system for diverse SL union, conversion, and identification. They decrypt the affair that accompanied by preceding SL characterization and recording synthesis methods by utilizing the recommended H-DNA architecture. Using the How2Sign, ISL-CSLTR, and RWTH- PHOENIXWeather 2014T datasets, they quantitatively and subjectively evaluated the model's performance. The suggested H-DNA structure is also qualitatively rated using a variety of variety criteria. Produced recording sequences demonstrate the high caliber of their work. They outperformed preceding approaches in terms of identification rate and initiate interpretation. Produced video segments demonstrate the high caliber of their work. They performed better than earlier methods in terms of detection rightness and output. Suggest method has impressive human evaluation scores, an average BLEU score of 38.56, an mean FID2vid tally of 3.46, an mean SSIM value of 0.921, an mean Inception Score of 8.4, an mean PSNR tally of 29.73, an mean FID tally of 14.06, and an mean TCM tally of 0.715. It also has a classification accuracy for SLR of over 95%. These results show a significant advancement over earlier models. Human assessors are used to assess the realism, relevance, and coherence aspects, and the outcomes are good in real-world circumstances.[2]

In order to enable simultaneous biosignal acquisition for biometric authentication across scattered electrode arrays, they synchronized the data sample timer in an anatomy detector network using the D-PkCOs procedure. By requesting just one collide package for every synchronization cycle, the D-PkCOs protocol decreases communication overhead. In order to reliably survey all BSN nodes susceptible to drifting clock frequency and changing processing latency, they employed a dynamic controller to alter the clock offset and skew for decreasing the sampling jitters. With this approach, the impacts of variable processing latency are automatically removed, and a more precise clock skew prediction is made for readjustment. Additionally, they employed the H control approach to create the D-PkCOs synchronization protocol's parameters. . Additionally, they employed the H control approach to create the D-PkCOs synchronization protocol's parameters. Because all nodes' sample errors are kept to a minimum in the BSN, the drifting clock and varying processing latency have no effect on the sampling jitters. With their D-PkCOs procedure, sampling jitters may be kept to under 1s in a 10-node IEEE 802.15.4 network, according to experimental results. It is shown that when DO-PkCOs are applied to the BSN, an HD-sEMG signal with a high SNR value develops, which improves the performance of gesture categorization.[3]

An acceptable smart glove that measures motion with an accelerometer and posture with a strain-sensitive resistive knit. In order to read data, extract features, and execute a pre-trained computational model, a microcontroller, a tiny customized PCB, and sensors are utilized. Real-time classification of SL stances and gestures is done using the system.This work highlights the potential of fusing cutting-edge microcontrollers, machine learning, and innovative soft sensors. Future research, however, will be able to describe the learning pipeline's capabilities, constraints, and architecture in greater detail.Future research should broaden the range of subjects studied to assess robustness and generalizability, including if the network can be set up and used by additional users immediately or if a tuning technique needs to be included.

They might as well examine elements like the user's level of hand skin conductance, ASL experience, size, etc. Results of cross validation may provide guide changes to the structure to increase robustness.

Adjusting Filtering or categorization windows may also cut down on latency. Examining the learning pipeline's capability and trade-offs is also important. Gestures might be added with very little effort. Memory or speed by by setting the softmax only up to the point where the LSTM's learning capacity is exhausted. How many scales depend on scaling the LSTM layer or adding layers? the microcontroller must carry out in order to analyze the network. How much is impacted by network size and gesture count as well. Data for training is necessary. To explain neural networks, these trade-offs between scale, velocity, precision, and training could be complex and application-specific.[4]

Novel graph spectral characteristics have been developed for dynamic form recognition. The proposed technique starts by pre-processing the input, such as movement of the hands, to generate a fully connected graph. Next, the suggested method analyzes the EV of the standardized Laplacian of the network adjacency matrix to produce the representative features. They used the eigenvector u0 because it serves as the primary representative feature and captures the specifics of the graph's structure. The suggested method outperformed the existing methods for three separate datasets, with accuracy values for numbers and symbols of 99.56% and 99.44%, respectively. The appropriate approach also has the characteristics of quick operation and rotational and flipping invariance.[5]

Brand-new air-handwritten password-based authentication mechanism for IoT devices is proposed. This method is an application of computer vision where it identifies the line which is drawn on the air by making use of a camera, pair light-weighted deep CNN models and a Kalman signal processing filter. This combination serves as the main differentiation between this framework and the alternatives.The findings demonstrated the acceptability and significance of the suggested authentication approach for accessibility metrics such user happiness, precision, and speed. The suggested approach is safe and resistant to dangers from physical observation. There are no equipment, wearable sensors, or depth cameras required for this procedure. Future iterations of the intended design will be quick, straightforward, and appropriate for managing devices like smart television, smart watches, smart refrigerators, and smart air conditioners. Suggested method's drawback is that it is ineffective in low light.[6]

A SNN-based FMCW radar-based gesture recognition system. They recommend taking the bandwidth, Doppler, and angle spectral analysis from video of RDIs and feeding them as characteristic pictures into their proposed SNN architecture. SNNs are intriguing for various applications involving human-machine interaction because they offer embedded minimal latency and power solutions. A significant problem while training the SNNs is improving the algorithms to learn multimodal peak trains. They demonstrate that the recommended peaking network, which has better learning principles and is significantly smaller in size, can match for eight motions. [7] Framework for automatic recognition of Arabic SL that makes use of a fresh ArSL Dataset that was captured on the grounds of their university. Three cameras—a Sony hand- held camera, a Kinect V1 camera, and a Kinect V2 camera—were used to record the dataset. Each 80-sign set was captured by the 40 petitioners five times, creating a sample with multiple modes that comprises depth photographs, RGB pictures, and bodily bones framework by V2 camera. In this study, they solely examined RGB pictures of the V2 camera by suggesting a sequential connection of both analogous networks—a 2-Dimensional convolutional grid and a second 1-dimensional convolutional skeletal grid. Their ideal network configuration picked up 88.89% of static indications and 98.39% of dynamic indications, which is highly encouraging for the outcomes of the automatic ArSL. The trial effectiveness for the signer-reliant approach was 89.62%, while the trial effectiveness for the signer-free mode was 88.09% when the exact system was conditioned using those static and dynamic signals. The usage of the reverse efficiency revealed accuracy-speed trade-off could have been made better with the appropriate amount if such models were used in production. [8]

For the recognition of SL, these deep neural models—SRN, DMN, and AMN—are suggested. The vital positions of the sign's principal poses are taught to the DMN stream. In this study, they suggest a method for addressing the differences between the sign samples by extracting key positions. This method makes use of the dominating postures that stand in for the important motion shifts in the symbol. They also recommended combining the sign's motions into a single picture using the AVM technique. The second suggested network, AMN, uses this image as its input. The characteristics out from DMN and AMN channels were pooled and utilized as input by the third network that was proposed, known as SRN. Two datasets were utilized to study these networks: the number of signers needed for histogram matching, bounding box computing, skin color separation, and region expansion; and signer free recognition is more difficult than signer-dependent recognition. Correlation-based comparison and feature point comparison are two gesture comparison techniques. Other aspects of the application include word to motion translation and word to voice out of word.[9]

A comprehensive hand gesture identification algorithm based on RGB-D by certain realistic hand motion communication with the digital world. After obtaining the hand gesture outline, the Distance Transform (DT) technique estimates the palm center for static hand motion recognition.

The fingernails are identified using the K-CCDD method. As supplementary features, the pixel spacing on the hand motion outline and the inclination between the fingers are utilized to generate a heterogeneous feature vector, and a specialized program is then introduced to accurately classify static hand movements. Furthermore, a cohesive explanation of each dynamic hand motion offers a IDTW approach to get dynamic hand gesture identification results by integrating Euclidean distance with bone modulus ratios separating the shoulder's center joints from the arm joints. They also develop an inexpensive live implementation of organic hand gesture interaction with the digital world. Finally, thorough trials are carried out to confirm and validate both the still and moving hand motion detection algorithms.[10]

III. METHODOLOGY

A. Input Preprocessing

A DL-Based Method for Recognizing SLR With Effective Hand Sign Representation. Using the Viola and Jones technique, the signer's face is located in the first method.

Trimming and spatial balancing lessen the influence of irrelevant characteristics in each frame. Contrarily, the second procedure trims and regulates the palm area to highlight more of the fingers arrangement. Two volumes per sample, each dimension 112 by 112 by 3 by 16, are produced during the pre - processing stage. The feature learning phase receives these two volumes, out of which one of them is dedicated to the hand region, another represents the whole gesture area.[11]

B. Feature Learning

Hand configuration's precise spatial and temporal properties are learned by the maiden C3D instance. Each input segment for this occasion favors The Hand. The second C3D instance, on the other hand, picks up on the coarse spatial and temporal properties of the whole-body setup. The result of this step is two feature vectors, each with 4096 size.[11]

C. Feature Fusion and Classification

After dimension reduction, we may acquire a precise representation of integrated features, resulting in reduced computing difficulty and higher face identification accuracy. Feature fusion aids in the complete learning of picture characteristics for the description of their rich internal information. Combining training picture features vector from the common weight network layer with extracted features made up of other numerical data allows the proposed model to use as many features as feasible for the subsequent classification.[11]

D. Gesture Spotting Algorithm

In a flow of frequently refreshed data, GSA locates the onset and termination of motions. The majority of earlier studies have suggested specific features that apply physical quantities like a tempo, an increase in tempo, or a recurrence in order to detect gestures. However, There are drawbacks to only using these "designed characteristics" for movement identification because There are too few of them to distinguish between genuine motions and other common fist movements like non-gestures or unintentional fist movements at their onset and termination. As a result, they suggested the new GSA. It employs a DL framework that can find more characteristics while being educated. The DL-based gesture identification method that has been developed measures the GPS, an unique notion described in this work. The scalar value between 0 and 1 represents the GPThe gS. This is used to indicate where a movement starts and finishes. For instance, When a gesture starts, the GPS score is virtually nil. On the contrary, when the movement is going to stop, the GPS score is very near to one. The sum of the speed standards for each time step is basically how the GPS is determined. The GPS may be mathematically described as follows.

jv(st) = |mj(st) − mj(st − 1)|, (st = 2, 3, . . . , ST , jv(1) = 0)

Vsum(st) = Xst i=1 jv(i)

GPS = { Vsum(1) Vsum(ST ) =0, Vsum(2) Vsum(ST ) , . . . , Vsum(ST ) Vsum(ST ) =1}

In (1), mj(st) ∈ R 10 is a metacarpophalangeal joint vector at time step t,

jv(st) ∈ R the second criterion of the disparity between the metacarpophalangeal joint vector at time steps st and st − 1. [12]

Sequence Simplification Algorithm: This introduces the SSA, which eliminates pace fluctuation in a succession of movements. A straightforward feature extractor, the SSA looks for significant modifications to the movement sequence. The SSA works as follows. Beginning with the first sensor out of ten, the algorithm determines which sensor has the highest disparity between the maximum and minimum points of the movement sequence being measured by the sensors.

The mathematical expression:

i ∗ = argmax i (max ts SS i (ts) − min ts SS i (ts))

i is the SSI, which lies between 1 to 10. In the movement sequence, SS is a collection of sensor readings, so SS i (ts) is the sensor output at time step ts measured by sensor i.

Second, It is defined as a line that runs across SS i's beginning and finishing locations. Third, The position of the data point with the greatest separation from the line is looked up. The following is a mathematical formulation for this:

k1 = argmax d(SS i ∗ (k), SS i ∗ (1)SS i ∗

(TE)) 1<k<TE

TE represents the end of the sensor assessment, and The range between position a and line b is determined by the function d(a, b).[12]
Gesture Recognition Algorithm: The SSA produces an abbreviated design, which the GRA uses to classify the movement. To increase the GRA's resistance to changes in the same movement that could not be completely ruled out after sequence reduction, a deep neural framework was used. The GRA consists of the following: an output layer of output aspect 11, three fully linked layers with 64 hidden units each, two LSTM layers with 64 hidden units, and three layers with 64 hidden units each. The output layer and the other three completely connected layers are activated by ReLU. Concatenation is not employed in the LSTM layer, which makes it different from the movement detecting method in certain ways. Analyzing the gesture sequence's context is crucial for the GRA. Therefore, the current hand shape does not need to be concatenated. The entire gesture sequence is not taken into account by the GRA. Since there are no more changes to the hand shapes after the GPS value crosses 0.9, it instead takes into account the gesture sequence from its inception to that point.[12]

CNN WITH IMPROVED PARALLEL SKELETAL

A SCN supplied by 3D data, which was initially inspired by the three dimension Intel Real sense SDK's coordinates X, Y, and Z. This kind of camera creates the hands' 3D important points within a 3D environment. The research findings revealed around 91.28% effectiveness covering around fourteen gestures in the hand gesture dataset, yet they also revealed certain shortcomings because of a decrease by a factor of 7% while there were twice as many classes totaling to 28. Their efforts to the redesigned SCN network focused on these key areas: First, they made the network architecture more straightforward by using 2D points rather than 3D, which required one-third smaller compute for each network. As a next step, we increased all input layers to include all 48 important areas on the body.[8]

1. Performance Benchmarks Following are the metrics we will use:

a. Precision       = True Positive rate/(True Positive rate + False Positive rate)

b. Recall           =  True  Positive rate/(True Positive rate + False Negative rate)

c. F1   score      = 2 *(precision* recall)/(precision + recall)

d. Accuracy   = True Positive rate/(True Positive rate + False Negative rate) [8]

2. IES

Providing the system several frames increases delay during the video detection procedure, while providing the system so very few frames lowers accuracy. The compromise between speed and precision has also been explored, and it has been determined that the IES may be applied in some circumstances. This following section is the most accurate amount of frames to transmit to our system. The following equation can be used to calculate

IES = Response Time 1 − Percentage Error

= Response Time PC

where PC is the percentage of correct answers. In this case, they suggest mapping RT to the frame count the system chooses. We are building our analysis on the assumption that frames propagate comparably fast from the camera to the initial OPL network. Once the fundamental prerequisite of a strong GPU is provided, according to the OPL design criteria for the optimum fps, the true aim may be attained. These various phases are added together to get the total pipeline latency:

a. Frame capturing

b. Elongation

c. OPL

d. Frame queuing for key locations

e. SKN infrastructure sign decision

A Gaussian distribution is a round-the-clock likeliness allocation which is totally narrated by pair of conditions:

the mean (a) and variance (b2).

Gaussian distribution formula is expressed as Equation

f (x,a,b) = 1 b √ 2π e −1 2 (x−a)^ 2 .b ^2

For instance, incase we acquire the Y position of a certain point 30 times/second, and every time it returns a number between 17 and 23.Hence, established on basis of Gaussian distribution

The Kalman filter algorithm recapitulate the forecasting phase and inform phase for every

latest data points, like following phases: Forecast phase: The latest points forecast based on preceding computation point and motion phase, like in Equation

Forecast Point approximation = Preceding Point + Motion Phase

Though, the points and motion phase are established on the basis of Gaussian distribution, which has a mean and variance (error rate). Therefore, the latest point is the grand total of duplet Gaussians, as in Equation: GD(A1, B1) = GD(Ai, Bi) + GD(Am, Bm )

= GD(Ai + Am, Bi + Bm)

where Ai and Bi are the mean and variance (error rate) of the preceding point, discreetly. Where Am and Bm are the mean and variance (error rate) of the Motion Phase, discreetly. The Forecast Point is a Gaussian distribution with the mean that equivalent (Ai +Am) and the variance that equivalent (Bi + Bm).[6]

Inform phase: The inform phase is used to extract the point by simply increasing the duo Gaussian distributions, Forecast Point point approximation GD (A1, B1), and the calculation or present position GD (A2, B2). The outcome of the amplification is Gaussian or foremost position approximation GD (A, B), and its mean is the nearest position of the accurate position. Then, the foremost position approximationGD (A, B) is used in the next Forecast Phase. [6]

A neural network that can categorize segmented samples or streaming real-time data is created using the pipeline's processing of gathered training data.

• TRAINING DATA COLLECTION
• DATA PROCESSING AND FEATURE

EXTRACTION:

• Segmentation Into Labeled Example.
• Data Augmentation
• Feature Extraction: Smoothing and Normalization

• NEURAL NETWORK TRAINING
• RESULTS AND DISCUSSION[4]

Conclusion

Here we discuss various hand gesture recognition algorithms and methods. The use of hand gesture recognition systems is thought to lead to more effective and intuitive tools for human-computer interaction. Applications span from sign language interpretation to virtual prototyping to medical education. One means of communication for those who are physically disabled, deaf, or dumb is sign language. The aforementioned analysis shows that the field of hand gesture identification has advanced significantly thanks to vision-based hand gesture recognition.

References

[1] Li, Min, Zhenjiang Miao, and Cong Ma. \"Dance movement learning for labanotation generation based on motion-captured data.\" IEEE Access 7 (2019): 161561-161572. [2] Natarajan, B., et al. \"Development of an End-to-End Deep Learning Framework for Sign Language Recognition, Translation, and Video Generation.\" IEEE Access 10 (2022): 104358-104374. [3] Zong, Yan, et al. \"Robust Synchronized Data Acquisition for Biometric Authentication.\" IEEE Transactions on Industrial Informatics 18.12 (2022): 9072- 9082. [4] DelPreto, Joseph, et al. \"A Wearable Smart Glove and Its Application of Pose and Gesture Detection to Sign Language Classification.\" IEEE Robotics and Automation Letters 7.4 (2022): 10589- 10596. [5] Alwaely, Basheer, and Charith Abhayaratne. \"Graph spectral domain feature learning with application to in-air hand- drawn number and shape recognition.\" IEEE Access 7 (2019): 159661-159673. [6] Elshenawy, Abdelghafar R., and Shawkat [7] K. Guirguis. \"On-Air Hand-Drawn Doodles for IoT Devices Authentication During COVID-19.\" IEEE Access 9 (2021): 161723- 161744. [8] [9] Arsalan, Muhammad, Avik Santra, and Vadim Issakov. \"RadarSNN: A Resource Efficient Gesture Sensing System Based on mm-Wave Radar.\" IEEE Transactions on Microwave Theory and Techniques 70.4 (2022): 2451-2461. [10] Bencherif, Mohamed A., et al. \"Arabic sign language recognition system using 2D hands and body skeleton data.\" IEEE Access 9 (2021): 59612-59627. [11] Luqman, Hamzah. \"An Efficient Two- Stream Network for Isolated Sign Language Recognition Using Accumulative Video Motion.\" IEEE Access 10 (2022): 93785- 93798. [12] Xu, Jun, et al. \"Robust Hand Gesture Recognition Based on RGB-D Data for Natural Human-Computer Interaction.\" IEEE Access (2022). [13] Al-Hammadi, Muneer, et al. \"Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation.\" IEEE Access 8 (2020): 192527-192542. [14] Lee, Minhyuk, and Joonbum Bae. \"Deep learning based real-time recognition of dynamic finger gestures using a data glove.\" IEEE Access 8 (2020): 219923-219933.

Copyright

Copyright © 2023 Mownika Reddy K A , Prajna Harish , Sambhrama K , Vinayak Bhat , Dr. Deepak G , Dr. Harish Kumar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.