Automatic Speech to Sign Language Software for Translation (ASSiST) for Hearing-Impaired

Abstract

A total of 63 million people in India have hearing impairment, which is a common cause of disability. Due to communication barriers, these individuals are at risk for reduced cognitive skills and language deficits which may contribute to their poor general and oral health. Instructing these individuals regarding oral health concerns, treatment options and prognosis is often challenging for dental professionals. An Automatic Speech to Sign Language Translation Software (ASSiST) was developed to improve brushing technique among children with hearing impairment. The programming of Automatic Speech Recognition (ASR) system was done using Linear Predictive coding (LPC) algorithm for extraction of Speech Signals and Artificial Neural Networks as a classifier and recognizer. This tool is tested to help imparting oral hygiene instructions in the form of a sequence-based brushing technique to the children chair side which when spoken by the dentist translates it into the respective regional sign language which is displayed on the screen.

Introduction

I. INTRODUCTION

A total of 63 million people in India have hearing impairment, which is a common cause of disability [1]. It can be due to infections, exposure to ototoxic agents, anoxia, prematurity, and trauma. Psychological and social disturbances, especially communication, become more pronounced with the increasing severity of impairment [2]. Due to communication barriers, these individuals are at risk for reduced cognitive skills and language deficits which may contribute to their poor general and oral health [3]. Instructing these individuals is often challenging because of the unfamiliar communication means used. Sign Language (SL) is a form of communication which is widely used by the speech and hearing impaired.

It is difficult for people to communicate with hearing impaired people if they don’t know the Sign Language and vice-versa. Learning this language maybe time-consuming, expensive and the tutor or instructor maybe not available all the time. Hence, there is a need to have a self-tutorial available to better communicate with those with hearing or speech impairment.

The present work is focused on the difficulty faced between a doctor and a patient. Instructing these individuals regarding oral health concerns, treatment options and prognosis is often challenging for dental professionals. Oral hygiene is the most important measure in the maintenance of good oral health and prevention of oral diseases. Tooth brushing among other mechanical methods is the most used and effective method for cleaning most tooth surfaces, thus maintaining oral hygiene [4,5]. In children less than 10 years of age, due to lack of motivation, poor manual dexterity, irregular brushing pattern and mechanical brushing habit with least attention, tooth brushing habit is less efficient [6]. For tooth brushing to be effective, coordinated muscular movements, psychomotor skills, hand function ability and an ability to understand instructions are necessary [7]. Tooth brushing is considered to be the most widespread technique for mechanical plaque control which aids in the long-term maintenance of periodontal health. It is an economic, effective and convenient method which is widely accepted and practiced [8].

In children with hearing impairment, the task of educating them to follow the correct method of tooth brushing becomes a challenging task for the care givers as well as dentists. These children learn and understand differently as compared to children with normal hearing. Hence, in the oral health perspective, children with hearing impairment are ought to be at a great risk for oral health problems [9]. Oral hygiene practices, an inability to access oral care facilities, dietary patterns, medication, attitudes of parents and health providers and physical limitations have majorly contributed to this high risk [10]. In these special children, due to communication problems, it is necessary to modify their knowledge and attitude toward dentistry by imparting education through proper communication.

Hence, an automatic speech to sign language software is needed to improve brushing technique among children with hearing impairment. This tool can help in imparting oral hygiene instructions to the children chair side which when spoken by the dentist translates it into the respective regional sign language which is displayed on the screen.

As the mode of communication plays an important role in imparting education, there is a need for professional modification of the ways of communication in educating these children. Oral hygiene education in written instructions and video modeling are some examples to bridge the communication barrier [11]-[13]. Various other methods of communication available for hearing-impaired children with are sign language, finger spelling, lip reading, cochlear implants and hearing aids, written information, and cued speech [2],[11],[12],[14]. Automatic Speech Recognition (ASR) is the technology in which machine understands the meaning of the human speech.

Poor oral hygiene and increased risk of caries due to unmet treatment needs is highly prevalent among hearing impaired individuals [15]. Oredugba, studied the knowledge and practices of a group of deaf adolescents regarding oral health care through a questionnaire-based evaluation in Lagos, Nigeria. The study involved 50 adolescents aged 10-19 years. Majority of these adolescents had not been to the dentist for a check-up or treatment. The authors attributed their non-attendance of disabled children could be due to social and economic deprivations. They recommended that school based oral health education programs will help impart knowledge regarding oral healthcare and improve their oral health practices [16].

A cross-sectional study conducted by Wei et al in China, revealed that prevalence of dental caries was 55.9% in children with hearing impaired as compared to 13.8% in children with normal hearing. They observed poor oral health and lack of knowledge of oral hygiene practices in children with hearing impaired possibly due to relatively lower proportion of awareness on dental health knowledge among this group [17].

Jain et al, in a cross-sectional study assessed the prevalence of caries and treatment needs amongst participants ranging 5-22 years of age in Udaipur, Rajasthan, India. They concluded that young people with hearing impairment have poor oral hygiene, a high prevalence of dental caries and unmet treatment needs. About 87.4% of the subjects required dental treatment and there was a prevalence of 83.2% of decayed teeth [18].

A study conducted in a special school in Bhimavaram demonstrated that prevalence of dental caries was 65% and that of treatment needs was 91.7%. They recommended that motivational and prevention-based programs will help to achieve adequate knowledge regarding oral hygiene and improve their oral hygiene levels [19].

A questionnaire-based study by Khaled et al, was done to

determine the extent of oral health-related knowledge in hearing impaired group of patients in Saudi Arabia. It was concluded that 83% did not receive any instructions regarding good oral health, 79% had no knowledge about how to brush their teeth, and 92% of the participants were not aware of oral and dental hygiene practice [20].

In another study by Saikiran et al, educational as well as therapeutic intervention to improve oral hygiene status was conducted. They evaluated the effect of educational intervention using sign language and video modeling and therapeutic intervention using liquorice on status of oral hygiene in children with hearing impairment. They noted a significant reduction in the oral hygiene, gingival and plaque scores in all the children [21].

Furthermore, a study was conducted in order to impart oral hygiene knowledge to children with hearing impairment by using tailor-made education tools handcrafted flipcharts, models to show tongue cleaning and leaflets. By evaluating their oral hygiene scores, it was concluded that these educational aids were visually appealing and helped children with hearing impairment to understand and comprehend better [22].

A special approach was reported by Rehanan et al, for the dental management of a 6-year old child with hearing impairment. Different modes such as sign language with actions, models, pictures and rating scales were used. They recommend that a well-constructed protocol will serve in the successful management of deaf children [23]. A number of rules on how to communicate with hearing impaired children in a dental clinic was given by Alsmark et al [24].

Lev Vygotsky et al. suggested that any method that visually please children with hearing impairment will significantly alter their attitude towards learning [25]. It is therefore necessary to teach these children in a customized way according to their learning needs [26].

Oi-Mean Foong et.al proposed a voice to sign language translation system using Speech and Image processing technique. System is trained with speech pattern based on some generic spectral parameter set which is stored as template in a database.

Template-based Recognition is done by matching the input speech parameter set with the stored templates to finally display the sign language in video format. Empirical results show that the system has 80.3% recognition rate [27].

Shekainah Paulson et.al proposed a speech to sign language translation system for Malaysian Deaf people. The speech recognizer converts the spoken sentences into utterances and silences, and is recognized as a text, and a video of the sign language interpretation of the spoken sentence is displayed [28]. Lei Wang et al. proposed a Cloud-based Automatic Speech Recognition System for Southeast Asian Languages. The strategies and guidelines of collecting scripted speech data and text data were presented. The acoustic model used in this is a HMM-DNN hybrid model [29].

II. PROPOSED METHOD

The proposed speech to sign language system architecture is shown in Fig. 1. The automatic speech recognition system presented here has two phases namely Feature extraction and Training and Testing.

The Automatic Speech to Sign language System is implemented on a general-purpose computer along with a noise cancellation microphone using MATLAB and NCH Wavepad Editor.

A. Data Base creation and training of neural network

1. Recording and Pre-processing

A standard microphone is used to record the spoken word in a closed noise free room. Sampling is done at a rate of 8000 Hz. The isolated word or the speech is recorded in the form of a wav file. This file is then filtered to remove any kind of distortion or noise that may be present.

After removing the noise, by keeping the threshold, number of samples will be reduced greatly. It is passed through high pass filter to boost the high frequency components, since the high frequency components are more susceptible for noise. Array of speech samples are then divided into many number of overlapping blocks and each block is multiplied by the hamming window. This is done because if we consider the block directly for the analysis there will be sudden discontinuity at the ends. So, to remove the sudden discontinuities the ends are tapered using the hamming window. The filtered signal is passed to the feature extraction block.

2. Feature Extraction

Linear Predictive Coding (LPC) and MFCC’s are usually used to extract the features of speech signal for recognition. This work has utilized the LPC method of feature extraction. The LPC technique attempts to derive the coefficients of a filter that would produce the utterance that is being processed. Its ability to extract and store time-varying formant information is the reason LPC is useful in speech signal processing.

In LPC of speech, the nth sample of speech is approximated as a linear combination of past output speech samples. The LPC coefficients captures almost all formant frequencies. Hence, we can effectively represent the voice samples through a lesser number of coefficients.       Equation for Linear Predictive coding can be written as,

Where,xn+1  is the predicted value of next value of input x(n+1) and w (0), w (1) …... w(p-1) are the weights to be multiplied for a ‘P’ length filter to obtain xn+1 . It is called Linear Prediction because, here we are trying to predict the next samples as linear combination of ‘P’ Previous samples.

The auto-correlation matrix and hence the LPC coefficients is given by the equation (2).

 rx(1)rx(2)?rx(p)=rx(0)rx*(1)rx(1)rx(0)???rx*(p-1)?rx*(p-2)?              ?rr(p-1)???rr(0)w(0)w(1)?w(p-1)     (2)

3. Artificial Neural Network Training

Artificial neural networks can be classified in two significant groups, feedforward networks and recurrent networks.

A feedforward network is a directed and weighted graph that contains no feedbacks. The input neurons contain no connection leading to them, and the output neurons contain no connections leading away from them.

A recurrent neural network is a directed and weighted graph that contains at least one feedback loop. The error calculated in the Feed Forward Network is propagated back to update the weights in order to match the obtained output to the desired value which is called back propagation algorithm.

In the proposed ASR technique for the purpose of constructing a classifier, a multilayer recurrent artificial neural network with 228 input layers (LPC coefficients for each word),100 hidden layers and 50 output layers was chosen. The LPC coefficients obtained are used to train the neural network.

4. Recognition

During recognition phase, spoken word is recorded with the same sampling rate i.e.,8000 samples/sec. The signal is processed to remove the silence, minimisation of the noise and normalization of the signal is done. The various features that are extracted from a word spoken are fed to the recognizing component and based on the closest match the corresponding text and sign language GIF is displayed.

Fig. 2 shows the testing input speech signal for word ‘UJJIRI’, Fig. 3 shows the normalized test signal and Fig. 4 shows the LPC trajectory of testing signal for the same word. The testing is carried out through MATLAB which allows user to speak the word to be converted and displays the recognized word in respective sign language and Kannada text form as shown in Fig. 5.

III. RESULTS AND DISCUSSION

Number of speakers chosen for training the neural network are 15 including 7 female speakers where in each word was spoken by every person 5 times and the system was tested for recognizing 82 different words with normal speakers (5 speakers including 3 female) with good pronunciation ability. Results are presented in Fig. 6. In the present context, if the input word is recognized correctly then the result of the test is true positive. If the speech input is correct and recognized word is incorrect then, the result of the test is true negative.

It was found that at least 52 out of the 82 words were true positive and the desired sign language video was obtained. That means the system is at least 60% accurate to display the correct sign language.

The effectiveness of this system was assessed by comparing this teaching method with a live demonstration of the sequence-based tooth brushing method. Sixteen children who participated in this study were randomly assigned to two groups.

1. Group A: Educational intervention using speech to sign language translation system
2. Group B: Educational intervention using live tooth brushing demonstration on a model

A self-designed questionnaire containing 8 close-ended questions was prepared and was provided to the study participants at baseline and after the intervention to evaluate the improvement in knowledge regarding the sequence-based tooth brushing technique. The responses of the questionnaires revealed that children with hearing impairment in this study had a poor knowledge regarding the brushing technique. In Group A, the mean was 3.38 out of 8.0 which was slightly increased to 6 after receiving the oral hygiene instructions through the automatic speech to sign language translation software. The mean in Group B increased from 3.38 before being taught using the live demonstration on the model to 6.5 after it. This improvement in the knowledge score was significantly better in Group B (live demonstration of tooth brushing on the model) (P < .0001).

This automatic speech to sign language system was successful in significantly improving knowledge regarding the sequence-based tooth brushing technique before and after the instructions were given using this novel system.

When it comes to children with hearing impairment, dental professionals need to be aware of the unique set of challenges concerning the establishment of communication with them [21]. Since sign language is their natural way of communicating, it seems to be the best way to communicate with these children. For most children with hearing impairment who lose their hearing before they learn to speak, the primary language is sign language and written/spoken language becomes their second language. This system is related to the values and culture of the hearing-impaired community. It is dependent on signs recognized nationally and regionally where words are spelt out using different shapes representing different letters of the alphabet [30].

Conclusion

Natural Language to sign language translation is the main scope of this research. This paper presents an ASR technique for individuals with hearing impairment for Kannada native language. Here, LPC is used for feature extraction method and ANN as a classifier. Compared with previous works, ASR system developed has given a better recognition rate. And also, the system is speaker independent and gender independent. It can be used for normal speakers, as well as speakers with partial speech disorder. The scope of the project was of recognizing 50 isolated words and it’s been achieved with maximum possible accuracy. Hence, for further work, we can increase the vocabulary size and can be modified and implemented for other languages as well. It can be concluded that providing a suitable means for communication with hearing impaired children will positively influence their level of understanding of the sequence-based brushing technique. Even though the participants in the live demonstration group had a significantly higher overall mean knowledge score, the automatic speech to sign language software was successful in significantly improving knowledge regarding motion of tooth brushing and the tooth brushing technique before and after the instructions were given using this novel system.

References

[1] Naik SM, Naik MS. Rehabilitation of hearing impaired in India-an update. Online J Otolaryngol; 3(1):20-31, 2013. [2] Champion J, Holt R. Dental care for children and young people who have a hearing impairment. Br Dent J; 189:155-159, 2000. [3] Year 2000 position statement: principles and guidelines for early hearing detection and intervention programs. Joint Committee on Infant Hearing, American Academy of Audiology, American Academy of Pediatrics, American Speech-Language-Hearing Association, and Directors of Speech and Hearing Programs in State Health and Welfare Agencies. Pediatrics; 106:798-817, 2000. [4] Simmons S, Smith R, Gelbier S. Effect of oral hygiene instruction on brushing skills in preschool children. Community Dent Oral Epidemiol;11(4):193–198, 1983. [5] Das UM, Singhal P. Tooth brushing skills for the children aged 3–11 years; 27(2):104–107, 2009. [6] Unkel JH, Fenton SJ, Hobbs G Jr, Frere CL.Toothbrushing ability is related to age in children. ASDC J Dent Child; 62: 346-348, 2015. [7] Ogasawara T, Watanabe T, Kasahara H. Readiness for tooth brushing of young children. ASDC J Dent Child; 59(5):353–359, 1992. [8] Tan E, Daly C. Comparison of new and 3-month-old toothbrushes in plaque removal. J Clin Periodontol; 29: 645-650, 2002. [9] Swallow JN. The dental problems of handicapped children. J R Soc Promot Health; 85:152-157, 1965. [10] Kamatchy KRJ, Joseph J, Krishnan CGA. Dental caries prevalence and experience among the group of institutionalized hearing-impaired individuals in Pondicherry. Indian J Dent Res; 14:29-32, 2003. [11] San Bernardino-Alsmark S, de Nova-García J, Mourelle-Martinez MR, Gallardo-Lopez NE. How to improve communication with deaf children in the dental clinic. Med Oral Patol Oral Cir Bucal; 12:576-581, 2007. [12] Davis A, Hind S. The impact of hearing impairment:a global health problem. Int J Pediatr Otorhinolaryngol; 49:51-54, 1999. [13] Sandeep V, Vinay C, Madhuri V, Rao VV, Uloopi KS, Sekhar RC. Impact of visual instruction on oral hygiene status of children with hearing impairment. J Indian Soc Pedod Prev Dent; 32:39, 2014. [14] Samnieng P. Dental cares for patients who have a hearing impair- ment. Int J Clin Prev Dent; 10:215-218, 2014 [15] Fageeh HN, Mansoor MA. The effectiveness of oral hygiene instructions in sign language among hearing impaired adults in Saudi Arabia. Spec Care Dent; 40(1):41–8, 2020. [16] Oredugba FA. Oral health care knowledge and practices of a group of deaf adolescents in Lagos, Nigeria. J Public Health Dent; 64(2):118–20, 2004. [17] Wei H, Wang YL, Cong XN, Tang WQ, Wei PM. Survey and analysis of dental caries in students at a deaf-mute high school. Res Dev Disabil; 33(4):1279–86, 2012. [18] Jain M, Mathur A, Kumar S, Dagli RJ, Duraiswamy P, Kulkarni S. Dentition status and treatment needs among children with impaired hearing attending a special school for the deaf and mute in Udaipur, India. J Oral Sci; 50(2):161–5, 2008. [19] Sandeep V, Kumar M, Vinay C, Chandrasekhar R, Jyostna P. Oral health status and treatment needs of hearing-impaired children attending a special school in Bhimavaram, India. Indian J Dent Res; 27(1):73–7, 2016. [20] Khaled K, Faris Y. Basic practices of oral hygiene and awareness of oral and dental disease among deaf and dump population in Saudi Arabia. Dentistry; 7(Suppl):10, 2017. [21] Saikiran KV, Kamatham R, Sahiti PS, Nuvvula S. Impact of educational (sign language/video modeling) and therapeutic (Glycyrrhiza glabra––liquorice mouth wash) interventions on oral health pertaining to children with hearing impairment: A randomized clinical trial. Spec Care Dentist; 1–10, 2019. [22] Ashwini K, Kushali S, Midhuna M, Shravani G, Sharanya R, Aishwarya A, et al. Assessment of Oral Hygiene of Children with Speech and Hearing Impairment Using Tailor-Made Oral Health Education Tools. J Dent Oro-facial Res;15(01):15–21, 2019. [23] Renahan N, Varma RB, Kumaran P, Xavier AM. Unique Approach to Dental Management of Children with Hearing Impairment. Int J Clin Pediatr Dent;10(1):107-110, 2017. [24] Alsmark SSB, García J, Martínez MRM, López NEG. How to improve communication with deaf children in the dental clinic. Med Oral Patol Oral Cir Bucal; 12(8):576–81, 2007. [25] Vygotsky LS. Principles of social education for deaf and dumb children in Russia. Int Confrence Educ Deaf; 227–37, 1925. [26] Hoferková, R. Deaf Students Learning English as a Foreign Language, Bachelor Thesis. Masaryk University Brno, Department of English Language and Literature. 2012. [27] Mean Foong O, Low TJ, La WW. V2S: Voice to sign language translation system for Malaysian deaf people. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics); 5857 LNCS (January 2015):868–76, 2009. [28] Paulson S, Thilagavathi B. An Adaptable Speech to Sign Language. Int J Eng Res Technol; 3(3):1813–5, 2014. [29] Wang L, Tong R, Leung CC, Sivadas S, Ni C, Ma B. Cloud-based automatic speech recognition systems for southeast asian languages. Proc 2017 Int Conf Orange Technol ICOT 2017.;2018-January:147–50, 2018. [30] Fitzpatrick EM, Hamel C, Stevens A, et al. Sign Language and Spoken Language for Children with Hearing Loss: A Systematic Review. Pediatrics; 137(1): 20151974, 2016. [31] Nandlal B, Shanbhog R, Godhi BS, Sunila BS. Change in Skills Observed with a Novel Brushing Technique Based on Sequence Learning; Evaluated Through Video Bio-Feedback System in Children. Oral Hyg Health; 1: 115, 2013.

Copyright

Copyright © 2022 Divya N Sahetya, Nandlal Bhojraj, Shreekanth T, Shreepriya Singhania, Samhita Bijlani. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.