Audio to Sign Language Translator

Abstract

[1] Einarson TR, Piwko C, Koren G. Quantifying the global rates of nausea and vomiting of pregnancy: a meta-analysis. J Popul Ther Clin Pharmacol. 2013;20(2):e171–e183. [PubMed] [Google Scholar] [2] Jansen LAW, Koot M, Hooft J, Dean C, Bossuyt PMM, Ganzevoort W, et al. Grooten, Iris. The Windsor Definition for Hyperemesis Gravidarum: a multistakeholder International Consensus Definition. Eur J Obstet Gynecol Reprod Biol. 2021:266. 10.1016/j.ejogrb.2021.09.004. [PubMed] [3] London V, Grube S, Sherer DM, Abulafia O. Hyperemesis Gravidarum: A Review of Recent Literature. PubMed. 2017;100(3-4):161–171. doi: 10.1159/000477853. [PubMed] [CrossRef] [Google Scholar] [4] Popa SL, Barsan M, Caziuc A, et al. Life-threatening complications of hyperemesis gravidarum. Exp Ther Med. 2021;21(6):642. doi: 10.3892/etm.2021.10074. [PMC free article] [PubMed] [CrossRef] [Google Scholar] [5] Boelig RC, Barton SJ, Saccone G, Kelly AJ, Edwards SJ, Berghella V. Interventions for treating hyperemesis gravidarum. Cochrane Database Syst Rev. 2016;11(5):CD010607. doi: 10.1002/14651858.CD010607.pub2. [PubMed] [CrossRef] [Google Scholar] [6] Maslin K, Dean C. Nutritional consequences and management of hyperemesis gravidarum: a narrative review. Nutr Res Rev. 2021;16:1–11. doi: 10.1017/S0954422421000305. [PubMed] [CrossRef] [Google Scholar] [7] Dean CR, Bierma H, Clarke R, et al. A patient–clinician James Lind Alliance partnership to identify research priorities for hyperemesis gravidarum. BMJ Open. 2021;11:e041254. doi: 10.1136/bmjopen-2020-041254. [PMC free article] [PubMed] [CrossRef] [Google Scholar] [8] Havnen GC, Truong MBT, Do MLH, Heitmann K, Holst L, Nordeng H. Women’s perspectives on the management and consequences of hyperemesis gravidarum – a descriptive interview study. Scand J Prim Health Care. 2019;37(1):30–40. doi: 10.1080/02813432.2019.1569424. [PMC free article] [PubMed] [CrossRef] [Google Scholar] [9] Heitmann K, Nordeng H, Havnen GC, Solheimsnes A, Holst L. The burden of nausea and vomiting during pregnancy: severe impacts on quality of life, daily life functioning and willingness to become pregnant again - results from a cross-sectional study. BMC Pregnancy Childbirth. 2017;17(1):75. doi: 10.1186/s12884-017-1249-0. [PMC free article] [PubMed] [CrossRef] [Google Scholar] [10] Poursharif B, Korst LM, Fejzo MS, MacGibbon KW, Romero R, Goodwin TM. The psychosocial burden of hyperemesis gravidarum. J Perinatol Off J Calif Perinat Assoc. 2008;28(3):176–181. doi: 10.1038/sj.jp.7211906. [PubMed] [CrossRef] [Google Scholar]

Introduction

I. INTRODUCTION

Sign language is used by deaf-mute people. In the overall world there are 1.5 billion people who cannot hear. Overall, in the world there are many sign languages. Many countries have their own sign languages this project we discuss about Indian sign language. Sign language is a mother tongue to deaf people. It does not require voice to communicate. Body movements and sign gestures are used to convey the message to deaf people. Sign language is a combination of body, hand, arm movements and facial expressions. It can be used as a communication language between deaf, dumb people. Normal people can use it to converse with deaf people.

This application takes speech as input, converts it into text and then displays the Indian Sign Language images.

1. The front end of the system is designed using EasyGui.
2. Speech which is taken as input through microphone uses PyAudio package.
3. The speech is recognized using Google Speech API.
4. The text is then pre-processed using NLP (Natural Language Processing).
5.  Finally, Dictionary based machine translation is done.

II. PROCEDURE

A. Audio Input is Taken

The user's audio input is collected initially using a microphone or another audio input device, and it is then transformed into a text string format by a machine translation process. Speech is recognized using google speech API. Following the conversion, the converted string is processed with the Python string module and compared to the corresponding output image in the ISL dataset.

B. The Input English Text is Parsed

Knowing the grammatical structures of the source and target languages is necessary to do rule-based language conversion. The way to learn this grammatical structure is by parsing. Three alternative outputs are possible with the Stanford parser: part-of-speech-tagged text, type dependency representation, and context-free grammar representation of phrase structure. The English sentence is parsed using Penn tree tags.

C. Elimination of Stop Words

Since ISL deals with words associated with some meaning, unwanted words are

removed these include various parts of speech such as TO, POS (possessive ending),

MD(Modals), FW (Foreign word), CC (coordinating conjunction), some DT (determiners like a, an,

the), JJR, JJS (adjectives, comparative and superlative), NNS, NNPS (nouns plural, proper

plural), RP (particles), SYM (symbols), Interjections, non-root verbs

D. Video Conversion

The programme will next look for matches from the data set that is available for each word once the stages have been completed and we have the ISL transformed text. This will be based on the fundamental string-matching method between the labels of the movies and the processed input text. Finally, a display of several videos is shown on the screen in sequential order.

III. LITERATURE SURVEY

A. Text to sign language and vice versa computer vision-based organization in Marathi

One of the fastest-growing areas of research today is sign language recognition, which is also the most natural form of communication for those with hearing impairments. A hand gesture recognition system can enable deaf people to interact verbally with hearing people without the use of an intermediary or interpreter. The Marathi sign language is automatically recognized by the system. providing instruction in Marathi for the purpose of educating deaf sign users. The system can teach new users who are not familiar with sign language, and the instruction will be given offline. It uses a database with predefined sign language alphabets and words to assist users learn sign language. The suggested system to identify isolated words from the common Marathi sign language that are photographed has been trained on a sizable sample set. There are 46 Marathi sign language alphabets in the system, and about 500 sign language words are used. Considering all the alphabets and words used in sign language, the database has 1000 unique gesture images. The suggested approach aims to translate some very fundamental sign language components from sign to text and vice versa.

B. In India, children with hearing impairment receive rehabilitation

There is a large frequency of deafness in India. The second most frequent reason for impairment is this. 6.3% of India's population, or 63 million people (about twice the population of California), have substantial hearing loss.1 In India, rehabilitation of children with hearing impairment is still a difficult effort.

The cornerstones of this initiative are early detection and intervention. The clinical features and surgical management of children with hearing impairment are not covered in length in this article.

Here, we go through the options and resources available in India for deaf children's schooling as well as the function of government agencies in rehabilitation. Low levels of public and even medical professional awareness exist about the education and rehabilitation of hearing-impaired people.

C. Study of Sign Language Translation using Gesture Recognition

 Communication is an integral part of human life. But for people who are mute & hearing impaired, communication is a challenge. To understand them, one must either learn their language i.e., sign language or finger language. The system proposed in this project aims at tackling this problem to some extent. In this paper, the motivation was to create an object tracking application to interact with the computer and develop a virtual human computer interaction device. There are two reasons behind this system. It operates in two different modes: Teach and Learn. Utilizing contour recognition [3], the project uses a webcam to identify the hand positions and signs produced, and it then outputs the Sign Language in PC onto the motion made. This will translate the gesture caught by the webcam into auditory output so that regular folks may comprehend what is being said. To convert sign language into text and audio, we created the Sign Language to Speech Converter project.

D. "Mental Arithmetic and Conceptual Understanding: The Pedagogical Struggle of Mathematics Education for the Deaf in the Late Nineteenth Century"

A consultative e-user group, made up of ministry employees, teacher educators, educational researchers, NGOs, foundations, and other development partners, provided guidance and assistance, commented on the draught initial report and draught final report, and answered questions in their respective fields of competence. The websites of significant governmental and non-governmental organizations were also searched, citations cited in identified papers were followed up on, and team members, the e-user group, and the team's professional contacts were consulted for recommendations of pertinent studies and "grey" unpublished reports and papers. Nine electronic databases for relevant literature and 17 key journals were manually searched.

IV. ACKNOWLEDGEMENT

We would like to acknowledge the contribution of Ankita Harkude and their research paper,” Audio to Sign Language Translation for Deaf People", which provided valuable insights and inspiration for our work. Their thought analysis and thoughtful conclusions helped to guide our own research and we are grateful for their important contributions to the field.

Conclusion

One of the helpful methods for facilitating communication between the deaf and mute populations and mainstream culture is sign language. Although sign language can be used to communicate, the recipient must understand sign language, which is not always achievable. This was intended to be a prototype to determine whether it would be possible to recognize sign language. The text will be translated into sign images so that regular people can communicate with deaf or dumb groups.

References

[1] Amit kumar shinde and Ramesh Khagalkar “sign language to text and vice versa reorganization using computer vision in Marathi” International journal of computer Application (0975-8887) National conference on advanced on computing (NCAC 2015). [2] Sulabha M Naik Mahendra S Naik Akriti Sharma \" Rehabilitation of hearing-impaired children in India \"International Journal of Advanced Research in Computer and Communication Engineering. [3] Neha Poddar, Shrushti Rao, Shruti Sawant, Vrushali Somavanshi, Prof. Sumita Chandak \"Study of Sign Language Translation using Gesture Recognition\" International Journal of Advanced Research in Computer and Communication Engineering Vol. 4, Issue 2, February 2015. [4] Christopher A.N. Kurz \"The pedagogical struggle of mathematics education for the deaf during the late nineteen century: Mental Arithmetic and conceptual understanding\" Rochester Institute of Technology, Rochester, NY USA. Interactive Educational Multimedia, Number 10 (April 2005), pp. 54-65.

Copyright

Copyright © 2023 Pallepati Sreelekha, Aedabaina Devi, Kurva Pooja, S. T. Ramya. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.