A Robust Business Specific Real-Time Sign Language Translator

Abstract

Communication is a great way of expressing yourself but not everyone is capable of communication. No, not everyone chooses to be silent, While some are born deaf and mute, others become one later in life due to certain conditions. The word “dumb” is quite offensive so I will address to someone who cannot speak as mute person throughout this paper. Returning to the topic this paper will enable a more effective way to fill the communication gap between deaf and mute person and normal person especially in places with crowd where special-abled person becomes nervous and anxious to make the business owner understand of his needs easily and are unable to express themselves. In order to remove this barrier and enable the effective communication between the special-abled buyer and the business owner, I have created our own data set of hand gestures and trained using Google Teachable Machine for common sentences that are exchanged between the buyer and seller.

Introduction

I. INTRODUCTION

Do special-abled people matter to the world? How would you think you can communicate to the special-abled person who cannot convey a message to you like a normal person? Do you know India constitutes around 2.5 million of deaf and mute population, which holds the world’s 20% of deaf and mute population?

This population lacks the amenities which normal people own. They have their own daily life problems that a normal person cannot even think of.

Normal person don’t learn sign language neither it is taught in schools and sign language is the only language that deaf and mute community can speak and understand. The problem is same as two different persons with different languages trying to communicate. So, how are these special-abled people are going to communicate?

To overcome this problem, we introduce a unique solution. Our model will be business specific. Meaning of which is that for example, in a clothing store the general sentences exchanged are [1] How many meters You want? [2] I will buy it. [3] I don’t want. [4] Make cheaper [5] 0-9 etc. and in a fruit store the general sentences exchanged are [1] How much You want? [2] I will take it. [3] I don’t want. [4] Make cheaper. [5] 0-9 [6] dozen etc. So this way the owner and buyer will not have problem understanding each other and they don’t need to remember the signs because it will be displayed on the screen and will pave effective way of communication between special-abled and business owner.

Various empirical studies have been conducted that introduced solution for the communication but they are either not very much practical in real life or are not affordable and larger special-abled population cannot use it.

While efforts are being made to mitigate this problem so, In this solution I introduce a deep learning model which accepts special gestures that are created by us and translates it into sentences so the normal person can understand. Basically, this project focuses on visual way of communication where the exchange of thoughts is done by using vision of the eye.

Further, This paper discusses the problem statement where you are going to explore more on the problem in the existing models that tried to fill this gap of expressing between two different language understanding spoken persons and how those gap can be filled in this paper, then goes on to describing the mathematical model to train the gestures and parameters to mimic this paper and later we will end this paper with discussion and the conclusion we made.

II. PROBLEM STATEMENT

AI technology and internet led to the inclination of the global Smart tools is apparently supported by and provided with more opportunities by the development of AI models which in turn is based on the concept of deaf and mute communication. Fortunately, the development in modern information systems, digital technologies, the universal access to technology development make the system more applicable.

Some other papers address this same problem of communication in different ways but are unable to reach for the larger population of deaf and mute usage due to their way of implementation and have some disadvantages. Following are some of the papers with different approach from each other and their disadvantages and the reason due to which it is not feasible for general population-

Today there is a growing interest in technology and different researchers with different background and different knowledge level try to solve one specific problem in different way to make it to the general population use.

As we have seen in the table above that different researcher led to different way of solving the same problem with different disadvantages and advantages.

I also solved the same problem with different approach. This is what I did, I created my own hand gesture data set apart from ASL because normal people are not taught hand signs and since we do not get to communicate in sign language on a daily basis so normal person may forget the sign language easily.

So due to the idea of creating own data set of gestures which will be displayed on the screen as in the image below, normal person does not need to remember the gestures and since the gestures are representing general sentences shared in a specific business place just to get the work done easily for deaf and mute population, the numbers of gestures will be less (around 25-30 gestures).

III. TRAINING MODEL

Requirements:

Tensorflow 2.8.0

keras 2.8.0

cv2 4.4.0

To create the model, capture the dataset using Google Teachable Machine and train the model using below parameter values.

Data – 1500 image per gesture

You can increase the number of training data and set the other parameters accordingly.

Epoch – 50

bath size – 16

learning rate – 0.001

After the model is trained, download the h5 file that is created with all the weights.

Use that file in an opencv project to detect and translate the gestures.

Success:

In this project, I have implemented a business specific sign language translation system that would assist the deaf and mute in effective communication with the normal people. The system can be used effectively in shops and residence or near the schools of deaf and mute person where the communication is more frequently needed.

IV. RESULTS AND DISCUSSION

Our proposed system works as follows: The business owner will place a screen connected to a camera and CPU where this program will run.

Next, the deaf or mute person will make the hand gesture in front of the camera which will be shown on the screen.

The Gesture will then be translated into specific sentences.

So, this way the two person with different language understanding can communicate with each other.

The best part is the deaf and mute population will not need to wear the electronic wearable which are created before and is not very practical in real life.

The proposed system make use of the visual ability to communicate.

Conclusion

In the proposed system, I have implemented an organization-oriented system that would assist deaf and mute person to communicate with a normal person easily. Without having to worry about anything and the hardware resource required will be low and most importantly, this model will serve larger population of deaf and mute as the required hardware is easily affordable.

References

[1] Sign Language Translator, International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl. [2] Microsoft ASP.NET Official Site. Available from: http://weblogs.asp.net/bleroy/archive/2004/08/03/Don_270 0_t-redirect-aftersetting-a-Session-variable-_2800_or-do-it- right_2900_.aspx [Accessed 9 August 2007]. [3] A New Data Glove Approach for Malaysian Sign Language Detection;2015 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS 2015) [4] Martin, Tony.; Selly, Dominic (2002), \'Visual Basic.NET At Work: Building 10 Enterprise Projects\', New York John Wiley & Sons, Inc. [5] Tatnall, Arthur (2005), \'Web Portals: The New Gateways to Internet Information and Services\', Hershey, PA: Idea Group Publishing. [6] Designing a Sign Language Translation System using Kinect Motion Sensor Device; International Conference on Electrical, Computer and Communication Engineering (ECCE), February 16-18, 2017, Cox’s Bazar, Bangladesh [7] Kanter, Rosabeth Moss (2000), \'Evolve!: Succeeding in the Digital Culture of Tomorrow\', Boston, Mass. Harvard Business School Press. [8] Das, Souripriya; Chong, Eugene Inseok; Eadon, George; Srinivasan, Jagannathan (2004), \'Supporting Ontology-based Semantic Matching in RDBMS\'. Paper presented at Very Large DataBase conference, Oracle Corporation. [9] Sign Language Recognition Using Leap Motion Sensor; International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056 Volume: 03 Issue: 11 | Nov -2016 [10] Vinarski.H-Peretz, Binyamin.G, and Carmeli.A,(2011). \"Subjective relational \"Subjective relational innovative behaviors in the workplace.\" Journal of Vocational Behavior, vol. 78, no. 2, pp. 290-304.

Copyright

Copyright © 2022 Waiz Khan. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.