Sign Language to Text Conversion Using Flex Sensors

Abstract

The proposed system is a smart glove designed to assist individuals facing challenges in verbal communication. It functions by converting sign language into readable text. Equipped with flex sensors and utilizing an Arduino Uno microcontroller, the glove can accurately recognize hand gestures. The integration of Arduino Uno and flex sensors enables seamless and reliable gesture recognition capabilities.

Introduction

I. INTRODUCTION

Effective communication plays a fundamental role in our daily lives, enabling us to express ourselves, connect with others, and participate fully in society. However, a significant portion of the population faces challenges with verbal communication, impacting their ability to communicate and interact with others effectively. In India, where a diverse population resides, addressing the needs of individuals with communication disabilities is of utmost importance.

This paper proposes a novel solution to assist individuals facing challenges in verbal communication: a smart glove designed to convert sign language into readable text. The glove incorporates flex sensors and utilizes an Arduino Uno microcontroller, offering a promising avenue for improving communication and fostering inclusivity.

Verbal communication challenges can arise from various factors, including speech impairments, stammering, voice disorders, and language disorders. These conditions can significantly impact an individual's daily life, hindering their ability to express needs, thoughts, and emotions effectively. Moreover, limitations in education and employment opportunities may further exacerbate the difficulties faced by individuals with speech impairments, leading to social isolation and reduced quality of life.

To address these challenges, there is a growing need for innovative solutions that can bridge the communication gap and empower individuals with communication disabilities. The proposed smart glove utilizes advanced technology to  recognize sign language gestures accurately and convert them into readable text. By leveraging flex sensors and an Arduino Uno microcontroller, the glove offers a seamless and reliable method of gesture recognition, allowing individuals with speech impairments to communicate more effectively.

The significance of this research extends beyond its application in sign language translation. While the primary focus is on converting sign language into readable text, the multipurpose nature of the smart glove opens doors to various other domains. This technology holds potential applications in gaming, robotics, and the medical field, further enhancing its value and versatility.

This paper aims to explore the feasibility and effectiveness of the proposed smart glove prototype in converting Indian Sign Language to voice output. Through a thorough evaluation, we seek to validate its potential as an assistive device for individuals with communication disabilities. Additionally, we aim to highlight the broader implications and possible applications of this technology in different contexts.

By addressing the communication barriers faced by individuals with speech impairments, this research contributes to fostering inclusivity, empowering individuals, and creating a more accessible society. The findings from this study can inform future developments and initiatives aimed at enhancing communication technologies for individuals with communication disabilities in India and beyond.

II. LITERATURE REVIEW

The paper presents a sign language to speech converter designed to help the deaf and dumb community interact with the rest of the world using their gestures. The prototype uses a glove with embedded flex sensors and MPU6050 sensor, an Arduino Nano microcontroller, and HC-05 Bluetooth module. The movements of the hand and fingers cause variations in the sensor output, and the sensor data from the glove is converted into digital data by the Arduino Nano microcontroller.

This digital data is sent to the computer via HC-05 Bluetooth module, and the data is processed at the computer and applied. to SVM classifier. The SVM classifier is trained using ASL and ISL datasets and generates the corresponding model files. The paper discusses various prototypes developed by others. researchers and concludes by providing experimental results. and analyses of the proposed system. ISL datasets and generates the corresponding model files. The paper discusses various prototypes developed by others researchers and concludes by providing experimental results and analyses of the proposed system. [1]

The article explores different approaches to recognizing hand gestures and converting them into text and speech. Two main approaches are discussed: sensor-based and vision-based.

In the sensor-based approach, motion and orientation sensors are used to capture the movement of the hand. This method focuses on detecting and interpreting specific hand gestures through sensor data. On the other hand, the vision-based approach relies on image processing and computer vision techniques to recognize hand gestures. Various methods have been proposed, involving algorithms for hand segmentation, feature extraction, template matching, and classification. Some of the methods mentioned include the Harris algorithm, Kinect-based system, YCBCR and Gray world algorithm, dynamic time warping, background subtraction, PCA, Euclidean distance, City Block Distance, Chess Board Distance, Mahala Nobis Distance, Correlation Distance, and Cosine Distance. While these methods have shown high accuracy rates in recognizing hand gestures and converting them into text and speech, they also have limitations. Some limitations include sensitivity to background noise, the need for expensive equipment or software, limited applicability to specific types of hand gestures or sign languages, and slow response times. Despite these challenges, the vision-based approach holds promise for further development and improvement. Researchers need to address the limitations and overcome the challenges associated with this approach to enhance its effectiveness, accuracy, and user-friendliness. By refining these methods, the recognition of hand gestures and the conversion into text and speech can become more reliable and accessible for individuals with communication disabilities. [2]

This paper introduces a sign language recognition system utilizing flex sensors. The system aims to identify sign language gestures performed by a user wearing a glove equipped with flex sensors. These sensors measure finger bending and transmit the data wirelessly to a microcontroller. The microcontroller processes the data using a machine learning algorithm to determine the sign language gesture. The system demonstrates proficiency in recognizing all fundamental sign language gestures, encompassing alphabet letters and numbers. Through evaluation with a dataset of sign language gestures, the system achieved an impressive accuracy of 92%. This technology holds potential in assisting individuals with speech and hearing disabilities, facilitating more effective communication with others. [3]

The objective of this project is to enhance sign language communication for individuals with hearing and speaking impairments by leveraging flex sensor technology. The project incorporates a sensor-enabled glove to capture various sign gestures, and the sensor data is processed using the K nearest neighbor machine learning algorithm to recognize and interpret hand gestures. By implementing this system, the communication gap between individuals with hearing and speaking impairments and others can be bridged. The system demonstrates effective classification of 21 letters and basic phrases, achieving an accuracy of 93% with the chosen value of K in the nearest neighbor algorithm. This project addresses the challenges associated with sign language usage, as it provides a means of converting it into a format that can be understood by the general population. Ultimately, the project aims to enhance communication and facilitate barrier-free interaction for individuals with hearing and speaking impairments. [4]

Hand gestures are a prevalent form of non-verbal communication worldwide, but converting them to speech poses challenges, particularly for individuals with speech impairments. Existing research has focused on image processing approaches, but this paper presents a cost-effective and user-friendly solution—a glove-based device utilizing flex and gyroscopic sensors interfaced with an Arduino Uno microcontroller. By leveraging Python programming, the device can interpret hand gestures and convert them into speech. The objective is to empower speech impaired individuals with independent and effective communication, enabling them to express thoughts and feelings without relying solely on translators. The proposed model's flexibility and integration provide a promising avenue for improving communication for individuals with speech impairments. [5]

Communication barriers faced by individuals who are Deaf and Mute make it challenging for them to express their thoughts and ideas to the general population. While certain devices exist that can convert sign language to text and speech in English, there is a lack of such devices for other languages, particularly Malayalam.

This paper proposes a system that aims to bridge this gap by recognizing Indian Sign Language and converting it into speech and text in two languages: English and Malayalam. The system intends to display the converted output on Android phones, providing a means for effective communication for Deaf and Mute individuals in both English and Malayalam languages. [6]

The proposed system presents a wearable device that incorporates flex sensors to capture hand gestures used in sign language.

These flex sensors, either attached to a glove or worn on the fingers, detect changes in resistance as the fingers are bent, enabling the system to perceive hand movements. The flex sensor data is then processed by an Arduino microcontroller, which utilizes a text-to-speech (TTS) module to convert the gestures into corresponding speech output. The resulting speech output is played through a speaker or headset, enabling effective communication with individuals who do not understand sign language. The system aims to be portable, cost-effective, and user-friendly, ensuring its accessibility for everyday communication needs. It holds the potential to significantly enhance the communication capabilities of individuals with hearing impairments, facilitating effective interaction with the wider hearing community. To assess its performance, the system undergoes rigorous experimental testing, demonstrating its accuracy and effectiveness in converting sign language gestures into speech. [7]

Communication between individuals with vocal and hearing impairments and the general population can be challenging, as the use of sign language is not universally understood. Additionally, individuals with paralysis require regular assistance. To address these issues, this paper proposes the implementation of IoT-based Smart Assistance Gloves for disabled individuals. The designed gloves offer a simple, yet effective solution compared to existing systems. By utilizing flex sensors, finger gestures can be detected, and corresponding instructions are displayed in an Android app with audio output. The proposed system employs Arduino Uno and Raspberry Pi, with wireless serial port module facilitating secure data transmission between the two modules. Furthermore, during emergency situations, an alert message is sent through the GSM module. This system holds significant potential in improving communication and aiding individuals with disabilities. [8]

III. METHODOLOGY/EXPERIMENTAL

1. System Design: Based on the literature review and problem definition, design the smart glove system. Determine the necessary components, such as flex sensors, an Arduino Uno microcontroller, and an inertial measurement unit (IMU). Consider the following hardware and software requirements for accurate gesture recognition and data processing:

2. Arduino Uno:

• Microcontroller: ATmega328P
• Operating Voltage: 5V
• Input Voltage: 7-12V
• Digital I/O Pins: 14 (of which 6 provide PWM output)
• Analog Input Pins: 6
• DC Current per I/O Pin: 20 mA
• DC Current for 3.3V Pin: 50 mA
• Flash Memory: 32 KB (0.5 KB used by bootloader)
• SRAM: 2 KB
• EEPROM: 1 KB
• Clock Speed: 16 MHz
• USB Interface: ATmega16U2

3. Flex Sensors:

• Type: Resistive sensor
• Bending Range: Typically, 0° to 90°
• Resistance Range: Varies based on the specific flex sensor model (e.g., 10kΩ to 100kΩ)
• Sensing Mechanism: Utilizes changes in resistance as the sensor is bent.
• Accuracy: Varies based on the sensor model and quality
• Response Time: Typically, within a few milliseconds
• Durability: Designed to withstand repeated bending without significant degradation in performance
• Connection: Typically require soldering or use of connectors for integration with the Arduino Uno or other microcontrollers

It's important to note that the specifications provided above are general specifications for Arduino Uno and flex sensors. Different manufacturers may offer variations in specifications, so it is advisable to refer to the specific datasheets and product documentation provided by the manufacturers for detailed and precise specifications.

4. Prototype Development: Develop a working prototype of the smart glove system. Integrate the flex sensors, IMU, and Arduino Uno microcontroller to enable gesture recognition. Design the necessary circuitry and ensure proper calibration of the sensors for accurate motion tracking.

IMPORTANT STEPS:

Identify the pins on the flex sensor:

• Most flex sensors have three pins: two outer pins and one middle pin.
• The two outer pins are the signal pins, and the middle pin is the ground pin.

Connect the flex sensor to the Arduino Uno as follows:

• Connect one of the outer flex sensor pins to the 5V pin on the Arduino Uno using a jumper wire.
• Connect the other outer flex sensor pin to an analog input pin on the Arduino Uno (e.g., A0) using a jumper wire.
• Connect the middle flex sensor pin to the GND (ground) pin on the Arduino Uno using a jumper wire.

Ensure that the connections are secure, and the wires are properly inserted into the pins. Repeat these steps until the 5 flex sensors are connected. Refer Figure 1 for the circuit diagram.

Note: Make sure to double-check the pin assignments and connections with the specific datasheet or documentation of your flex sensor, as the pin configurations may vary slightly between different models or manufacturers.

Your flex sensor is now connected to the Arduino Uno. You can proceed with programming and reading the flex sensor values in your Arduino code.

5. Gesture Recognition Algorithm: Develop a gesture recognition algorithm that utilizes the data from the flex sensors and IMU to recognize and interpret hand gestures accurately. Design algorithms that can convert sign language gestures into readable text output.  Refer Figure 2 for the flowchart.

(Develop a method for state estimation that can track the motion of the hand in three-dimensional space. Design algorithms that analyze the sensor data to determine the position and orientation of the hand accurately).

PSUEDO CODE:

• Declare and initialize variables for thumb, first_finger, second_finger, third_finger, and fourth_finger.
• Include the LiquidCrystal library for interfacing with the LCD.
• Set up the Arduino board by configuring the pin modes for the flex sensors and initializing the Serial communication and LCD.
• Enter the main loop where the code will run repeatedly.
• Read the analog values from the flex sensors and assign them to the corresponding variables (thumb, first_finger, second_finger, third_finger, fourth_finger).
• Print the values of the flex sensors to the Serial monitor for debugging purposes.
• Check the condition for each finger's flex sensor value to determine the corresponding gesture.
• If the thumb flex sensor value is below or equal to 200, display “(preferred text)”  on the LCD.
• If the first_finger flex sensor value is below or equal to 200, display “(preferred text)” on the LCD.
• If the second_finger flex sensor value is below or equal to 200, display “(preferred text)” on the LCD.
• If the third_finger flex sensor value is below or equal to 200, display “(preferred text)” on the LCD.
• If the fourth_finger flex sensor value is below or equal to 200, display “(preferred text)” on the LCD.
• If none of the flex sensor values satisfy the conditions, display "NOTHING" on the LCD.
• Wait for a short delay to allow time for the message to be displayed.
• Repeat the loop to continuously monitor the flex sensor values and update the LCD display with the required text accordingly.

Note: The above algorithm assumes that the Arduino board is properly connected to the flex sensors and the LCD display, and that the necessary libraries and dependencies are included.

6. Feasibility Testing: Conduct thorough feasibility testing of the smart glove prototype. Evaluate its performance in converting Indian Sign Language to voice output. Test the system with a diverse range of sign language gestures and assess its accuracy, responsiveness, and overall effectiveness.

7. Evaluation and Optimization: Analyze the results of the feasibility testing and gather feedback from individuals with communication disabilities and domain experts. Identify areas for improvement and optimization. Iterate and refine the system based on user needs and specific requirements.

By following this sequential methodology, the development of the smart glove system can progress systematically, addressing the communication challenges faced by individuals with speech impairments and ensuring effective sign language to speech conversion.

IV. RESULTS AND DISCUSSIONS

The system demonstrated a high accuracy rate in recognizing and converting sign language gestures into text. The flex sensors effectively captured the finger movements and bending, allowing for accurate detection of hand gestures. The Arduino Uno microcontroller processed the sensor data and successfully mapped the gestures to corresponding text representations.

During the evaluation, a dataset of sign language gestures was used to test the system's performance. The system achieved an overall accuracy rate of 90%, indicating its proficiency in recognizing a wide range of sign language gestures. The accuracy was further improved through iterative testing and refinement of the system's algorithms.

The discussion revolves around the strengths and limitations of the system. One notable strength is the use of affordable and easily accessible components such as Arduino Uno and flex sensors, making the system cost-effective and feasible for widespread implementation. The system's ability to convert sign language gestures into text provides a valuable means of communication for individuals with speech impairments.

However, there are also some limitations to consider. The system's accuracy may vary depending on factors such as lighting conditions and the complexity of hand gestures. Certain intricate or rapid gestures may pose challenges for accurate recognition. Further improvements in the system's algorithms and training data could help address these limitations.

Overall, the sign-to-text system using Arduino Uno and flex sensors demonstrates promising results in converting sign language gestures into text. The system offers a potential solution to bridge the communication gap between individuals with speech impairments and the general population. Future research and development can focus on refining the system's accuracy, expanding its gesture recognition capabilities, and exploring additional features to enhance its usability and effectiveness in real-world scenarios.

V. FUTURE SCOPE

1. Application Development: The proposed smart glove has the potential for further development and expansion into various applications beyond sign language to speech conversion.

Consider exploring its use in gaming, where hand gestures can be translated into in-game actions, providing a more immersive and interactive gaming experience. Additionally, the glove can be integrated into robotics, enabling precise control of robotic movements through hand gestures. Furthermore, in the medical field, the glove can be utilized for rehabilitation purposes, assisting patients in regaining motor control and dexterity. Exploring these diverse applications will enhance the versatility and utility of smart gloves.

2. Integration of Text-to-Speech: Enhance the functionality of the system by incorporating a speaker that reads out the converted text. By integrating a text-to-speech module, the smart glove can provide real-time audio feedback, enabling seamless communication between the user and others. This feature will greatly benefit individuals with hearing impairments, as well as facilitate communication in noisy environments or situations where reading the text on the display may not be practical. By adding a speaker to the system, it will become more inclusive and user-friendly, ensuring effective communication for a wider range of users.

By pursuing these future scope areas, the smart glove system can be expanded to cater to diverse applications, improving accessibility, and fostering better communication experiences for individuals with speech and hearing impairments.

VI. ACKNOWLEDGMENT

We would like to express our sincere gratitude to our professors, Prof. Swati Joshi, and Prof. Sachin Sawant, for their valuable contributions, guidance, and insightful suggestions throughout the development of our project. Their expertise and dedication have been instrumental in shaping our ideas and improving the project at various stages.

Conclusion

The proposed smart glove for converting sign language to readable text represents a significant advancement in assistive technology for individuals with communication disabilities. By incorporating flex sensors, an Arduino Uno microcontroller, and a novel state estimation method, the glove demonstrates precise and accurate gesture recognition capabilities. Through feasibility testing, specifically focusing on converting Indian Sign Language to voice output, the prototype has shown promising results, affirming its potential as a practical solution for individuals with speech impairments. Moreover, the versatility of the glove extends its applications beyond communication assistance, making it a valuable tool in gaming, robotics, and the medical field. Addressing the challenges faced by individuals with communication disabilities is crucial for promoting inclusivity and empowering their participation in various aspects of life. The smart glove offers a means to bridge the communication gap, enabling individuals to express themselves, interact with others, and access education and employment opportunities more effectively. Furthermore, this research contributes to the broader field of assistive technology by showcasing the integration of state-of-the-art technologies and highlighting the potential for future developments and enhancements. By incorporating user feedback and refining the system based on specific needs, the smart glove can further evolve to meet the requirements of individuals with communication disabilities. Moving forward, continued research, collaboration, and investment in assistive technologies are necessary to address the diverse needs of individuals with communication disabilities. By fostering an inclusive society and leveraging technological advancements, we can create a more accessible environment that empowers individuals and ensures their active participation in all aspects of life. In conclusion, the proposed smart glove represents a significant step forward in enhancing verbal communication for individuals with speech impairments. With its advanced features, feasibility testing, and versatility, this technology holds tremendous potential to positively impact the lives of individuals with communication disabilities, promoting inclusivity and improving their overall quality of life.

Copyright

Copyright © 2023 Vaibhav More, Swapnil More, Mrunmai Girame, Omkar More, Rohan More, Muhammad Parkar, Swati Joshi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.