A Machine Learning Approach for Air Writing Recognition

Abstract

Air Writing Recognition is a step towards the safety of people. Aim of this project is to make a step towards smart technology and provide an alternative when phones are unable to use. It helps the women, children and citizens in emergency situation. By writing a character (h, p) in air infront of CCTV, it recognizes that someone needs help. This project also send a voice message, text message to get emergency help and support. We hope that this project makes more safety to women, children and citizens and makes the crime rate less. The Air Writing Recognition project is a combination of computer vision object tracking and handwriting recognition. The Air Writing Recognition system uses the webcam of a computer to track character digits written in the air by the user, then uses a optical character recognition(OCR) algorithm to classify the character digits. And further it uses a Twilio account to make calls and sends message depends upon the character that the system recognized.

Introduction

I. INTRODUCTION

Handwriting analysis could be a classic, wellexplored downside within the realm of introductory Machine Learning that encompasses several necessary topics of neural networks. Air Writing is particularly helpful for user inter phases that don’t permit the user to kind on a keyboard or pen a trackpad/ touch screen, or for test input for sensible system management, among several applications. Air Writing differs from typical handwriting; the one that performs Air Writing will solely use associate degree unreal coo-ordinate to guide the writing motion. The varability of motion knowledge represents a letter is so significantly broader in Air Writing than in paper writing. Our project aims to use a mixture of computer vision and hand writing recognition to form a system that acts as a virtual white board. model users would be ready to write “air words” facing a web camera either real-time or in advance and have those gestures translated into letters or digits. The aim of the project is to further explore the task of classifying hand written text and to convert hand written texts into a digital format and achieving a virtual white board system at a cost that is accessible to the average user. We want to introduce alternative interfaces for communication that have high affordability, usability, and accessibility.

A. Motivation

We settled on computer vision and handwriting analysis because we believe air writing recognition is a worthwhile topic to pursue. It allows for an entirely new text input interface that does not require much more than the computer itself. We were not the first to pioneer the idea. Other systems such as the HTC Vive [15] virtual reality (VR) system have products that follow a similar idea. The Vive has a virtual whiteboard experience that allows users to write in the air while immersed in VR, but the system comes at a high cost that also requires an addition extensive (expensive) tracking system.

The motivation of our project is to achieve a virtual whiteboard system at a cost that is accessible to the average user. We want to introduce alternative interfaces for communication that have high affordability, usability, and accessibility, and further to get an help whenever the people are in emergency situations, when the phones are unavailable and unable to use.

B. Objective

The objective of our project is to create a system that needs only a computer and a built-in webcam to recognize different letters and digits written in the air, and further make a call whenever the people need help.

C. Existing System

Text entry takes an important role of effectively delivering the intention of users to computers. In the recent trends of developing technologies like increasing contactless services due to covid-19,a more advanced type of text entry is required.

In this it recognizes the characters when represented in the air. It will exactly predicts what the character is, and also it will classify the characters into one of 62 classes.

D. Proposed System

The proposed system uses the camera for Air Writing Recognition. In our prototype, two basic techniques are used that includes processing of a video input frame and then pattern recognition of processed frame using Pytesseract. The Air Writing Recognition system uses the webcam of a computer to track characters written in the air by the user. Our proposed sysytem uses a Pytesseract to classify the character digits into one of 62 classes(0-9 digits, 26 uppercase letters, 26 lowercase letters ). By writing a character (h, p) infront of CCTV, it recognizes the characters and it will think that someone needs help. Our proposed system uses Twilio Account for making calls based on the characters represented by the people infront of the CCTV.

II. LITERATURE SURVEY

As we have studied a lot of base papers that include the detail information about Air Writing identification and recognition of the characters and their gestures where they used different algorithms to identify the characters accurately.

In the proposed system of our project we identify and recognize the character, depending on that we help the people when they are in an emergency situation.

1. A Wearable Realtime character Recognition system based on Edge Computing Enabled Deep Learning for Air-Writing

Now-a-days wearable sensors play a vital role in the detection of human emotions, innovating an alternate and intuitive form in human computer interaction. This system enables users to have the freedom and flexibility to write characters in free space by moving fingers and uses a deep learning algorithm to recognise 36 characters from the motion data captured by Inertial Measurement Units and processed by microcontroller. In this study, we represented a Wearable Air Recognition, which can collect finger stroke data from a built in IMU sensor to realise the recognition of 10 digits(0-9) and 26 lower-case English letters here the tensor flow and lite based CNN algorithm with combination of acceleration and angular velocity data is used to recognise finger strokes in air.

2. Air-Writing Alphanumeric  Character Recognition

Our project develops a system that can recognize the air-written words in 3D space. Air-Writing is the new way of writing linguistic characters or words in free area using hand or finger movements. This uses the digital camera of a PC to track character digits written within the air by user. Several current systems use advanced and high-priced chase setups to realize gesture recognition.

3. Air-Writing Recognition Based on Deep Convolutional Neural Networks

To date, some of the fundamental problems in isolated writing have not been addressed effectively. This paper presents a simple effective air- writing recognition approach based on Convolutional Neural Networks. In this project it performs the hand tracking only, avoiding the use of complicated procedures for finger tracking. And also segmentation of writing acts(push-towrite) along with this restrictions on the users writing due to the limitation of an imaginary box is implemented.

4. Computer vision Based Air Writing Recognition System

This model recognizes gestures written in the air and converts it into text, users would be able to write "air words" facing a web camera either real time and have those gestures translated into letters or digits. This model recognizes gestures written in the air and converts it into text, users would be able to write "air words" facing a web camera either real time and have those gestures translated into letters or digits. It is a constant video based venture which permits the drawing and composing of English letters in order or number (0-9) through the air before web camera or versatile camera.

5. Air-Text: Air-Writing and Recognition System      

Text entry takes an important role of effectively delivering the intention of users to computers. In the recent trends of developing technologies like increasing contactless services due to covid-19, a more advanced type of text entry is required. To tackle this issue, we propose Air-Text to write in the air using fingertips as a pen. Air-Text provides various functionalities of Air-Writing and TextRecognition  modules. Specifically, the AirWriting module takes a sequence of RGB images as input.

In this paper, examples of single digit recognition with MNIST classifier (96% accuracy) and word-level recognition with text recognition model(79.36% accuracy) are provided.

6. Air Writing Recognition And Speech Synthesis

Gesture recognition is what we see here by using mathematical algorithms. The convolutional character inputting methods are using keyboard, digital pen and touch screen. Here we use a vision based character inputting system. It requires small hardware Device that is camera. So it can be used for embedded applications and smart mobile phones. It acts as reading and hearing aid for the disabled. Now the interaction with machines is more flexible the user does not have to wear special sensors or touch devices.

7. Trajectory-Based Air-Writing Recognition Using Deep Neural Network and Depth Sensor

Trajectory-based writing system refers to writing a linguistic character or word in free space by moving a finger or handheld device. It is widely applicable when the writing         systems are troublesome. A depth camera or sensor as an Intel RealSense SR300 camera is used. As it tracks the fingertip and recognizes the character. For better feature selection it uses the nearest neighbour and root point translation to normalize the trajectory.

8. Air Writing Recognition Framework using Image Representation of IMU signals [IMAIR]

They proposed an Air Writing recognition frame work by first encoding the time-series data obtained from a wearable Inertial Measurement Unit (IMU) on the wrist as images and then utilizing deep learning based models   for identifying for written alphabets. It can be utilized to provide a user with a rapid and touch-free input alternative that can be used in Human Computer Interaction applications. In this work , an Air Writing recognition framework using image representation of time-series data is collected through 3-axis accelerometer and gyroscope sensors from an Inertial Measurement Unit (IMU).

9. Air-Writing Via Receiver Array Based Ultrasonic Source Localization

Air writing systems have recently been proposed as tools for human machine interaction where the instructions are represented using letters or digits written in the air. Different technologies are using the air writing systems, the proposed system consists of two components  like motion tracking component, text recognition component.

They proposed a novel 2-D DOA (Direction of arrival) estimation algorithm that can track the change in the direction of the transmitter using measured phase difference between receiver and array elements. The proposed phase difference projection (PDP) algorithm can provide accurate tracking with 3-sensor receiver array.

10. Fingertip Detection And Tracking For Recognition For Air -Writing In Videos

Air-writing is the process of writing character or words in free space using finger or hand movements without the aid of any hand-held device. In spite of recent advances in object detection and tracking of the fingertip remains a challenging task. A character        recognition experiments give a accuracy of 96.11% using the proposed Air Writing system.

III. SYSTEM MODEL

A. Data Explanation

Our handwriting classification network is trained on the EMNIST(extended MNIST) Dataset of handwritten number digits [14]. This dataset is a popular choice for handwriting classification networks, and we chose to use the "by class" format– which separates between digits, uppercase, and lowercase letters– to train on. The EMNIST dataset of over 814,225 samples is already split  90% training and 10% testing, but we decided to take the training set of data to be both our training and our evaluation data. We split this dataset 8:2, training to evaluation, using the Scikit-Learn train_ test_ split method [15]. This second separation allowed us to save the original 10% test data for unseen testing.

B. Modules

1. Video Capturing and Recognition

With the help of open cv library we are able to capture webcam by using Video Capture method. By using Hand Tracking Module in CVzone library we can detect the hand by using Hand Detector method. writing the characters with index finger in front of webcam it will display in another prompt. After completion of writing character in front of web cam it save like a picture.

2. Hand Character Recognition

In this project we are  using pytesseract predefined library to classify the character digits into one of 62 classes: 10 digits, 26 uppercase letters, 26 lowercase digits. It will recognize and read that text embedded in images.

3. Calls to Exigency System

In the calling system first we have to sign up in the twilio account by using twilio.com. The signup process includes verifying your personal phone number. This is a security measure that is mandatory before you can try Twilio. Once you finish signup, you should see your Console Dashboard.

By using this phone number, we make a call to the police station or hospital. If the character is h,it makes a call to the nearest police station. If the character is p, it makes a call to the nearest hospital.

C. Algorithms and Techniques OCR Algorithm

It is hard for computers to recognize characters because of the different fonts and variations on how one letter can be written. Handwritten letters complicate matters even further. Nevertheless, optical character recognition algorithms take on this challenge. Every OCR solution operates in four main steps.

1. Image Acquisition

The process involves using an optical scanner to capture a digital copy of the paper document. The document has to be properly aligned and sized.

Pre-Processing

The goal of this phase is to make the input file usable by the OCR algorithm. The noise and background are eliminated. Pre-processing includes the following steps:

Layout analysis: identifying captions, columns, and graphs as blocks.

De-skew: titling the digital document to make lines horizontal in case if it wasn’t properly aligned during scanning.

Image refinement: smoothing the edges, removing dust particles, increasing contrast between text and background.

Text detection: some algorithms detect separate words and divide them into letters while others work with text directly without splitting it into characters.

Binarization: converting the scanned document into black and white format, where dark areas represent characters(alphanumeric or numeric) and white areas are identified as background. This step helps to recognize different fonts.

2. Character Detection

During this phase, optical character recognition algorithms perform different manipulations to recognize letters and numbers. There are two main approaches:

Pattern recognition: OCR algorithms are trained on a wide variety of fonts, text formats, and handwriting styles to compare distinct characters from the input file to what they have learned.

Features recognition: some algorithms benefit from known character properties, such as crossed and curved lines, to identify characters in input files. For example, a letter “H” is identified as two vertical lines and one crossing horizontal line. OCR algorithms powered by neural networks (NN) use a different logic where the first NN layers aggregate pixels from the input file to create a low-level feature map of the image. After detecting characters, the program converts them to American Standard Code for Information Exchange (ASCII) to facilitate further manipulations.

Post-processing

The output can be basic like a character string or a file. is a computer vision package that makes us easy to run like hand tracking, pose estimation and also image processing.

IV. RESULTS AND ANALYSIS

A. Hand Gesture Recognition

Hand gesture recognition is a process of understanding and classifying meaningful movements by the human hands. Hand gestures are the natural way of interactions when one person is communicating with one another and therefore hand movements can be treated as a non verbal form of communication.

V. FUTURE WORK

In this project we are using h and p characters for implementing help to the people, whenever they are in emergency situation. As mentioned in the results section above, we ended up this project with the characters h and p.

In the future the proposed approach will be extended furtherly, to get more accuracy in the character recognition by using different algorithms. As well as proposed approach can be extended by implementing to the other characters like f, s, t, g, m and so on. ‘f’ implies for food, ‘s’ related to shopping, ‘t’ comes under tickets booking, ‘g’ comes under groceries and m implies to medicines. And also our proposed system can be extended furtherly by adding GPS tracker also.

Conclusion

Text entry takes an important role of effectively delivering the intention of users to computers. In the recent trends of developing technologies like increasing contactless services due to covid-19, a more advanced type of text entry is required. To tackle this issue, all are using the Air Writing Recognition System. By using Pytesseract, OCR and Twilio account we designed a model which helps the people when they are in an emergency situation. Our proposed system is highly affordable, usable and accessable to all. The main goal of this project is to recognize the characters and classify that characters and furtherly makes a call and sends message to help centers via Twilio account. The different processes like Preprocessing, Segmentation, Feature Extraction, Training a Neural Network, Post-Processing and Image Acquisition are performed for the Air Writing Recognition in Machine Learning.

References

[1] Chaur- Heh Hsieh, You- Shen Lo, Jen-Yang Chen, Shen- Kai Tang, “Air-Writing Recognition Based on Deep Convolutional Neural Networks”, January 2021, IEEE, Also available at researchgate.net/pub. [2] Md. Shahinur Alam, Ki- Chul Kwon, Md. Ashraful Alam, Mohammad Y. Abbass, Shariar Md Imtiaz, Nam Kim, “Trajectory-Based AirWriting Recognition Using Deep Neural Network and Depth Sensor”, January 2020,Also available at researchgate.net/pub. [3] Sun- Kyung Lee, Jong-Hwan Kim, “Air-Text: Air-Writing and Recognition System”, October 2021, Also available at dl.acm.org/doi/10.11. [4] Ms. M. Saranya, Kariketi Tharun Reddy, Madhumitha Raju, Manoj Kutala, “Air-Writing Alphanumeric Character Recognition”, 7 July 2022, Also available at www.ijcrt.org. [5] Hongyu Zhang, Lichang Chen, Yunhao Zhang, Renjie Hu, Chunjuan He, Yaqing Tan, Jiajin Zhang, “A Wearable Real time character. Recognition system based on Edge ComputingEnabled Deep Learning for Air-Writing”, 4 May 2022, Also available at hindawi.com/journals. [6] Catharine Elizabeth Kurian, Chelsa Martin, Deepthi P J, Eldhose K A, “Air writing recognition and speech synthesis”, vol. 5, p. 2395-0072, May 2018. Available: https://www.irjet.net. [7] Akash kumar M B, Ashwini R, Dhanaraj S, Shwetha B K, Dr. V Keerthika, “Computer vision Based Air Writing Recognition”, vol. 7, 2021. Available: https://www.ijariie.com. [8] Ayush Tripathi, Arnab Kumar Mondal, Lalan Kumar, Prathosh A. P. “Air writing recognition framework using Image Representation of IMU signals [IMAIR]”,2022, IEEE. Available: https://www.ieee.org/publications/index. [9] Hui Chen, Tarig Ballal, Ali H. Muqaibel, Xiangliang Zhang, Tareq Y. Al-Naffouri, “Airwriting via receiver array based ultrasonic source localization”, IEEE Available: http://hdl.handle.net/10754/662701. [10] Mukherjee, Sohom; Ahmed, Arif; Prosad Dogra, Debi; Kar, Samarjit; Pratim Roy, Partha, “Fingertip detection and tracking for recognition for Air -writing in videos”, September 2018. Available: https://ui.adsabs.harvard.edu. [11] G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000). [12] François Chollet et al. 2015. Keras. https://keras.io. [13] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv:1702.05373 (2017) [14] HTC and Valve Corporation. 2016. HTC Vive. https://www.vive.com/eu/. [15] F. HTC and Valve Corporation. 2016. HTC Vive. https://www.vive.com/eu/. [6] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J.

Copyright

Copyright © 2023 Koye Navya , Mallela Sowmyah, Shaik Ayesha Amreen, Vemireddy Sravani, Yarrakula Gayathri Devi, Ms. Perli Nava Bhanu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.