Hand Gesture Controlling System

Abstract

As a result of the industrial 4.0 revolution, hand gestures are becoming more and more significant in the disciplines of robotics and IoT. Hand gestures are often used in the IoT field in applications for smart homes, wearable technology, vehicles, virtual reality, and other things. Let\'s try to infuse this work with some of our own originality. Combining Python and Arduino for Laptop/Computer Gesture Control. We\'ll utilise two Ultrasonic sensors to determine where our hand is with relation to a media player (VLC) and control it. We\'ll mount two ultrasonic sensors on top of the monitor and use an Arduino to gauge how far away it is from our hand. We will take particular activities in response to this measurement. We can do computer operations using the Python PyautoGUI module. The computer gets commands from the Arduino via the serial port. Currently, scientists are working to develop a hand gesture computer that runs entirely on hand gestures and sensors instead of any hardware. Few researchers have actually shown that we can control the video player, web browser, and text document with hand movements using Arduino and ultrasonic sensors.

Introduction

I. INTRODUCTION

Leap Motion is a technique that enables us to operate some elements on our laptop or computer by simply waving our hand in front of it. Ultrasonic sensors are used to control the computer. The cost of these laptops is astronomically high, despite the fact that it is tremendously cool and pleasurable to do. So, let’s try building our own Gesture Control Laptop/Computer in this project by combining Python and Arduino’s capabilities. We’ll utilise two Ultrasonic sensors to determine where our hand is with relation to a media player (VLC) and control it.

We can perform computer tasks using the Python PyautoGUI module. The computer gets the orders from Arduino via the serial interface. This data will then be read by Python, which is now running on the computer, and an action will be made in response to the read data.

The Fourier transform is then applied to the incoming time-domain signals following buffering. The Arduino can be connected to a computer or laptop for serial communication as well as powering the module. With this method, magnitude vectors are spread uniformly across the spectral width. After each FFT vector has been created, the bandwidth of the signals, gesture recognition speed, and motion detection are all determined.

Next, commands for the computer are generated from the detected motions. For many years, the main input devices for computers have been keyboards and mouse. Hand or body movements, however, are becoming more and more significant as ubiquitous and ambient technology (like PlayStations) and tools that enable users handle virtual goods become more and more widespread. Today, gesture controllers are a crucial component of any system for human-computer interaction.

First, we mount two ultrasonic sensors on top of the laptop, spaced apart from one another. We fasten the Arduino uno board to the laptop’s top panel before connecting its circuitry to the ultrasonic sensors. The distance between the ultrasonic sensor and our hand in front of it is now measured.

The operation that is performed depends on the distance. The main objective of our research is to use hand gestures to operate miniature games without using a keyboard, mouse, or any other kind of hardware controller—just an Arduino board and some ultrasonic sensors.

II. HARDWARE COMPONENTS SETUP

A. Basic Hardware Setup

To implement the proposed method, we require an Arduino uno board, a pair of ultrasonic sensors, a USB cable and finally jumper wires. Initially, we use double-sided tape to secure the Arduino Uno board to a laptop's front face or panel. The power supply from the PC to the board is then tested using a USB cable. The sensors are mounted on either side of the top of the screen after the power supply has been tested.

B. How Ultrasonic Sensors Work

The radar system and the ultrasonic sensor both function according to the same principles. Here, waves are created using electrical energy, and a transmitter module transmits these waves into the air to find something. The receiver module receives the echo waves that are reflected when they strike an object, and the distance is calculated using the formula below.

Distance = (Speed of the waves × Time taken to travel)/2  

Because the waves travel both to and from this location, we divide the total distance in half.

C. The Arduino Uno Board Overview

An open source, programmable microcontroller board called the Arduino-Uno was created as a more straightforward alternative to the Arduino Mega 328. Using the Arduino IDE, the board's 14 digital input/output pins and eight analogue pins can be programmed (Integrated Development Environment).

The official Arduino website is where you may find the clear details of setups and technical specs, as indicated below.

By connecting with a type B USB cable, it may be programmed using straightforward C/C++ programming. It can accept voltage from a power source that ranges from 7 to 20 volts. The board's clock frequency is 16 MHz, and the SRAM has a 2 KB capacity. Depending on the distance, a specific operation is carried out. Our paper's main goal is to use hand gestures to control mini games without the use of a keyboard, mouse, or any other physical device, merely an Arduino board and ultrasonic sensors.

Our paper's main goal is to use hand gestures to control mini games without the use of a keyboard, mouse, or any other physical device, merely an Arduino board and ultrasonic sensors.

III. LITERATURE REVIEW

[1]. S. Lian, W. Hu and K. Wang, "Automatic user state recognition for hand gesture based low-cost television control system. This paper proposes an automatic user state recognition scheme to recognize the TV user's state and activate the camera-based gesture recognition module only when the user is trying to control the TV. Specifically, the user's behaviour active or not is detected by low-cost sensors, the user's gaze watching TV or not is tracked by the face-based view detection, and the user's state is then decided according to a finite-state machine composed of four states: Absent, Other Action, Controlling, and Watching. The prototypes based on an ultrasonic distance sensor array, a red-green-blue (RGB) camera, and a depth camera are implemented and tested. The results show that the proposed scheme can effectively reduce the power consumption or computational cost of the original hand gesture-based control schemes.

[2]. X. Gao, L. Shi and Q. Wang, "The design of robotic wheelchair control system based on hand gesture control for the disabled. This paper designed a control system for robotic wheelchair for the aged and the disabled, which consisted of two main parts: gesture interaction and intelligent wheelchair. This design used the Kinect camera to develop the problem of hand gesture segmentation and gesture tracking.

[3]. Oudah, M.; Al-Naji, A.; Chahl, J. Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. Hand gestures are a type of nonverbal communication that can be employed in a variety of contexts, including medical applications, human-computer interface (HCI), robot control, and communication between deaf-mute individuals.

Many diverse methodologies have been used in research publications based on hand gestures, including computer vision and instrumented sensor technology.

[4]. Z. Meng, J. -S. Pan, K. -K. Tseng and W. Zheng, "Dominant Points Based Hand Finger Counting for Recognition under Skin Color Extraction in Hand Gesture Control System. In this study, a novel method for recognising hand gestures is proposed. This method counts the fingers on the hand using dominating spots under the extraction of skin tone. In order to obtain the hand gesture contour, skin colour detection is employed as a preprocessing segmentation. In this hand gesture control system, counting hand fingers is done using a dominating points-based technique following hand segmentation.

IV. PROBLEM IDENTIFICATION

The current one, which solely controls the VLC MEDIA PLAYER, will be helpful when using VLC because it is not supported by other apps.

 The customer wishes to use gestures on the laptop to control other programmes like Chrome, PowerPoint, etc.

1. Diverse Backgrounds: No of the environment, gesture recognition should produce excellent results, whether you're at home, at business, or simply out and about.
2. Movement: It stands to reason that a gesture is more like a movement than a still image. Thus, pattern recognition should be possible for gesture recognition.
3. Combination of Movements: In addition, gestures may include multiple motions, therefore we must provide context and identify common gestures, such as moving fingers in a clockwise direction while extending the thumb to indicate a small area or a finite number of files.
4. Expensive: Laptops with gesture controls have a very high price tag. Therefore, we are creating our own "Gesture control Laptop" for this project.
5. Battery Draining: In the current setup, the camera must operate continually in order to run this programme, which increases battery draining.

V. PROBLEM METHODOLOGY

The In this paper, a method based on sensor-based distance measurement and subsequent performance of a specific function is presented. According to client input, the solution is to use hand gestures to manage laptop apps. Actions are first recognised using a sensor, after which various gesture recognition techniques are suggested. We developed a few widely used techniques based on the sensors' ability to recognize actions. For speedy operation, a sensor device is mounted on a computer at the top of the screen. There has been a lot of study in this area, however it has mostly focused on hand identification, real-time finger recognition, and alphabet character recognition.

 A lot of functionality, including video control, music player, gaming, manipulating the features of a PDF reader, etc. uses real-time human computer interactions. Real-time gesture recognition algorithms are used in all of these interactions. A physical device that tracks and recognises the body language or movements is always necessary for a gesture controller resolution so that the computer can interpret them. The distance of the hand can be determined using an ultrasonic sensor, which serves as an input. Depending on the hand's distance, a specific function is carried out. The solution is completed using Arduino and Python libraries.

VI. ARCHITECTURE DESIGN

The ground pin is connected to the first GND pin, while VCC is connected to the Arduino-Uno board's 5V power pin for the left sensor. The trigger pin is then connected to pin 2 on the Arduino board, and the echo pin is lastly connected to pin 3. The VCC pin is connected to the 3.3V power pin for the right sensor, while the ground pin is connected to the second GND pin. Last but not least, we wire up the echo pin to pin 5 and the trigger pin to pin 4 on the Arduino board.

B. Arduino Programming To Detect Gestures

The 5 distinct hand motions or acts that we have designed for are as follows:

1. Gesture 1: Slowly move the hand away from the right ultrasonic sensor after briefly placing it in front of the sensor at distance (between 15CM and 35CM). This action will scroll the web page down.
2. Gesture 2: For a brief period of time, place the hand in front of the right ultrasonic sensor at a distance (between 15CM and 35CM), then move the hand in the direction of the sensor. This action will scroll the web page upward.
3. Gesture 3: Swipe your hand in front of the right ultrasonic sensor in the third gesture. The Next Tab will be reached by this action.
4. Gesture 4: Move your hand in front of the left ultrasonic sensor in gesture number four. This action will play/pause the video or switch to the previous tab.
5. Gesture 5: Swipe your hand across both sensors in gesture number five (Left Sensor first). The Tasks will alternate after this action. The following Arduino programme has been built based on the aforementioned motion.

Conclusion

The This article offers one method, among many others, for using hand gestures to control a computer. It is one of the simplest forms of human-computer interaction. It is a model with a low cost that only uses an Arduino UNO and an ultrasonic sensor. For the purpose of achieving diverse processing and controlling methods for developing new gesture control solutions, the Python IDE enables a seamless interaction with Arduino UNO. Additional opportunities for gesture recognition exist in medical settings where a nurse or doctor may not be able to touch a display or trackpad but still has to operate a system for reasons of health and safety. In other situations, the medical practitioner might not be close to the display but still needs to control the information being displayed on it. The gadget can be used more safely and quickly by employing appropriate gestures, like hand swipes or using a finger as a virtual mouse. With the aid of the Arduino microcontroller ATMEGA32, an effective ultrasonic-based hand gesture monitoring system is created in the current study. The hand gestures are accurately recognised by ultrasonic sensors, and the corresponding hotkeys are then engaged to execute computer applications like scrolling, adjusting the volume, changing windows, stopping, and switching between tabs in a browser. It is demonstrated that no additional hardware is needed to identify hand movements and that hand gestures may be recognised at various ranges using only low-cost ultrasonic sensors. By using the example of a little call race, we have demonstrated how we can use an Arduino and ultrasonic sensors to play more compact games with hand gestures. Here, sensors are used to gauge the separation between our hand and the sensors. With the aid of the Arduino IDE, we can programme the Arduino board to carry out a number of tasks based on that distance. Hand gestures can be utilised to control the keyboard using the Python library pyautogui. In the future, we plan to use high-performance and high-quality sensors to play big games using the same strategy.

References

[1] Cohen, Charles J., Glenn Beach, and Gene Foulk. \"A basic hand gesture control system for PC applications.\" Proceedings 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery. IEEE, 2001. [2] Khan, Rafiqul Zaman, and Noor Adnan Ibraheem. \"Hand gesture recognition: a literature review.\" International journal of artificial Intelligence & Applications 3.4 (2012): 161. [3] Song, Shining, Dongsong Yan, and Yongjun Xie. \"Design of control system based on hand gesture recognition.\" 2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC). IEEE, 2018. [4] Premaratne, Prashan, and Q. J. I. C. V. Nguyen. \"Consumer electronics control system based on hand gesture moment invariants.\" IET Computer vision 1.1 (2007): 35-41. [5] Lian, Shiguo, Wei Hu, and Kai Wang. \"Automatic user state recognition for hand gesture based low-cost television control system.\" IEEE Transactions on Consumer Electronics 60.1 (2014): 107-115. [6] Sziládi, Gergely, Tibor Ujbányi, and József Katona. \"Cost-effective hand gesture computer control interface.\" 2016 7th IEEE International Conference on Cognitive Infocommunications (CogInfoCom). IEEE, 2016. [7] Oudah, Munir, Ali Al-Naji, and Javaan Chahl. \"Hand gesture recognition based on computer vision: a review of techniques.\" journal of Imaging 6.8 (2020): 73.Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specification, IEEE Std. 802.11, 1997 [8] Pavlovic, Vladimir I., Rajeev Sharma, and Thomas S. Huang. \"Visual interpretation of hand gestures for human-computer interaction: A review.\" IEEE Transactions on pattern analysis and machine intelligence 19.7 (1997): 677-695.

Copyright

Copyright © 2023 V. Krishna Vijaya, Puvvala Harsha, Sricharan Murukutla, Kurra Eswar, Nannapaneni Sravan Kuma. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.