Gesture Based Handling of Virtual Mouse and Keyboard

Abstract

Keyboard and Mouse handling is very essential for Human Computer Interaction. So, we promotes an approach for the Human Computer Interaction (HCI) where cursor movement can be controlled using a real-time camera, it is an alternative to the current methods including manual input of buttons or changing the positions of a physical computer mouse. Instead, it uses cameras and computer vision technology to control various mouse events and can perform all the tasks that a physical computer mouse can do. The Virtual Mouse program will constantly acquire real-time images where the images will undergo a series of filtration and conversion. When done, the program uses image processing to get the co-ordinates of the targeted points from the converted frame. After that, it will proceed to compare the existing co-ordinate within the frames with a list of predefined co-ordinates, where different combinations consist of different mouse functions. If the current co-ordinates combination found a match, the program will execute the mouse and keyboard function, which will be translated into an actual mouse and keyboard function to the users’ machine.

Introduction

I. INTRODUCTION

The computer screen captures the video of the person sitting at the computer and a small green screen is created in the middle of the screen. In that green box, the objects shown will be processed by the code and matched with it if it which means the computer has identified the object and then by moving the object the mouse cursor can be moved.

This will not only help in the security of the compute but also help in generating a virtual computational experience. Here in the place of different objects, using hand gestures one gesture will be moving the cursor, the different gesture will be used for right click and different for left click, similarly with a simple gesture can do the keyboard functions virtually that may have been done on some keyboard as a physical aspect.

II. LITERATURE SURVEY

Adjania et al. 2010[1] Create a virtual keyboard using shadow focus. The system detects keyboard, hand shadow, finger cues using color segmentation and Sobel processing. Ambient lighting should be from the system. The system can analyze 3 frames per second Hernanto and Supriana 2011 [2] developed a method for using a virtual keyboard using a webcam. This method uses two functions () for fingerprint recognition and location measurement.

The system uses skin and two different webcams to detect location. The average time per character for this virtual keyboard is 2.92 ms and the actual average for the system is 88. 61% Cecotti 2016[3] developed a system for people with disabilities called Multimodal Glance Control Virtual Keyboard. The virtual keyboard has 8 key commands to select a menu to enter 30 different characters and a clear button to recover from errors. They measured the performance of the system using command and application-level speed and message forwarding.   

Saraswati, Sigit and Harsono 2016[4] introduced a system called Eye Gaze System for Work Virtual Keyboard for the disabled. First, it detects the user's face and acquires the eye gaze, which is used as a marker in the next step of. Use the HaarCascade method to extract facial features and use the Integral Projection method to obtain the position of eye movement. According to his experiments, the comparison of normal typing time for two languages ??using their is 1:13. Mj et al. 2018 [5] proposed an improved keyboard system based on fingerprint recognition. The system was developed using the OpenCV library and Python. Palm detects enhanced keyboard typing. Virtual keyboard tracks finger movements

III. PROPOSED SYSTEM

We describe the proposed architecture for Gesture Based Handling Of Virtual Mouse & Keyboard :

As shown in Fig 1 our System will take the images as input. Pre-processing on the input will be done by various techniques. Generally, removal of noise, resizing of image, converting the image into grey scale is done in pre-processing. After this, image segmentation will be done with the help of cv2. Features will be extracted from the image such as edge detection, etc. On completion of the above process we will be using Linear Regression for the detection of the gesture. After detection the gesture, required task will be carried out by mouse/keyboard. As shown in Fig.1 our System will take the images as input. Pre-processing on the input will be done by various techniques. Generally, removal of noise, resizing of image, converting the image into grey scale is done in pre-processing. After this, image segmentation will be done with the help of cv2. Features will be extracted from the image such as edge detection, etc. On completion of the above process we will be using Linear Regression for the detection of the gesture. After detection the gesture, required task will be carried out by mouse/keyboard.

IV. IMPLEMENTATION

User has to create an account before using the product. While creating a account user has to provide some basic credentials like his email, mobile no., etc. If the user is already registered to the system he can just use log in button and log in to the system using his user name and password. Once the user gets logged in he has option of whether he wants to use virtual mouse or the virtual keyboard. While using any either mouse or keyboard, the camera is required to capture the gestures. The gestures are captured using the camera and are processed using background separation and convolutional neural network algorithms with the help of mediapipe. Mediapipe is a popular python package developed by Google to identify various motions and gestures. Basically, mediapipe is used in the project to identify palm points. 21 palm points are detected and accordingly the gesture is detected based on previously trained data. Once the gesture is correctly gets detected the corresponding event gets triggered. The tip of forefinger is used to move the cursor freely. Specific threshold is specified for the distance between the forefinger and thee middle finger. The moment at which the distance between these two fingers gets below threshold a click event is triggered.

V. SCOPE OF FUTURE USE

Future work of this project includes making the Fingertip detector module invariant to illumination changes and 3D pose estimation of panel which can be used for Argumentation reality for 3D objects. In the future, we will take advantage of the other graphic features in the human-computer interface to detect touch events on the projected screen. In the future, we plan to add more features such as enlarging and shrinking window, closing window, etc. by using the palm and multiple fingers.

Conclusion

VI. CONCLUSION We successfully proposed the project of Operating Virtual Mouse and Keyboard using hand Gestures which detects various gestures of our hand and then operates accordingly. With the help of this we can move one step ahead towards the Virtual Reality. We would try to build the most optimal system but there would be lot for future scope. The results we will get would be moderate and of good accuracy.

References

[1] Y. Adajania, J. Gosalia, A. Kanade, H. Mehta, and N. Shekokar, “Virtual keyboard using shadow analysis,” in 2010 3rd International Conference on Emerging Trends in Engineering and Technology, pp. 163–165, 2010. [2] S. Hernanto and I. Supriana, “Webcam virtual keyboard,” pp. 1–5, 07 2011. [3] H. Cecotti, “A multimodal gaze-controlled virtual keyboard,” IEEE Transactions on Human-Machine Systems, vol. 46, pp. 1–6, 05 2016 [4] V. I. Saraswati, R. Sigit, and T. Harsono, “Eye gaze system to operate virtual keyboard,” in 2016 International Electronics Symposium (IES), pp. 175–179, 2016. [5] J. Mj, M. Surya, A. M. Bt, and R. Poovaraghavan, “Finger recognition and gesture based augmented keyboard,” International Journal of Advance Research, Ideas and Innovations in Technology, vol. 4, pp. 662–665, 2018.

Copyright

Copyright © 2023 Vinod Khade, Ashutosh Shimpi, Pritesh Agrawal, Panthilraj Patil, Swati Firame. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.