Gesture Recognition based Virtual Mouse and Keyboard

Abstract

The field of computer vision has advanced significantly, enabling computers to identify their users using simple programs based on image processing. This technology has been widely used in various day-to-day applications, such as face recognition, color detection, and autonomous driving. This research project aims to use computer vision to develop an optical mouse and keyboard that can be operated through hand movements. The computer camera will capture images of different hand gestures made by the user, and the mouse pointer or cursor on the computer screen will move accordingly. Different hand gestures can be used to execute right and left-clicks. Similarly, the keyboard functions can be performed using different hand actions, such as using a finger to select an alphabet and a four-digit swipe left or right. The virtual mouse and keyboard can be used wirelessly or externally, and the only hardware required for the project is a webcam. The project is developed using Python on the Anaconda platform, offering a flexible and easy-to-use development environment. The proposed technology provides a cost-effective and user-friendly alternative to traditional input devices, eliminating the need for physical mouse and keyboard. This research project demonstrates the potential of computer vision in developing innovative human-computer interaction systems, which could have significant applications in various fields, such as healthcare, education, and entertainment.

Introduction

I. INTRODUCTION

Personal computer systems have become an essential part of our daily lives, serving different purposes in areas such as work, education, and entertainment. However, most computer applications still rely on traditional input methods, such as mouse and keyboard. While facial recognition technology has been widely used, there is a lack of efficient systems to assist individuals with disabilities in operating personal computers.

The current system that exists allows users to control the desktop without using their hands. The computer camera captures the user's video, and a small green box is generated in the middle of the screen. The displayed objects are matched with the code, and if it matches, a red border is generated, indicating that the computer has identified the object. Moving the object will move the mouse cursor accordingly. This system not only enhances the security of the computer but also creates a virtual computing experience. Using hand gestures, different actions  can be performed, such as moving the cursor, right-clicking, and left-clicking, eliminating the need for physical mouse and keyboard. The system can be operated through simple gestures that simulate the functions of a keyboard. If the gesture does not match, the green box will be displayed, but a red border will occur when the known gesture is observed.

The proposed technology offers a cost-effective and user-friendly alternative to traditional input devices, providing an innovative solution to assist individuals with disabilities in operating personal computers. The system utilizes computer vision and image processing techniques to create a virtual mouse and keyboard that can be operated through hand gestures. The system is expected to have significant applications in various fields, such as healthcare, education, and entertainment.

II. LITERATURE REVIEW

1. A research study on human-computer interaction proposes a virtual mouse control system using hand motion and finger detection based on live video. Two techniques are used to track fingers: hand gesture detection and coloured caps. The system involves three main stages: hand motion tracking, finger detection via colour recognition, and cursor implementation. A convex hull is constructed around the contour for hand gesture tracking, and the hand features are obtained using the area ratio of the generated hull and contour. The system is tested thoroughly to validate its practical application.

2. The focus of text-entry research has switched from physical to soft keyboards as a result of the rise in mobile computing. However, the lack of tactile feedback in soft keyboards results in higher typo rates and reduces usability by taking up screen space. A fully imagined keyboard (I-Keyboard) with a deep neural decoder (DND) is proposed to get over these restrictions. The invisibility of the I-Keyboard maximises the use of mobile devices and DND.

3. The development of HCI technology has led to advancements in the interaction between humans and computers. Although wireless or Bluetooth mouse technology is constantly evolving, it still requires additional gadgets like a connecting dongle and a battery. This limitation is overcome by a virtual mouse system based on HCI technology that uses hand motion and computer vision. The system tracks the movements of the hand with coloured caps on the fingertips and processes them to perform various computer cursor functions like scrolling and clicking. The suggested mouse solution eliminates the need for a device to use a mouse and thus simplifies the usage of the system.

4. Today, computer vision has advanced to the point where a machine can recognise its owner by running a straightforward picture processing programme. People use this vision in many parts of daily life at this stage of evolution, including face recognition, colour detection, automatic cars, etc. In this project, an optical mouse and keyboard are created utilising hand motions and computer vision. The computer's camera will scan the image of various hand gestures made by a user, and the computer's mouse or pointer will move in accordance with the movement of the movements. Users can even conduct right and left clicks using various gestures. Similar to this, several gestures can be used to access keyboard features, such as the one-finger.

5. A computer can now recognise its owner using a fundamental picture handling calculation thanks to advancements in computer vision. At this stage of development, people use this vision in many aspects of their daily lives, including automated driving, face recognition, and colour location. In this case, computer vision is used to create a virtual mouse and console that uses hand and face motions. The computer's camera will scan an individual's hand and eye motions to capture various gestures, and the mouse or pointer will move in reaction to the movements, including doing right and left clicks with different gestures. Various signals, such as a one-finger gesture, can also be used to control the console.

III. REQUIREMENTS AND SPECIFICATIONS

A. Functional Requirements

1. User should perform gesture of hand and eyeball properly.
2. User system should have specific resolution webcam.
3. After developing this application user should be able to access their system through hand and eyeball gesture

B. Software Requirements

Operating System - Windows 10

IDE – Spyder

Technology used – Python

C. Hardware Requirements

Processor- Intel Pentium 4

RAM- 8 GB

D. Algorithm Used

The primary aim of the proposed Hand Gestures-based Virtual Mouse and Keyboard system is to allow users to control mouse cursor and keyboard functions through eye movement and hand gestures, respectively, instead of using a physical mouse or keyboard. This system is developed on the machine learning domain, using open cv libraries with Haar cascade algorithm and MediaPipe framework.

The Haar-cascade algorithm is typically used for protest detection, especially in identifying faces, people on foot, items, and outward appearances in an image, and is primarily utilized for face detection. To train a cascade function, the Haar cascade uses a machine learning method that requires a large number of positive and negative images. In feature extraction, it employs training data to identify features, where the Haar cascade features are rectangular frames that traverse across pictures to match features.

MediaPipe framework is an open-source framework by Google used for applying with the machine learning pipeline. It is a cross-platform development tool, and it is constructed using time series data. This framework is commonly used with various audio and video applications. Moreover, it is used for constructing and analyzing the system using graphs and system development for application purposes.

OpenCV is an open-source library for computer vision that can be used on various platforms. Computer vision is a field of study that focuses on enabling computers to understand and interpret visual information from digital images or videos. OpenCV is designed to be a comprehensive solution for image processing and analysis, including tasks such as face detection, object detection, and image segmentation. It provides a wide range of tools and functions that make it easier to implement these tasks, even for those without a deep understanding of computer vision techniques.

Regarding the proposed system, access to it is restricted to registered users only. This means that users must first register for the system and create a unique username and password. They can then log in using the same credentials each time they want to use the system. This helps to ensure that only authorized users can access the system and perform the intended tasks, and provides a level of security and privacy for user data.

2. Keyboard

a. After successfully logging in, the user is presented with two options: Virtual Mouse and Keyboard.

b. To access the virtual keyboard, the user must select the Virtual Keyboard option, which then displays a keyboard on the desktop screen.

c. The virtual keyboard is pre-programmed to contain only the characters and letters that have been defined in the system's coding.

d. To type a letter or character, the user can simply point to it with their finger using the webcam of their laptop or computer.

e. As the user points to each letter, it is displayed in a small rectangular field below the keyboard, indicating which letter has been selected.

f. Once the desired characters have been selected and displayed in the rectangular field, the user can either copy and paste the text into a document or use it for any other intended purpose.

Overall, the process allows for a touchless and convenient way of typing on a virtual keyboard using hand gestures, which can be especially useful for individuals with physical disabilities or in situations where physical contact with devices may be unsafe, such as during a pandemic.

3. Mouse

a. Upon selecting the "Virtual Mouse" option, a small window appears on the screen.

b. The webcam is activated in the first step, allowing for live video capture of the user's face and surrounding environment.

c. In the next step, the webcam detects and tracks the user's eye coordinates and performs pre-processing and feature extraction on the captured data.

d. The extracted features are then used to determine the user's eye movement and translate that into corresponding mouse cursor movement on the screen.

e. The system also enables the user to click on any desired position on the screen by detecting an eye blink, which acts as a mouse click.

In summary, the proposed system uses computer vision techniques to track the user's eye movement and translate it into corresponding mouse cursor movement on the screen, thereby eliminating the need for a physical mouse. The system also allows for clicking on desired positions through eye blink detection.

IV. FIGURES AND TABLES

V. SPECIFICATION

A. Advantages

Assistive technology has been a game changer for people with disabilities and handicaps. It provides them with the tools they need to perform tasks that would otherwise be difficult or impossible. From mobility aids to communication devices, assistive technology has a wide range of applications. Ultimately, this technology is tremendously helpful in empowering disabled individuals to lead more independent and fulfilling lives.

In addition to helping disabled individuals, assistive technology has also played a role in preventing the spread of infectious diseases. With touchless interfaces and voice-activated controls, assistive devices can reduce the need for physical contact and limit the spread of germs. Additionally, remote monitoring and telehealth technologies can help people receive medical care without having to physically visit a healthcare facility. Ultimately, these applications of assistive technology can help to keep people safe and healthy, particularly in times of heightened concern around infectious diseases.

B. Future Scope

The future scope of gesture recognition based virtual mouse and keyboard is promising. As technology continues to advance, we can expect to see even more sophisticated and intuitive systems that allow for seamless gesture-based control of digital devices. One potential application for this technology is in the field of virtual and augmented reality, where gesture recognition could enable more natural and immersive interactions with digital environments. Additionally, gesture-based interfaces could be used in a range of settings, from education and training to healthcare and rehabilitation. Another potential area of growth is in the development of more precise and accurate gesture recognition algorithms, which could enable more nuanced and complex interactions with digital devices. For example, future systems may be able to recognize more subtle hand movements or incorporate haptic feedback to provide a more realistic and intuitive user experience. Overall, the future of gesture recognition based virtual mouse and keyboard technology is bright, with many exciting possibilities for innovation and growth. As the technology continues to evolve, we can expect to see even more innovative and accessible solutions that empower users to interact with digital devices in more natural and intuitive ways.

VI. SOCIAL ASPECTS

The social aspects of the proposed system are significant as it can greatly benefit handicapped or physically disabled individuals who may have difficulty using traditional input devices like a physical keyboard and mouse. By using hand gestures and eye movements to control the virtual mouse and keyboard, these individuals can have greater access to technology and the ability to perform various computer inputs without relying on conventional devices.

Moreover, in the current COVID-19 situation, it is important to reduce the risk of virus transmission through physical contact with devices. The proposed virtual mouse and keyboard system provides a safer alternative as it does not require physical contact with the computer. Instead, it relies on hand gestures and eye movements, which can be detected by a webcam or internal camera. This can greatly reduce the risk of virus transmission and help to maintain a safer and healthier environment.

Overall, the social implications of the proposed system are significant as it can enhance accessibility and reduce health risks for individuals who may have difficulty using traditional input devices or during pandemic situation.

Conclusion

The implemented system enables the control of mouse cursor movement by tracking the movement of the user\'s eyeballs and hand gestures. The system replaces the conventional input devices such as a mouse and keyboard by combining their functionalities. The main aim of this system is to provide a comfortable data entry method that is versatile and portable, especially for small mobile devices. The virtual mouse and keyboard system utilize gesture recognition, cognition, and image processing to move the mouse cursor in accordance with the eyeball movement and to perform keyboard functions using hand gestures. The objective of this system is to make machines more interactive and responsive to human behavior while reducing the cost of hardware. The system provides a technology that is portable, inexpensive, and compatible with any common operating system. The virtual mouse and keyboard system can be customized as per the convenience of the user, and it offers greater security than the conventional keyboard. Although the virtual keyboard and mouse take longer to input characters than conventional keyboards, this technique allows for the design of a keyboard layout that suits individual needs. In summary, the virtual mouse and keyboard system offer a practical and versatile solution for controlling computer inputs without the need for conventional input devices.

References

[1] Real-Time Iris Tracking Using Deep Regression Networks for Robotic Ophthalmic Surgery\", IEEE Access, vol.8, pp.50648-50658, 2020. [2] A Video Processing Based Eye Gaze Recognition Algorithm for Wheelchair Control\", 2019 10th International Conference on Dependable Systems, Services and Technologies (DESSERT), pp.241-247, 2019. [3] Gesture Recognition Based Virtual Mouse and Keyboard,\" 2020 4th International Conference on Trends in Electronics and Informatics (ICOEI)(48184), 2020, pp. 585- 589, doi: 10.1109/ICOEI48184.2020.9143016 [4] Algorithm for decoding visual gestures for an assistive virtual keyboard,\" in IEEE Latin America Transactions, vol. 18, no. 11, pp. 1909-1916, November 2020, doi: 10.1109/TLA.2020.9398632. [5] S. Sadhana Rao,” Sixth Sense Technology”, Proceedings of the International Conference on Communication and Computational Intelligence– 2010, pp.336-339. [6] Game P. M., Mahajan A.R,”A gestural user interface to Interact with computer system ”, International Journal on Science and Technology (IJSAT) Volume II, Issue I, (Jan.- Mar.) 2011, pp.018 – 027. [7] J. Katona, “A review of human–computer interaction and virtual reality research fields in cognitive InfoCommunications,” Applied Sciences, vol. 11, no. 6, p. 2646, 2021. [8] D.-H. Liou, D. Lee, and C.-C. Hsieh, “A real time hand gesture recognition system using motion history image,” in Proceedings of the 2010 2nd International Conference on Signal Processing Systems, July 2010. [9] S. Wankhede and S. Chhabria, “Controlling Mouse Cursor Using Eye Movement,” International Journal of Application or Innovation in Engineering & Management, no. 36, pp. 1–7, 2013. [10] Imperial Journal of Interdisciplinary Research (IJIR) Vol-3, Issue-4, 2017

Copyright

Copyright © 2023 Pratiksha Kadam, Prof. Minal Junagre, Sakshi Khalate, Vaishnavi Jadhav, Pragati Shewale. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.