Virtual Human Interaction System Using CV Learn

Abstract

This project proposes a way to control the position of the cursor with the bare hands without using any electronic device. While the operations like clicking and dragging of objects will be performed with the different hand gestures. The proposed system will only require a webcam as an input device. The software that will be required to implement the proposed system are OpenCV and Python and the libraries used for this model is MediaPipe. The output of the camera will be displayed on the system’s screen so that it can be further calibrated by the user. The python dependencies that will be used for implementing this system are NumPy, math and mouse. It’s a new approach for controlling mouse movement using a real time camera. Major approaches consist of adding more buttons or charging the position of the tracking ball of mouse. Instead we suggest to change the design of hardware. Our concept is to use a camera and computer vision technology, as a image segmentation and gesture recognition, to control mouse tasks and we show how it can perform everything that current mouse devices do.

Introduction

I. INTRODUCTION

A. Project Scope

This project is titled as “Hand tracking,  virtual mouse  using Open cv”. The main objective of the proposed virtual mouse system is to develop  the alternative to the regular and traditional mouse system to perform   and  control   the mouse functions, and this can be achieved with the help of web camera that captures the hand gestures and hand tip and then processes these frames to  perform   the particular mouse functions such as left click, right click, and scrolling function.

B. Project Purpose

The main goal of the project is to manage computers and other devices with gestures other than pointing and clicking a mouse or touch display directly. This is project avoids the physical requirement of mouse and the application has been used designed to be cost effective and uses low cost input tools like webcam for the use  of capturing hand as input.

C. Project Features

The functions of mouse like controlling of movement of virtual object have been replaced by hand gestures. The main feature of the project is to identify the hand gestures and with the help of them we can drag, drop, click and scroll the mouse. This project is designed  to  operate with help of the webcam . It provides an efficient and, interface for a user to interact with the computer and access the various applications, effortlessly.

II. LITERATURE SURVEY

Certain experiments have been Performed from past few years by different research and development groups. Here are some of the following groups:

1. J.Katona proposed A review of human interaction in virtual reality research fields in the easy Cognitive info communications. Cognitive infocommunications (CogInfoCom) is a young and evolving discipline that is at the crossroads of information and communication technology (ICT) and cognitive sciences with many promising results. The goal of the field is to provide insights into how human cognitive capabilities can be merged and extended with the cognitive capabilities of the digital devices surrounding us, with the goal of enabling more seamless interactions between humans and artificially cognitive agents. Results in the field have already led to the appearance of numerous CogInfoCom-based technological innovations. For example, the field has led to a better understanding of how humans can learn more effectively, and the development of new kinds of learning environment have followed accordingly. The goal of this paper is to summarize some of the most recent results in CogInfoCom and to introduce important research trends, developments and innovations that play a key role in understanding and supporting the merging of cognitive processes with ICT.
2. B.J. Bourah, A.K. Talukdar and K.K. Sarma has proposed a way that Development learning-Tool using hand gesture based human computer interaction system . A hand gesture recognition system is a natural and simple way of communicating in today’s world. The development of teaching methods by using technology-dependent useful items to increase communication and interaction between the teacher and the student is a major part of today’s e-learning. In this paper, we have proposed an interactive learning-aid tool based on a vision-based hand gesture recognition system. The system uses MediaPipe for hand gesture recognition. The recognized hand gestures use a virtual-mouse-based object controlling system to control various virtual objects created using Unity. The system has been tested using six hand gestures and it is found that the system can be used effectively for controlling various virtual objects.
3. V.V. Reddy, T. Dhyanchand, G.V. Krishna has proposed a way that virtual colored mouse control using colored fingertips and hand gesture recognition. In human-computer interaction, virtual mouse implemented with finger tip recognition and hand gesture tracking based on image in a live video is one of the studies. In this paper, virtual mouse control using finger tip identification and hand gesture recognition is proposed. This study consists of two methods for tracking the fingers, one is by using colored caps and other is by hand gesture detection. This includes three main steps that are finger detection using color identification, hand gesture tracking and implementation on on-screen cursor. In this study, hand gesture tracking is generated through the detection of the contour and formation of a convex hull around it. Features of hands are extracted with the area ratio of contour and hull formed  and detailed tests are performed to check this algorithm in real world scenarios.

III. PROPOSED SYSTEM

This system is based on gesture recognition method proposed for interactive projection systems. This new method should be more either make human life more comfortable and more productive and more fun. It provides greater flexibility than the existing system.

It uses the library called MediaPipe to detect the hand gestures. It can provide more functions depending on the choice of object. It is easy to modify and adapt. This model is less prone to physical damage due to absence of a fixed physical choice. This model will avoid the mouse-related wrist damage like CTS and RSL.

Also, there is a certain degree of fun and entertainment associated with the whole idea.

Conclusion

A new method for hand gesture is introduced in this project. Using the hand gestures we are going to control the operations of the mouse. On the other hand we are also able to recognize the number of fingers for identifying or recognizing the persons identity. From the results of the model, we can come to a conclusion that the proposed AI virtual mouse system has performed very well and has a greater accuracy compared to the existing models & also the model overcomes most of the limitations of the existing systems. Since the proposed model has greater accuracy, the AI virtual mouse can be used for real-world applications and also it can be used to reduce the spread of COVID-19. Since the proposed mouse system can be used virtually using hand gestures without using the traditional physical names.

References

[1] J. Katona, “A review of human-computer interaction and virtual reality research fields in Cognitive Info Communications”, Journal of Applied Sciences, vol. 11, no. 6, p.2648 in February 2021. [2] B. J. Boruah, A. K. Talukdar and K. K. Sarma, “Development of a Learning-aid tool by using Hand Gesture Human Computer Interaction System,” 2021 Advanced mode of Communication Technologies and Signal Processing (ACTS), 2021, pp.1-5, doi:10.1109/ACTS53447.2021.9708354. [3] V. V. Reddy, T.Dhyanchand, G.V.Krishna, “Virtual mouse control using coloured fingertips and hand gesture recognition”, IEEE-HYDCON, 2020, pp:1-5, doi:101109/HYDCON48903.2020.924267 November 2020. [4] P. Nandhini and J. Jaya,“ Image segmentation for food quality evaluation using computer vision systems”, International Journal of Engineering Research and Applications (IJERA) vol.4, no.2, pp:1-3, March 2020 [5] K. H. Shibly, S. Kumar Dey, M. A. Islam, and S. Iftekhar Showrav, “Design and development of hand gestures based virtual mouse,” in proceedings of the 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), pp. 1-5, Dhaka, in Bangladesh. [6] L. Thomas, “Virtual mouse using hand gesture,” International Research Journal of engineering and Technology (IRJET) vol. 5, no.4, July 2018 [7] A. Haria, A. Subramanian, N. Asokkumar, S. Poddar, and J. S. Nayak, “Hand gesture recognition for human computer interaction,” Procedia Computer Science, vol. 115, pp.364-317, 2017. [8] J.Jaya and K.Thanushkodi, “Implementation of classification system for medical images, “European Journal of Scientific Research, vol.53,no.4,pp.561-569, August 2017.

Copyright

Copyright © 2022 A. Sri Divya, Vineela Gurram, Supraja Masanpally, Devendhar Reddy Kancharla, Harish Naik Ramavath. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.