Computer Vision based Virtual Illustration using OpenCV

Abstract

During the COVID-19 epidemic, it has been extremely difficult to engage kids in the classroom via an online platform. Children needed a classroom that was free of dust as a result. This article describes a novel paint application that recognises hand movements and monitors hand joints using MediaPipe and OpenCV. With the use of hand gestures, this programme enables people to interact with computers in a natural way. The main objective of HCI is to enhance human-computer interaction. This study put out a method for writing in the air with your fingertips without gloves or sensors. Air writing enables us to write characters in open spaces and draw shapes with a specific colour on the fingertip. The colour marker is applied to the user\'s fingertip and aids the camera in recognising hand motions.

Introduction

I. INTRODUCTION

The old form of writing is being supplanted by digital art in the age of the internet. Digital art refers to methods of artistic expression and dissemination using digital media. One of the defining features of the digital manifestation is its reliance on current science and technology. Traditional art is any work that was produced before digital art. It may be simply broken down into visual art, audio art, audio-visual art, and audio-visual imaginative art from the perspective of the recipient, which encompasses music, dance, theatre, painting, sculpture, and other forms of art. The relationship and dependence between traditional and digital art are mutual. Although social progress is not a result of popular demand, basic human necessities nonetheless serve as the primary impetus. In art, the same thing takes place. Because conventional and digital art now coexist in a symbiotic condition, it is important to carefully comprehend the fundamental differences between the two types of art.

In this work, we present a virtual paint application that allows real-time sketching or painting on the canvas using hand movements. Utilising cameras to record hand motion, hand gesture-based paint applications may be built. Using vision-based real-time dynamic hand movements, an intangible interface is developed and put into use to carry out tasks including tool selection, canvas writing, and canvas cleaning. A single-shot detector model and MediaPipe are used to process the hands' pictures in real time as they are being captured by the system's web camera. This enables the machine to interact with its user in a matter of milliseconds.

II.  RELATED WORK

Computers may employ automatic object tracking for a variety of purposes, including computer vision and human-machine interaction [1-2]. In the literature, several uses for tracking algorithms are proposed. One group of researchers utilised it to decipher linguistic signals, others to observe hand movements, another text-tracking group also recognised characters based on finger tracking, etc. The body watch the visual movement of things. Bragatto et al. created a method for translating Brazilian Sign Language using video input.  

Use the real-time video capturing capability of the NN multilayer (Neural Perceptron), a network with a line separated into portions. The means NN complex time is decreased by this activation function. They also employ NN in two phases for colour identification and procedures to verify hand form.

Their findings demonstrate that the approach is effective, with an acquisition rate of 99.2%.  Cooper also presented a standard strategy for managing the most complicated 3D cell bioprinting. Cooper created a method for reducing tracking by spotting mistakes in the tracking and thesis division processes. Cooper employed two different therapy approaches, one for his mobility and the other for clarifying the structure of his hands. The screen was utilised by him to increase his vocabulary. Viseme plays a crucial role in oral pronunciation of phonemes and in the creation of visual representations of phonemes. You become less formal as a method to recognise the personalities over time.

Raised hand touch recognition by Araga et al. Jordan's Recurrent Neural Network (JRNN)-using software. Their software utilised a typical sequence of still photos to compare between 5 and 9 distinct hand postures. He then captured the footage when the contour of the hands started to split during re-installation. After a transient behaviour of sequence of places has been identified, JRNN receives input touch. They also developed a fresh approach to training. The accuracy of the suggested procedure is 99.0%.

Yanget al. described another comparative solution sequence of pictures in the pattern, which often happens with the touch of a hand touch, while achieving 94.3% accuracy of nine touches in five distinct places. Skin colour patterns do not support the suggested strategy, and it may be effective in the incorrect divisions. They consist of both the cross-cluster technique used for classification and recognition. On both models, their data suggest a superior 5% performance loss.

Neumann et al. created a method to discover and view text in actual photographs. They employ a multiple-line text-handling hypothesis structure in their own article. Last but not least, they employ the most stable (MSER), which delivers firmness in geometric forms and lighting. They also use artefacts features to train the algorithm.

Wang et al. also covered interior and exterior motion detecting system colours and locations. Use a camera and a tracking device on a t-shirt in the suggested method. The outcome of the suggested approach demonstrates that it may be used for applications in physical world. Jari Hannuksela and others. Babe finger recognition methods by Toshio Asano et al. and Sharad Vikram et al. are based on finger tracking. In order to measure two separate motions, the author develops the based movement tracking algorithm, which combines two Kalmans filtering techniques with expected expansion (EM) approaches. Using the camera, move your fingers. The rate in which we count each image is supported by moving structures. Its goal is to enable simple finger swiping in front of cameras to operate cell phone gadgets. The writers talk about seeing wind-borne Japanese katakana letters. They employ a digital camera and LED pen to track hand motions. They alter the signal pencil to reflect new traffic laws. Codes are often used to complete results of typing speed and provide explanations of 46 Japanese characters. They achieve a character identification accuracy of 92.9% with a single camera, and a 9° directional accuracy with several cameras.

III.  METHODOLOGY

The goal is to create an open area where one may easily draw in air. The RGB camera picks up the fingertip and follows its movement across the whole screen. The first thing to do whenever the hand appears in front of the camera is to find the fingertip. Fingertip detection may be done in many different ways.

A. Hand Pose Recognition

An essential first step in starting airborne composition is recognising the position of the composing hand and identifying it through other indications. Contrary to traditional writing, where the write flows up and down, writing inside the debate isn't organised in the same way. Events. By counting the number of lifted fingers, the framework determines the position of a piece hand and distinguishes it from a non-writing hand.

B. Hand Region Segmentation

After using the over technique to accurately capture the hand, the hand zone is divided using a two-step process, namely the skin division and the removal of the foundation, and the final parallel picture of the hand is obtained as an aggregation of the two. The suggested approach produces division that is somewhat exact and performs well in real-time. Despite the fact that skin tones vary greatly from breed to breed, it has been observed that while skin brightness varies greatly, skin colour only varies somewhat between different skin types.

Background subtraction: Because accurate hand location with the Speedier R-CNN handheld locator followed by sifting of skin colour at the candidate's hand's boundary yields a logically high division result, the background subtraction step is as if used to remove any skin-coloured objects (not a part of the hand) that may be present and are within the bounding box of the recognised hand.

???????D. Drawing Lines and Shapes using Position of the Contour

The purpose of this computer vision extension is to create a Python deque, which is an information structure. We will use these gathered points to create a line using OpenCV's drawing capabilities. The deque will memorise the location of the diagram in each successive outline. Use the layout position to select whether to click a button or draw on the provided sheet right now. Several buttons are located towards the top of the canvas. The pointer is considered to have agreed to the strategy displayed in this region when it enters this range.

1. Position of Object: Image has been removed from the video setup. Remove the colour image from the source: This suggested method monitors the blue file finger's growth. Since we lack a reference image, any previous image may serve as a point of comparison. Take the contrast in the images right now and separate the object's colour and development.
2. Edge Upgrade (EE): The Edge Upgrade technique strengthens the computation of the question area against noise, various lighting circumstances, object obscuration, and object blurring, in fact in different pictures.

???????

Conclusion

This application might put conventional writing techniques to the test. removes the need to carry a handheld device around in order to take notes and provides a convenient option to accomplish the same while on the road. It will once more serve a greater good by making communication simpler, especially for people who are familiar with them. The software is simple enough to use for those who have trouble using the keyboard. Soon, this program\'s capability will allow for the control of IoT devices. Additionally, air painting is possible. With the help of this system, people will be able to interact with the digital world more effectively while wearing smart gear. The material may come alive by illustrating the unpopular reality of taxpayers. Wind-writing programmes should solely respond to their master\'s commands and not be swayed by outside influences. Following are several discovery algorithms that YOLO v3 may use to increase the speed and accuracy of fingerprint identification. The effectiveness of writing in the air will increase in the future as artificial intelligence research advances.

References

[1] Yash Patil, Mihir Paun, Deep Paun, Karunesh Singh, Vishal Kisan Borate, “Virtual Painting with Opencv Using Python”, First International Conference on Computer Engineering International Journal of Scientific Research in Science and Technology Volume 5 Issue 8, November 2020. [2] Hemalatha Vadlamudi, “Evaluation of Object Tracking System using Open-CV In Python”, International Journal of Engineering Rese?rch & Technology (IJERT) Vol. 9 Issue 09, September-2020. [3] Kavya Venugopalan, Safa tp, “Survey On Air Writing Recognition”, IJREAM, Volume 05, Issue 02, DOI:1035291/2454-9150.2019.0084, May 2019. [4] Siddharth Mandgi, Shubham Ghatge, Mangesh Khairnar, Kun?l Gurnani, Prof. Amit H?tekar, “Object Detection And Tracking Using Image Processing”, Vol. 8, Issue 2, ( Part -1) pp.39-41, February 2018. [5] Y. Huang, X. Liu, X. Zhang, and L. Jin, \"A Pointing Gesture Based Egocentric Interaction System: Dataset, Approach, and Application,\" IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 370-377, 2016. [6] P. Ramasamy, G. Prabhu, and R. Srinivasan, \"An economical air writing system is converting finger movements to text using a web camera,\" International Conference on Recent Trends in Information Technology (ICRTIT), Chennai, pp. 1-6, 2016. [7] Saira Beg, M. Fahad Khan and Faisal Baig, \"Text Writing in Air,\" Journal of Information Display, Volume 14, Issue 4, 2013. [8] Rafiqul Zaman Khan, Noor Adnan Ibraheem, “HAND GESTURE RECOGNITION: A LITERATURE REVIEW”, IJAIA, Volume 3, No. 4, July 2012. [9] Brown, T. and Thomas, R.C. (2000). “Finger Tracking for the Digital Desk”, Proceedings of First Australasian User Interface Conference (AUIC 2000), 11–16, Canberra, Australia. [10] Alper Yilmaz, Omar Javed, Mubarak Shah, \"Object Tracking: A Survey\", ACM Computer Survey. Vol. 38, Issue. 4, Article 13, Pp. 1-45, 2006.

Copyright

Copyright © 2023 Shamela Yousuff, Shravya , Shreya Rao, Raza Farhan, Rejeesh Rayaroth. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.