Abstract

The Whiteboard Application Using OpenCV is a digital platform designed to replicate the functionality of traditional physical whiteboards while harnessing the capabilities of computer vision technology. This application allows users to draw, write, and collaborate in a digital space, enhancing remote communication, creative expression and interactive learning experiences. The application supports real-time collaboration, allowing multiple users to engage in synchronous drawing and annotation on a shared canvas. The integration of collaborative features promotes remote teamwork, virtual brainstorming, and interactive educational sessions. The application includes an undo/redo functionality that maintains a history of actions, enabling users to correct mistakes or explore different design iterations without limitations. The user interface (UI) is designed for ease of use and efficient navigation. The intuitive interface ensures that users, regardless of their technical proficiency, can seamlessly access and employ the application\'s features.

Introduction

I. INTRODUCTION

The whiteboard application is used to train our laptop or our screen to read whatever the user will be writing Infront of the screen in the air and that will be displayed on it. We will use some of the machine and deep learning libraries which includes open cv algorithms and functions are used for various tasks related to image processing, computer vision, and user interface development.

To track the detected hand over time, we use various tracking algorithms, such as optical flow, deep sort, to estimate the motion of pixels between consecutive video frames, allowing you to monitor the hand's movement freely. Weal souse CNN algorithm to train dataset of hand motions images to recognize specific gestures. And a user interface which is like a software on which all this can be happen and usedby the end users. These traditional whiteboard software requires highly cost devices, Needs the latest device equipment which are not affordable to every user The primary goal of the project is to create software that prioritizes user-friendliness and affordability, ensuring accessibility and ease of use for variety of users. To provide a virtual teaching board platform that uses hand gesture recognition in air to write on screen. To make the teaching process more effective we developed different user interfaces with functionalities like selection and drawing, Selecting colors for writing and Eraser for erasing with the help of OpenCV, Media Pipe, NumPy libraries and Python in this Project. Hand movement recognition acts as a pathway for computers to interpret human body language, fostering a stronger connection between machines and humans. The user advocates for enhancing user- friendliness by enabling computer systems to be controlled using manual gestures. They use the application which involves the integration of advanced computer vision and machine learning techniques to recognize and interpret gestures, facilitating users in creating and manipulating digital content effortlessly through hand gestures. It features a user-friendly interface suitable for individuals of varying skill levels and utilizes libraries like MediaPipe, NumPy, and OpenCV for gesture recognition within a whiteboard application. We have to perform gesticulate at the forefront of camera so thatthe project can recognize hand moment with thehelp of mediapipe library.

II. RELATED WORK

This application tracks users hand movement from live web cam and use OpenCV colour features to track hand movement and implement draw, erase and write on screen features. The main objective of this The goal is to design an environment enabling freehand drawing in the air, leveraging a webcam to capture RGB images frame by frame. This system detects the fingertip, monitors its movements across the screen, and translates hand gestures into drawing actions each time the hand is within the camera's view.

This tool is created by utilizing several techniques and methodologies related to computer vision, hand tracking, and user interaction.

A.  Hand Tracking

It makes use of a hand tracking module to detect and track hand landmarks in real-time. This likely involves computer vision techniques, possibly based on Convolutional Neural Networks (CNNs) for hand landmark detection.

B. User Interface Design

The code implements a simple graphical user interface (GUI) that displays the webcam, the drawing canvas, and the selected header image The system's interface dynamically adapts and evolves in response to hand motions and user interactions, ensuring a seamless and responsive user experience.

C. Gesture Recognition

This recognizes hand gestures by analyzing the locations of specific landmarks on the hand. Different finger configurations are used to determine whether the user is in selection mode or drawing mode. This is achieved by checking the whereabouts of individual fingers.

D. Drawing on Canvas

The user can draw on the canvas using their hand gestures. It captures the hand position and user's input text is displayed on the canvas. Different colors and brush thicknesses are used for writing and erasing.

E. Image Processing

Some image processing techniques are used, such as converting the canvas to grayscale, creating an inverted binary image, and combining it with the original frame to achieve a drawing effect.

F. Real-time Video Processing

The code continuously captures frames from the webcam, processes the hand landmarks in real-time, and updates the display accordingly. This involves using OpenCV for video processing.

G. Aim-based Classification

The goal is to establish an open canvas for freehand drawing in the air, utilizing an RGB camera to identify and track the fingertip's movements across the screen whenever the hand is brought into view. To enhance real-time performance in tracking fingertip movements on the screen, we opt for the KCF tracking algorithm over the computationally intensive faster R-CNN handheld detector, which improves frame rates and overall tracking efficiency. Each frame is analyzed using the MediaPipe library to extract hand landmarks, numbered from 0 to 20, which serve as key points for hand pose identification. These landmarks are subsequently input into a pre- programming gesture recognition network to accurately recognize hand-gestures.

III. METHOD AND EXPERIMENTAL DETAILS

While testing the model, we first look whether it is recognizing all the hand landmarks when all the fingers are up and also ensure you have the "Header" folder containing header images in the same directory as your Python script. Each landmark have x, y & z. So to test, we use x & y co-ordinate to find information or to find location for the landmark on hand. Now we first use the two fingers, Index and Middle for selection mode by raising only these two fingers and assigning x1, y1 as lmList[8][1:] for Index finger and x2, y2 as lmList[12][1:] for Middle finger by which it can able to perform the selection. Similarly, when only Index finger is detected then it changes into drawing mode. Also testing how smoothly it is drawing on the screen.

1. If Selection Mode: In this step we test whether finger-tip values are detected and if they are two fingers up then selection process will start. User can select color from the header and each selection will have different colors and eraser.
2. Drawing Mode: For this we check whether index finger is up or not and find position of the finger and take input as color and draw on screen. This is shown in figure one which demonstrates the 21 hand points as identified by MediaPipe used for recognizing hand gesture. MediaPipe uses a single- shot palm detection model to recognize the region of the hand, after which it performs exact point localization of 21 3D palm coordinates within this area. These key points from zero to twenty play vital roles in identification of hand poses and are fed into pretrained gesture recognition network. The MediaPipe pipeline combines several models: a palm detection model responsible for extracting an oriented hand bounding box, and a hand landmark model designed for obtaining high-quality 3D key points  on hands  in the  cropped image region.

???????

Conclusion

The Whiteboard application using machine learning model project provides an exciting and interactive canvas for users to express themselves, explore their artistic talents, and engage with technology in a creative way. By seamlessly combining hand tracking, gesture recognition, and various interactive features, this project offers a unique and enjoyable drawing experience, making it suitable for both entertainment and educational purposes. Whether used as a creative outlet, a tool for communication, or a gaming platform, this project promises to captivate users and immerse them in a world of digital artistry.

References

[1] Dr. S. K. Sonkar1 , Gouri Aher2 , Nikita Anerao 3, Shivam Gavhale AdityaKadlag in International Journal of Advanced Research Science,Communication and Technology (IJARSCT) Volume 6, Issue 1, June 2021. [2] Swapna Singh, Hiya Shukla, Keshav Sharma, Harsh Tyagi, Jaya Prasad in IPEC Journal of Science & Technology, Vol. 02 (01), June 2023. [3] Ashvita Salian in Industrial Engineering Journal ISSN: 0970-2555 Volume : 52, Issue 7, No. 4, July : 2023. [4] Dr.G.Shyama Chandra Prasad, 2Varsha Bandi , 3Baline Yugendhar 4 Daki AnilRaj in IJCRT | Volume 11, Issue 5 May 2023. [5] Faria Soroni, Sakik al Sajid, Md. Nur Hossain Bhuiyan, Junaid Iqbal, Mohammad Monirujjaman Khan propsed a Hand Gesture Based Virtual Blackboard Using Webcam in IEEE in oct 2021

Copyright

Copyright © 2024 Mr. V. Narasimha, K. Sandeep, V. Eshwar, Saba Zareen. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.