Eye Controlled Mouse

Abstract

Numerous individuals who have neuro-locomotor deficits or are disabled by injury are unable to use PCs for abecedarian tasks, for example, sending or receiving dispatches, scouring the web, watching their TV programme, or stirring filmland. In a previous study, eyes were set up to be a great candidate for ubiquitous computing since they always move when they engage with calculating gear. Using this underlying knowledge from eye movements, these instances might be renewed using computers. For this aim, we offer a mouse gesture control mechanism that can only be used by mortal eyes. The goal of this study is to provide an open-source generic eye- gesture control system that can efficiently capture eye movements and allow the stoner to do conduct counterplotted to specified eye movements gestures via computer camera. After identifying it on the stoner\'s face, it tracks the pupil\'s motions. It must be accurate in real time for the stoner to use it comfortably, just like any other everyday prejudice.

Introduction

I. INTRODUCTION

Most individuals currently die as a result of illnesses that severely impair them, such as paraplegia, which causes a person to be unable to utilise his body from the neck down. Females are more affected by handicaps than men in the great majority of OECD (Association for Monetary Cooperation and Development) countries [1]. Their eyes are the only organ capable of performing a wide range of movements. In 2011, 518 million people out of a population of 7 billion reported having a disability. On February 7, 2018, around 650 million individuals were handicapped, accounting for 10% of the world population.

Many people with Amyotrophic Lateral Sclerosis (ALS) [3] or who are handicapped are unable to use computers for everyday tasks. In any case, they require the aid of another person when eating. These individuals require assistance with their daily routines. People with impairments now type on computer keyboards by mouthing long sticks. People with impairments will be able to live their lives. More independence as a result of the technique we provide. It will allow them to work, socialize, and enjoy themselves. Innovative and cutting-edge HCI approaches are being created at a rapid pace. This examination sector employs a large number of professionals. Natural eyes provide a wealth of data that may be obtained and used in a variety of applications [2] (for example, connecting with PCs). A person's degree of interest may be determined by their eye movement. Following eye movements allows you to acquire data from a genuine eye. Using Eye developments as control signals to allow direct engagement with interfaces without the need for console or mouse input. Conventional computer input devices such as a mouse, keyboard, and others have been used to link with digital instruments. These PC input devices are inaccessible to people with impairments. This project creates a computer input device for wearable computing and visually challenged people that can only be controlled by the user's eyes [4]. Furthermore, this information might be utilised to give suitable outputs for computer operation, such as moving wheelchairs or commercially available robotic equipment, such as the robotic arm, allowing these people to feed themselves. This would provide them the physical capability while also forcing them to contribute to society. The goal of this inquiry is to investigate and expand on present paths in the global positioning framework for ocular signals. Particularly those locations where physically handicapped people can use computers and programmable controlled devices. As a result, these individuals were able to execute their tasks, enhance the quality of their lives, and go about their everyday activities without the need for ongoing assistance. The vast majority of contemporary eye tracking systems employ real-time video-based pupil tracking.

A. Proposed Framework

To control the framework, the eye signal control framework connects directly with natural eye vision. Eye gesture is a real-time gesture assurance tool that operates a computer mouse cursor using the user's eye motions. The sole requirements for employing the mouse approach are the ability to control a computer and great eyesight in at least one eye. Adults and children with cerebral paralysis, spinal cord injuries, mental injuries, ALS, multiple sclerosis, brainstem strokes, and other conditions can utilize it. The eye gesture control system may be used in many different situations, including homes, schools, medical institutions, long-term care facilities, and enterprises.

A user may use a computer mouse, execute software, and access the internet and email by watching the operation of a system displayed on a screen.

B. Benefits of the Planned System

1. A Mouse Control System That Is Hands-Free
2. Assisting the disabled in using computers.
3. The mouse's pointer is controlled by eye movements.
4. Real-time eye tracking and eye gaze estimates are made feasible by using eye-based human-computer interaction.
5. Using their eyes to do mouse actions such as left-clicking, right-clicking, double-clicking, and other actions.

II. LITERATURE REVIEW

It is well known that there are several sorts of physical computer mice in current technology. The following will discuss the various types and variations of physical mouse.

A. Mechanical mouse

The trackball mouse, which was popular in the 1990s, has a ball within the mouse that is supported by two rotating breakers to show ladies by the ball itself. One comber detects the left/right stir, while the other detects the forward/backward stir. To enable for more precise identification, the mouse's wall is nevertheless coated with a subcaste of firm rubber. The mouse features left and right buttons as well as a scroll, however it is susceptible to declination due to periodic disunion generated by the mouse ball and the breakers themselves.

Operation may cause the sovereign to degrade over time, leaving it unsuitable for descry stir and hence useless. Similarly, the switches in the mouse button are no different, because long-term use can cause the mechanics to become loose, resulting in no mouse clicks until it is removed and repaired.

B. Optic And Ray Mouse

The optic mouse stir depicts motions connected with the underlying face using light emitting diodes (leds), whereas the ray mouse uses coherent ray lights. Unlike its forefather, the mechanical mouse, the optic mouse detects movement using an image array of print diodes rather than breakers.

The purpose of creating this is to remove the declination limitation that impeded its predecessor, allowing it to give greater continuity while giving increased resolution and perfection. The optic mouse can detect stir on even the opaquest verbose face, but not on poly face. Long-term operation without proper cleaning or conservation may also result in dust flyspeck entrapment between leds, resulting in optic and ray swell degradation.

III. METHODOLOGY

1. Packages Used

a. opencv-contrib-python: solve computer vision problems i.e., it helps a lot in image processing.

b. mediapipe by google: which helps to detect eye movement.

c. PyAutoGUI: provides the ability to simulate mouse cursor moves and clicks.

2. Screen size

a. screen_w, screen_h = pyautogui.size() give the size of the screen.

3. while loop

a. while loop will run forever as it will be reading every frame of your video.

b. _, frame = cam.read():We tell cv2 to read whatever is coming from the camera.

4. Flipping the frame

a. frame = cv2.flip(frame, 1) which means you are flipping the screen vertically so that your reflection will be visible on the frame.

5. You have to show the prompt 'Eye Controlled Mouse'.

6. You have to wait for 1 second if you press the key.

7. Declare a variable named face mesh.

8. Then assign mp.solutions.face_mesh.FaceMesh(refine landmarks=True) to face mesh where refine landmarks=True is used torefine the landmarks.

9. Mediapipe i.e., mp has many solutions but we have to just select face mesh and then call Face Mesh() method.

10. Declare a variable named rgb_frame and assign itwith        cv2.cvtColor(frame, cv2.COLOR_BGR2RGB).

11. This variable is used for colour conversion.

12. output = face_mesh.process(rgb_frame), where the output variable is used to process the output of rgb_frame i.e., the video it is capturing is processed.

13. for loop

a. for  id, landmark in enumerate (landmarks [474:478]) where the enumerate give the id or index and the element.

b. This is done so that the cursor also moves by detecting the eye movement.

14. Landmark points = output.multi_face_landmarks, where landmark_points is used for detecting points on face i.e., face recognition.

15. We have taken only one landmark as only one face will be detected by the camera.

16. Each landmark has 3 points:

a. x-axis: detects horizontally.

b. y-axis: detects vertically.

c. z-axis: detects distance between the camera and face.

17. These points are represented in the form of pixel numbers.

18. Frame Height and Frame Width

Take 2 variable named as frame_h and frame_w.

Then assign these 2 varibles with frame.shape which tells which is the height and which is the width.

Multiply x-axis with frame_w and y-axis with frame_h.

x = landmark.x * frame_w

y = landmark.y * frame_h

By multiplying them you will get height and width in floating number which is very big.

But to draw point/ circles and integer number is required.

So typecast it into integer number so that number becomes small and easy to understand.

19. Drawing circle on face

The circle has to be drawn on frame, on the center x and y, radius of circle is 3 and the give the colour to the circle.

20.. Scaling the Screen

We have to scale the screen so that the cursor can move freely without bothering weather the screen in big or not.

21. Detecting the click

Click is detected by blinking of left eye and cursor movement is detected by right eye.

Specify the landmarks to be detected of lest eye.

Then repeat the for loop as there will be movement of face.

After running the code, you will have 2 circles on the left eye and 4 circles on the right eye.

Conclusion

In conclusion, it is unsurprising that the actual mouse will be replaced in mortal computer commerce by a virtual non-physical mouse in which every mouse movement may be done with the touch of your eyes far and wide and at any moment without any environmental constraints. This design had developed a colour recognition programme with the purpose of replacing general physical mouse without immolating the delicacy and effectiveness it\'s appropriate to fete colour movement combinations and restate them into actual mouse function because the delicacy and effectiveness plays an important part in making the programme the programme is useful as the physical mouse view ways had to be enforced we can just apply any function or click on any tab by just blinking of your eyes. The constructed system seeks to provide a less-cost eye- shamus which lets the user to control a computer system\'s mouse cursor. This system is less costly and effective; if necessary, the spatial field of view A history on the world process may be developed to indicate eye movements and where the stoner spent a lot of time looking in order to modify the interface or simply recoup spatial attention data for the objectives defined in the \"future operations\" section. Finally, we can see that the design is flexible to a variety of environmental circumstances, requiring just minor adjustments to the brightness and discrepancy settings to ensure its longevity.

References

[1] Disabled persons Ratio [online] Available At: https://www.disabledworld.com/disability/statistics/(Accessed 2 Jan 2018 [2] The Eye Writer Project [online] Available At: http://www.eyewriter.org (2009-Present) [3] Muhammad Usman Ghani, Sarah Chaudhry “Gaze Pointer: A Real Time Mouse Pointer Control Implementation Based On eye gaze tracking [4] Schmidt, Jochsen, Vogt and Nieman \"Calibration–free hand–eye calibration: a structure–from–motion approach.\" Pattern Recognition. Springer Berlin Heidelberg, 2005. 67-74.) (2005) [5] A Wearable Gestural Interface:www.pranavmistry.com/projects/sixthsense/ (05.04.2017) [6] Heiko Drewes and Albrecht Schmidt. “Interacting With Computers Using Gaze Gestures” https://link.springer.comm/chapter/10.1007/978-3-540-74800-7_43 (2009) [7] PUPIL: constructing the space of visual attention. Diss. Massachusetts Institute of Technology, 2012.) Kassner, Phillip, and Patera (2012)

Copyright

Copyright © 2023 Abhishek Hundalekar, Siddesh Pingale, Vamshi Naidu, Prof. Khushabu Trivedi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.