Smart Eye Tracker Control System

Abstract

Many people are physically injured or paralyzed as a result of an accident, making it impossible for them to engage in daily activities like watching TV or browsing the internet. Even after becoming paralyzed, individuals can instinctively move their eyes and use it. The primary goal of this is to address the issue of any physically challenged individuals who wish to operate freely on a computer system, without depending on any of the person for the daily activities. It is a system that is accessible for free that uses open-source software and a webcam to take pictures of the user\'s face and pupil before doing actions in response to their movements.

Introduction

I. INTRODUCTION

There are various illnesses like hemiplegia and helplessness that result in paralysis and prevent people from using their body parts. Males are less harmed than females, according to the organisation for economic cooperation and development. Only the eyes have some of the capabilities of other organs. 518 million people out of a population of 7 billion have some form of disability. Within the estimated 1.3 billion people who make up the world population in 2022, some individuals with impairments already exist. A illness called amyotrophic lateral sclerosis (ALS) prevents paralysed people from using computers for routine chores. They require assistance even while drinking the water or performing other basic duties. They are reliant on other people and require support to carry out regular tasks. People used to type on keyboards with long sticks in the past. By providing a platform for them to be entertaining, working, interact, and pursue their objectives, the technology we have created will enable physically impaired persons to work independently and without assistance. HCI techniques are rapidly produced. And experts are working on the research field, eye is an organ which is very helpful for individual interest. In most of the cases it is found that all the other body parts are easily affected rather than eyes of any person. Techniques for HCI are developed quickly. The eye is an organ that is very beneficial for individual interest, and researchers are studying in this area. The majority of the time, all other part of body are proven to be more easily impacted than anyone's eyes. You can interact with digital instruments by using input devices such a keyboards, webcams , mouse, microphone. Any human with a disability is unable to utilise a computer.

A. Purpose System

With the help of eye pupil the vision of Normal human gets interacted and based on it the gestures are defined. According to the movement of the pupil it operates the functions of computer system, thus it can make changes on the report of actions of eyeballs.

B. Advantages Of Purposed System

1. Mouse control with a free hand.
2. Allows people with disabilities to use computers more easily.
3. Through a AI-based gesture control system, real-world time tracking and gaze estimation can be done.
4. The blinking of the eyes causes it to do activities, such as selecting if we double blink, selecting if we move  eyes in the right direction, and vice versa.
5. The right eye acts as the right mouse key when we blink it, and the left eye acts as the left mouse key when we blink it.

II. LITERATURE REVIEW

To accomplish the objectives, the literature was reviewed, grasp of the research domain, concentration on the research issues, design of the information collecting strategy, explanation of vocabulary, as well as proper framework identification. The most difficult challenge was understanding the research field, which entailed eye detection as well as that of the mouse cursor.

While reading the literature, the emphasis was on how to develop a system that will be able to meet the needs of physically disabled people while also being simple to understand.

A team at MIT [6] had developed a technology called "The sixth sense" with the objective of transforming Human Computer Interaction by making the use of hands and eye gesture. The entire model might be attached to the user's helmet and used anywhere, and it can also be projected onto surfaces such as the walls. The challenge is that it is not able to generate a system which could possibly communicate with other gadgets or provide increased assistance and accessibility to paralysed persons.

Although Drewes, and Heiko (2009) [7] provided a thorough assessment, it was highlighted that most algorithms required additional improvement because they adopted time-consuming and laborious techniques toward calibration. Schmidt, Jochen [8] described the use of the structure from motion algorithm for human computer interaction (HCI) as a straightforward, adaptable strategy. Kassner, Moritz Philipp, and William Rhoades Patera [9] considerably aided this by using the same sfm (structure from motion) technology, improving it, and expanding its applicability as an effective student tracking strategy. In order to represent this spatial experience and make it useful for eye gesture tracking, they created the PUPIL framework, It seeks to critically investigate the link between a human being and space. A Hough transform-based eye-tracking tracking method was created in 2018 [10]. This technology can recognize a person's face and eyes. A camera is used to recognize the user's face and eyes. The system is built on MATLAB. The challenges with this technology include real-time monitoring and a time-speed constraint The procedure is relatively slow [11] and high-quality products are pricey. computers are required for it to function effectively. An improved system was created for disabled patients in 2017. This method was created for patients who are disabled. This system employs a webcam via MATLAB and directs the mouse cursor using a person's pupil [12, 13]. It took a long amount of time for this system to be able to identify a person's pupil, which was a concern. It employed various algorithms and methods in order to find the pupil.

A wearable eye-gaze monitoring system that is based on vision was introduced in 2016 [14]. A high-infrared camera is employed by this system to function. By employing an infrared camera, it can identify the user's eyes. The system's delay and price were an issue [15].

The circular Hough transform technique was used to introduce pupil centre coordinate detection in 2015 [16]. This method utilizes a camera and Hough-transform Algorithm used to determine a person’s pupil [17]. The disadvantage of this technique was that it took a long time and was not suitable for real-time application. The body takes a long time to capture before moving on to the face, eyes, and finally the pupil.

In 2014, an eye and face-controlled gadget based on MATLAB was developed [18], [19]. The mouse is controlled by moving the webcam's eyes and face. The challenge is that it only functions within a few centimetres of the source. In 2013, [20], a system based on pictograms was developed using eye movement tracking technologies.

It makes the system reliable by utilizing various eye-tracking approaches. The problem is that it will not work if there is any fluid in the eyes. The mechanism malfunctions when women apply eyeliner or mascara to their eyes, for example.

III. METHODOLOGY

The use case diagram represents the cost-effective tracking system that enhances frameworks such as Human-Computer Interaction (HCI) and computer vision to assist individuals with disabilities. The system utilizes real-time input from a camera to track the user's pupil. It performs various tasks by processing the tracked data using computers or micro controllers. The primary objective is to analyse the real-time movement of the pupil and store eye movement data to control the computer mouse pointer. This enables people with disabilities to engage in socializing and communication.

Most computers nowadays are equipped with built-in high resolution (HD) cameras, and they often come with freely available open-source software modules that are easy to install. These modules can be seamlessly integrated with laptops and desktop computers currently in use. To create a prototype of this system, it requires the implementation of specific research works and collaboration with open and free-to-use communities. By incorporating the principles of HCI and computer vision, this cost-effective tracking system empowers individuals with disabilities to effectively control computer interaction through eye movements, enhancing their ability to engage and communicate with others.

 This system represents a comprehensive approach that encompasses the entire process, starting from conceptualization to design and proof of concept.

It involves a combination of research paper implementation and collaborative work with the open-source community. The focus is on designing and constructing a prototype that adheres to the principles of open-source development. This approach ensures the utilization of affordable, easily accessible, and commercially available off-the-shelf (COTS) components.

 By incorporating these elements, the system aims to provide a cost-effective and user-friendly solution for its intended purpose. The use of high-definition web cameras, coupled with the compatible software module, ensures the capture of detailed and clear visuals. The open-platform design allows for easy installation on a variety of devices, enabling widespread adoption among users. Python is a popular and versatile programming language that finds application in various domains. Its dynamic features allow for runtime code modification, enhancing flexibility.

Python's dynamic type checking, among other features, can impact performance, leading to the optimization of dynamic code for efficient programming practices.

1. NumPy: NumPy is a powerful library written in C++ (and FORTRAN) that provides high-performance mathematical functions. By utilizing multidimensional arrays and structures, NumPy efficiently addresses the issue of slower algorithms in Python. Any algorithm can be expressed as an array operation, allowing for faster execution.
2. SciPy: SciPy is a library built on top of NumPy, specializing in scientific computing tasks. It leverages the NumPy array model as the primary data structure and offers modules for various scientific programming tasks, such as linear algebra, integration, differential equation solving, and signal processing (including eye signal processing through webcams).
3. OpenCV: OpenCV is a computer vision library widely used in image and video processing tasks. Initially, new modules are developed separately and stored in the opencv_contrib repository. As modules mature and gain significance, they are integrated into the main OpenCV repository.
4. PyAutoGUI offers a range of automation capabilities, including mouse movement simulation, mouse clicks, mouse dragging, key presses, key holds, and keyboard hotkey combinations. On Windows, PyAutoGUI operates independently without any external dependencies, except for modules like Pillow that are installed automatically with PyAutoGUI via pip. It does not require the installation of the pywin32 module since it relies on Python's native modules.

B. The Activity Diagram

Here is the activity diagram describing the process from start to end:

1. Activate the webcam and begin recording.
2. The computer system starts analysing the captured video feed.
3. The system detects the face of the person.
4. Check if there are any eyes available within the detected face.
5. If eyes are found, proceed to the next step. Otherwise, go back to step 2.
6. The system focuses on the eyes and begins analysing them.
7. Detect the pupils of the eyes using geometrical patterns on the user's face.
8. The system tracks the movement of the pupils to interpret user intentions.
9. Simultaneously, the system monitors the mouse for user gestures.
10. If the user performs a scroll gesture, the system responds by scrolling up or down.
11. If the user performs a gesture indicating horizontal movement, the system responds by moving crosswise while scrolling from left to right.
12. After performing the desired action, the system returns to step 2 to continue monitoring the user's eyes and mouse gestures.
13. The entire process is repeated indefinitely in a cyclic fashion until the recording is stopped or the application is closed.

a. Moving crosswise while scrolling from left to right.

b. The entire operation is repeated indefinitely in a cyclic fashion.

V. IMPLEMENTATION

A. Use Case Diagram

In this Python-based implementation of the system called iMouse, the following steps are followed. First, the camera is opened and video recording begins, generating frames. A frame is selected and converted to grayscale since it simplifies object identification once images are in binary form. The system then uses Haar-cascade to extract the face from the frame. Haar-cascade is a cascade function trained on numerous positive and negative images, enabling it to detect objects in other photos. After identifying the face in the frame, it is cropped for further processing. Subsequently, a technique is employed to detect the eyes using the processed frame. Haar-cascade, trained on positive and negative images, is utilized to recognize objects in photos. In this case, it identifies the face in the frame, crops it, and proceeds to the subsequent processing stage. The eyes are then cropped from the processed frame using Haar-cascade. The eye-cascade is a variation of the Haar cascade specifically designed to detect eyes The starting positions where the camera first detects the eyeballs are denoted by x and y. By utilizing these variables, the image can be cropped by drawing a rectangle around the eyes. The rectangle starts at point x and ends at x+w horizontally, and it starts at point y and ends at y+h vertically.

B. System Evaluation and Analysis

Firstly, the implemented contour detection technique using the Hough Transform and Haar-cascade functions is evaluated, and the results are presented.

???????C. Feature-based cascade classifier

This algorithm enables the computer to perform two operations detecting a person’s face and eyes. The face operation is demonstrated in the cascade diagram. The user's face is initially detected from an image using face-cascade. The image is then cropped, and a box is drawn around the face for subsequent processing. Once the face is extracted from the image, the user's eyes are captured. As both eyes move simultaneously, tracking can be done by observing the movements of only one eye. The program selects either the left or right eye from the image captured by the user's camera and tracks its motion. A box is drawn around this eye, and the image is cropped.

???????D. Hough-Transform Algorithm

The Hough Transform is employed to create a circle around the pupil, enabling the tracing of its movements. This method focuses solely on the pupil of the eye, disregarding other dark spots, and identifies the pupil's location in each frame. When the coordinates of the eyes change, it indicates eye movement; otherwise, the eye is considered stationary. The total processing delay is less than one second, making it suitable for real-world applications.

???????E. Graphical Representation Mouse and Eye

The histograms comparing mouse control and eye control are displayed below. It's important to note that these readings were obtained under ideal lighting conditions and may vary under different circumstances if pupil detection is inaccurate. The 0 and 0 illustrate this point. Results for 0 and 0 are rounded to the nearest millisecond. It can be observed that once proficiency is achieved, the user can perform similar tasks in approximately the same amount of time as with a mouse.

VI. RESULTS

Individuals with disabilities often face challenges when performing simple computer tasks. In such cases, a system can promptly recognize the person's condition. By detecting the movement of the person's pupil, the system establishes a connection between eye motions and mouse movements. Consequently, the mouse pointer aligns with the pupil's movement, while mouse clicks are triggered by eye blinks. To evaluate the effectiveness of this method, tests were conducted involving eye movement and fluid injection. The technology's performance was assessed across varying distances, yielding diverse results.

The following table presents the statistics obtained from the distance evaluations:

Table I.  Distance Coverage by IMouse System

The results demonstrate that our approach achieves accurate and real-time tracking up to a maximum distance of 50 cm. Currently, the system exhibits rapid response, with the maximum movement measuring 6.458533 cm and the minimum movement measuring 4.50689 cm. On average, the mouse travels 1.0965423 cm. At a distance of 20 cm, the average mouse movement is 0.5898252 cm, indicating proper functionality. Similarly, at 30 cm, the average movement is 0.4281392 cm, affirming successful mouse control. As the distance from the system decreases, the mouse speed diminishes gradually. At 50 cm, the mouse movement is slightly slower, but still adequately responsive. The mouse clicking action is swift and devoid of any delays. Utilizing the camera, the system employs the pupil for initiating mouse clicks.

All paragraphs must be indented.  All paragraphs must be justified, i.e. both left-justified and right-justified.

Conclusion

The computer systems tries to offer a simple eye tracker that enables users to operate their computers. System is simple to use and inexpensive because it simply needs a webcam and C++. a Python programming language software module. The spatial views can be rendered in the world procedure as needed to show eye movement and identify where the user spends the majority of their time looking intently in order to customise the interface and extract attention data for the future applications outlined in the future applications section. Finally, take note that the project can be easily deployed across a variety of environments. This is a remarkable accomplishment for such inexpensive eye-tracking technology, requiring only the brightness and contrast tweaks necessary to preserve durability.

References

[1] Lupu, Robert Gabriel, Florina Ungureanu, and Valentin Siriteanu. \"Eye tracking mouse for human computer interaction.\" 2013 E-Health and Bioengineering Conference (EHB). IEEE, 2013. [2] Bozomitu, Radu Gabriel, et al. \"Pupil centre coordinates detection using the circular Hough transform technique.\" 2015 38th International Spring Seminar on Electronics Technology (ISSE). IEEE, 2015. [3] López, Alberto, et al. \"EOG signal processing module for medical assistive systems.\" 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA). IEEE, 2016. [4] Bukhalov, Aleksei, and Viktoriia Chafonova. \"An eye tracking algorithm based on hough transform.\" 2018 International Symposium on Consumer Technologies (ISCT). IEEE, 2018. [5] Klaib, Ahmad F., et al. \"Eye tracking algorithms, techniques, tools, and applications with an emphasis on machine learning and Internet of Things technologies.\" Expert Systems with Applications 166 (2021): 114037.Khekare, Ganesh, et al. \"Experimental Performance Analysis of Machine Learning Algorithms.\" Proceeding of 2021 International Conference on Wireless Communications, Networking and Applications. Singapore: Springer Nature Singapore, 2022. [6] Li, Dongheng, David Winfield, and Derrick J. Parkhurst. \"Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches.\" 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\'05)-Workshops. IEEE, 2005. [7] Bagherzadeh, SH Ziafati, and S. Toosizadeh. \"Eye tracking algorithm based on multi model Kalman filter.\" HighTech and Innovation Journal 3.1 (2022): 15-27. [8] Dhule, Chetan, and Trupti Nagrare. \"Computer vision based human-computer interaction using color detection techniques.\" 2014 Fourth International Conference on Communication Systems and Network Technologies. IEEE, 2014. [9] Llanes-Jurado, Jose, et al. \"Development and calibration of an eye-tracking fixation identification algorithm for immersive virtual reality.\" Sensors 20.17 (2020): 4956. [10] Khekare, Ganesh, et al. \"Testing and analysis of predictive capabilities of machine learning algorithms.\" Integrating Meta-Heuristics and Machine Learning for Real-World Optimization Problems. Cham: Springer International Publishing, 2022. 419-442.

Copyright

Copyright © 2023 Kalyani Ijardar, Mansi Dhakite, Mayuri Kakde, Palak Jaiswal, Tushar Barai. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.