Driver Drowsiness Detection and Alert System based on Visual Analysis using Machine Learning

Abstract

The main idea behind this project is to develop a system that can detect the drowsiness of the driver and issue a timely warning. Driver Fatigue is the main reason for a large number of road accidents. The detection can be done in many different ways and by using different parameters. The proposed system uses the behavior of parameters. Our proposed method can distinguish the simulated drowsy and sleepy states from the normal state of driving on the resolution images of faces and eyes observed from an oblique viewing angle. This system treats the automatic detection of the driver\'s drowsiness based on visual information and artificial intelligence.

Introduction

I. INTRODUCTION

Driving over long stretches of motorway may be monotonous and exhausting. In recent years, sleepiness and exhaustion have overtaken other factors as the leading causes of serious traffic accidents in India and around the world. The considerable growth in the percentage of traffic accidents owing to tiredness and exhaustion captured the researcher's attention. Also, it has been noted that when tiredness increases, so does a driver's performance [1], [2]. According to NHAI research, 90% of accidents that occur in the middle of the night are caused by sleepy or fatigued drivers. This viewpoint has significantly increased the production of intelligent automobiles. Driving while fatigued can be effectively managed with the help of autonomous vehicles [3].

Future Smart Vehicle Systems (SVS) will incorporate drowsiness detection by taking into account a variety of driver fatigue parameters, including eye blinking rate, eye closure time, eye brow shape, yawning, drivers gestures other than vehicle speed, steering motion, break and accelerator pattern, and continuous driving duration. There are numerous methods used to identify exhaustion and drowsiness. Medical characteristics like heart rate, pulse rate, and so forth might be used as additional parameters. High vision cameras are integrated into SVS to take pictures of the drivers while they are driving and produce notifications accordingly. Regarding the aforementioned issue, several academics have suggested various methods to automate the technique of determining a driver's level of tiredness. A light-insensitive system using Haar algorithms [5] [6] to recognize objects and faces was proposed by Malla et al. [4] based on vision. The degree of eye closure was used as a gauge to determine the driver's level of intoxication. Later, Vitabile et al. [7] published a sleepiness detection system that used an infrared camera to identify the signs of a sleepy driver. The researchers came up with an algorithm for eye tracking and identification using the idea of bright pupils. The system used to sound an alarm when it noticed the driver was becoming asleep.

To identify a driver's tiredness, the aforementioned current methodologies all used sophisticated techniques. An easy solution for the same was suggested in this post. The proposed piece is structured as follows: The proposed approach is described in Section-II, which is then followed by Section-III, which discusses the experimental findings, including the accuracy levels. Section-IV concludes by listing the proposed work's conclusion, future scope, and limitations. After the future scope of the investigation, concluding remarks will be provided.

II. PROPOSED METHODOLOGY

To determine the level of sleepiness and exhaustion, the suggested work implements pre-existing features for facial landmark identification. Using low resolution photographs of faces and eyes taken from an oblique viewing angle, our proposed approach can distinguish between the simulated drowsy and sleepy states from the usual driving state. So, our system may be able to properly monitor the driver's degree of attention without the need for additional cameras. Our strategy might improve the efficacy and scope of current vision-based strategies for detecting driver fatigue. The proposed system requires us to determine if the driver is sleepy or not. CNN and the Haar cascade algorithms were employed (i.e. Harr cascade algorithm to detect face and CNN algorithm to trained dataset of Model).

A. Workflow

High definition cameras are integrated to watch, record, and extract individual frames in order to produce alerts as necessary. With the use of Haar Cascade Classifiers, the aspect ratios of the eyes and mouth are calculated for each extracted frame in order to analyze the pattern of facial features [12]. A blink or yawn is regarded when the EAR and MAR readings are above their respective threshold values. If eye blinking rate and yawning are suspected for a predetermined number of consecutive frames, the system warns the driver by playing an alarm. The alarm is set off to get the driver's attention and continues to ring until the driver wakes up.

B. Facial Features and Gestures Detection

1. Frame Acquisition: A high-end digital camera with a close-up mode positioned in the vehicle allows the driver's full view to be taken without obstructing his or her vision. Real-time video has been recorded, and to determine the driver's current condition, frames have been retrieved and analysed in real-time.
2. Face Detection: Using the Harr Cascade and Convolutional Neural Network (CNN) detectors, the driver's face is located in a frame. Convolutional neural network and Harr cascade detectors were combined primarily to improve accuracy while reducing false positives.
3. Eye Detection & Yawn Detection: To find the 68 pixels in (X, Y) coordinates of a face's facial landmarks, we use the pre-trained shape predictor in the dlib library. In this, facial landmarks are identified using image intensities and regression trees. To find the lips and eyes in a face region, the likelihood of distance between pixel pairs is used.

C. Blink Rate Calculation

Eye Aspect Ratio (EAR), which is the ratio of the vertical distance between the lower and upper eyelids to the horizontal length of an eye, is used to detect blinks. After a blink, the distance between the respective eyelids tends to grow. While the eye is open, the vertical distance between the lower and upper eyelids reduces accordingly. As a result, the blink count is increased while EAR drops (getting closer to zero). The EAR count less than the threshold value reports suspicion in driver’s behavior.

Mathematically, EAR is calculated using equation

D. Yawn Counts

With the dlib landmarks predictor function, the mouth is represented by the 8-coordinates. The landmarks are marked in a clockwise direction, beginning at the left corner of the mouth. The horizontal and vertical coordinates have been seen to have some relationship with one another. Calculating MAR involves dividing the horizontal distance between the lip corners by the ratio of the vertical distance between the lower and upper lips. The gap between the lower and upper lips widens as a person opens his mouth to yawn. In contrast to EAR, the yawn count value is increased as soon as the MAR value rises above a certain level.

E. Alarm Activation

When the blink rate and yawn count surpass the designated thresholds for a predetermined period of consecutive frames, the system will infer that the driver is nodding asleep and will sound an alarm to wake him or her up.

III. EXPERIMENTAL RESULTS

A car's inbuilt Driver Drowsiness Detection and Alert System, which is based on Visual Analysis using Machine Learning, is used to test real-world driving scenarios. In the suggested system, photos are taken with a high-resolution camera in night mode under various lighting situations and on varied subjects. 15 frames per second are used to capture and process the images. The suggested technique is developed in the Open CV environment with the goal of real-time yawning and eye blinking rate detection of the driver's state of drowsiness and weariness. If a threshold state is sustained for at least 10 frames in a row, the rate of blinking and frequency of mouth opening and shutting are used to determine the state of sleepiness. To detect yawning, the Drowsiness system uses Eye Aspect Ratio (EAR), which performs better than Mouth Aspect Ratio (MAR). The combined EAR and MAR features are more effective at assessing the driver's level of exhaustion and level of sleepiness.

After collecting the frames for two minutes, the real-time data in Tabs. 1 and Tabs. 2 be obtained. With an increase in frame rate, the detection rate drops. Based on the rate of eye blinking, Tab. 1 displays the drowsiness detection for 10 samples in real-time dataset. This dataset demonstrates that, out of 10 occasions, drowsiness was detected. The only sample that fails is when the data is collected at night and the driver's face is illuminated directly. When this happens, eye blinking typically slows down, raising the possibility of sleepiness. In a similar vein, Tab.2 offers yawn data from 10 real-time datasets that indicate drivers' tiredness and fatigue based on yawn features. The combined properties of yawning and eye blinking are shown in Tab.3. Combined feature of eye blinking and yawn shows 100% accuracy in the result.

IV. ACKNOWLEDGMENT

The preliminary project research paper on "Driver Drowsiness Detection and Alert System that is based on Visual Analysis using Machine Learning" is presented with great joy. I would want to take this opportunity to thank Mrs. Mohini Avatade, my guide, for providing me with all the support and direction I required. I sincerely appreciate their thoughtful assistance. Their wise advice was quite beneficial. For his valuable guidance and suggestions, I am particularly grateful to Dr. Amol Dhakne, Head of the Computer Engineering Department at the Dr. D. Y. Patil Institute of Engineering, Management & Research, Akurdi, Pune. I would like to extend a particular thank you to Dr. Amol Dhakne for offering several resources for our Project, including a laboratory with all necessary software platforms and ongoing guidance.

Conclusion

The Driver Drowsiness Detection and Alert System is used in the suggested work to develop a real-time sleepiness and fatigue detection system for drivers. The project is based on behavior analysis, a high-end camera installation, and a traditional algorithm to find potential coordinates to identify eyes and mouth. State of the art approaches already in use are computationally complex compared to our suggested solution. Eye blinking and yawning are thought to be crucial indicators of driver weariness and drowsiness based on real-time data collection and processing, which can be used to trigger an alarm if necessary. In the future, wearable technology should be provided to identify additional indicators, such as blood pressure and pulse rate, in addition to eye blinking and yawning, to more effectively and correctly detect driver drowsiness and exhaustion and reduce the risk of traffic accidents.

References

[1] J. A. Horne and L. A. Reyner, “Sleepinees detection for vehicle driver or machine operator”, Nov. 6 2001, uS Patent 6,313,749. [2] D. Dinges and M. Mallis, “Managing Fatigue by drowsiness detection: Can technological promise be realized?”, in International Conference on Fatigue and Transportation, 3rd, 1998, Fremantle, Western Australia, 1998 [3] M. V. Yeo, X. Li, K. Shen, and E. P. Wilder-Smith, “Can SVM be used for automatic eeg detection of drowsiness during car driving?”, Safety Science, vol.47, no.1 , pp. 115-124, 2009. [4] A. M. Malla, P.R. Davidson, P. J. Bones, R. Green, and R. D. Jones, “Automated Video-based measurement of eye closure for detecting behavioral microsleep”, in 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. IEEE, 2010, pp. 6741-6744. [5] P. Viola, M. Jones et al., “Rapid object detection using a boosted cascade of simple features,” CVPR (1), vol. 1, no. 511-518, p. 3, 2001. [6] R. Lienhart and J. Maydt, “An extended set of haar-like features for rapid object detection,” in Proceedings. International Conference on Image Processing, vol. 1. IEEE, 2002, pp. I–I. [7] S. Vitabile, A. De Paola, and F. Sorbello, “Bright pupil detection in an embedded, real-time drowsiness monitoring system,” in 2010 24th IEEE International Conference on Advanced Information Networking and Applications. IEEE, 2010, pp. 661–668. [8] B. Bhowmick and K. C. Kumar, “Detection and classification of eye state in ir camera for driver drowsiness identification,” in 2009 IEEE International Conference on Signal and Image Processing Applications. IEEE, 2009, pp. 340–345. [9] N. Otsu, “A threshold selection method from gray-level histograms,” IEEE transactions on systems, man, and cybernetics, vol. 9, no. 1, pp. 62–66, 1979. [10] A. S. Baquhaizel, S. Kholkhal, B. Alshaqaqi, and M. Keche, “Person re-identification based on sad and histogram in a non-overlapping camera network,” Advances in Systems Science and Applications, vol. 18, no. 3, pp. 1–16, 2018. [11] “https://hackaday.io/project27552-blinktotextlog/68360-eye-blink-detection-algorithms.” [12] A. Sahayadhas, K. Sundaraj, and M. Murugappan, “Detecting driver drowsiness based on sensors: a review,” Sensors, vol. 12, no. 12, pp. 16 937–16 953, 2012.

Copyright

Copyright © 2023 Rohit Karegaonkar, Akshay Gaikwad, Rutik Ahirekar, Mohini Avatade. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.