Driver Drowsiness Detection Using Face Recognition

Abstract

Every year, hundreds of people die in traffic accidents all over the world. According to national statistics, human factors play a determining role in 90-95% of car accidents in Iran. Globally, 25% of accidents are caused by driver fatigue and around 60% of road accidents result in death or serious injury. In a National Transportation Research Institute (NTSRB) study of 107 randomly selected car crashes, fatigue accounted for 58% of all crashes. Drowsy driving is a major factor in serious road accidents that claim thousands of lives every year. The application of intelligent systems in automobiles has developed considerably in recent years. These systems use wireless sensor networks to monitor and transmit car and driver status. Smart cars that use software technology to control engine speed, steering, transmission, braking, etc. Management and control, the quality of management has been greatly improved. Ad hoc networks are the first systems to develop automatic navigation in cars. A notable weakness of these systems is that their response to changes in the environment is not in real time. This is especially important in driving, where time is a key factor in a driver\'s decision. On the other hand, another way to check driver fatigue is to monitor the physical condition and facial expression of the driver, which cannot be processed and transmitted accurately enough by wireless sensor networks. Driver fatigue is a major factor in a large number of road accidents. Recent statistics estimate that 1,200 deaths and 76,000 injuries are attributable to fatigue-related accidents each year.

Introduction

I. INTRODUCTION

Drowsy driving is a leading cause of 4,444 road deaths. Truckers who drive for  long periods of time (especially at night), bus drivers on long trips, or night buses are more likely to have this problem. Drowsy drivers are a nightmare for passengers in every country. Every year, many people are injured or killed in fatigue-related traffic accidents. Therefore, driver fatigue detection and  indication is an active research area  due to its wide practical applicability.A basic drowsiness detection system consists of three modules/modules: an acquisition system, a processing system, and an early warning system. Here,  video from the driver's  face is captured in the acquisition system and transmitted to the processing block, where it is processed online to detect drowsiness. The warning system sends a warning or alert to the driver if drowsiness is detected.  In general, drowsy driving detection methods fall into three categories: vehicle-based, behavior-based, and physiology-based. In the vehicle-based approach, various parameters such as steering wheel movement, acceleration or braking pattern, vehicle speed, lateral acceleration, deviation from lane position, etc. .are monitored constantly. Any abnormal changes detected in these values ??are considered  driver drowsiness. This is a non-intrusive measurement because the sensor is not attached to the conductor. In behavior-based methods [1-7], the visual behavior of the driver is, blink, close your eyes, yawn, lower your head, etc. Analysis to detect drowsiness. It is also a non-intrusive measurement as  simple cameras are used to detect these features. In  physiological methods [8,9], physiological signals such as electrocardiogram (ECG), electrooculogram (EOG), electroencephalogram (EEG), heart rate, pulse, etc. The level of drowsiness or fatigue  is monitored and detected from these measurements.This is an intrusive measure because the sensor is attached to the driver, distracting the driver. Depending on the sensors used in the system, the cost and size of the system increases.  However, including more parameters/features increases the accuracy of the system somewhat. These factors motivated us to develop a real-time, low-cost driver drowsiness detection system with acceptable accuracy. Therefore, we propose a webcam-based system to detect driver fatigue from facial images only using image processing and machine learning techniques, which makes the system cheap and portable.

II. METHODOLOGY

Originally, the video was recorded with a webcam. Cameras will be placed in front of the driver to capture perspective. Extract frames from  video to get 2D images. Detect faces in an image using a directed gradient histogram (HOG). After detecting a face, facial features such as eyes, nose and mouth are marked on the image [11].Quantify from facial features, eye aspect ratio, mouth opening rate and head position, and use these features and machine learning methods to judge driver drowsiness. If drowsiness is detected, an alert is sent to the driver to remind him. 

A. Data Acquisition  

video was recorded with a webcam (Sony CMU-BR300) and images were extracted and processed on a laptop computer. Once the frames are extracted, image processing techniques are applied to these 2D images. Currently, synthetic player data has been generated. The volunteers were asked to watch a webcam while intermittently blinking, closing their eyes, yawning and tilting their heads. The video was shot for 30 minutes. 

B. Face Detection 

After extracting the image, first detect the face. There are many face detection algorithms on the internet, and this time I am learning Histogram of Oriented Gradients (HOG).This method takes fixed window size positive samples from images and computes HOG descriptors on them. Then the negative sample  (the sample that does not contain the object you want to detect, i.e. e) takes a face of the same size and calculates the HOG description. Usually the number of negative samples is much larger than the number of positive samples. After obtaining the features of the two classes, train a linear SVM for the classification task. 

C. Working

This project revolves deeply around the prediction of visual cues for a given face. We can accomplish a lot with these metrics. From detecting eye blinks in videos [3] to predicting a subject's emotion. The applications, consequences and possibilities of visual cues are vast and interesting. Dlib's pre-built model, which is basically an implementation of [4], not only allows fast face detection, but also allows us to accurately predict 68 2D face landmarks.

Using these predicted facial features, we can create suitable features that additionally allow us to detect certain actions, such as detecting eye blinks using eye aspect ratio (more on that below) or Blink detection using mouth aspect ratio - same as yawning etc. can even detect beaks. In this project, these movements were programmed as triggers to control drowsiness. The PyAutoGUI library is used to move the cursor. Eye Aspect Ratio (EAR)

You will see that the eye aspect ratio [1] is the easiest and most elegant function to get the most out of facial features. Ears help us blink [3] and wink, etc.

You can see  the EAR value dropping when you close your eyes. We can train a simple classifier to detect drops. However, the  normal state works  fine. Sometimes as follows: 

If EAR<=SOME_THRESHOLD: EYE_STATUS = 'CLOSE'

D. Mouth Aspect Ratio (MAR)

Highly inspired by the EAR feature, I tweaked the formula a little bit to get a metric that can detect opened/closed mouths. Unoriginal but it works.

Similar to  EAR,  MAR values ??increase when the mouth is opened. A similar intuition applies to this measure. 

E. Classification

 After calculating the three features, the next task is to detect drowsiness in the extracted images. To get started, consider adaptive classification thresholds. Later, machine learning algorithms were used to classify the data. Threshold calculations for each function assume that the driver is initially fully awake.

This is called the setup phase. The  records the  EAR value for the first three hundred frames of the setup phase (10 seconds at 30 fps). Of these three hundred initial images containing faces, an average of 150 maxima is taken as the hard threshold for EAR. Higher values ??are taken into account so that blinding does not occur.

If the test value is below this threshold, closed eyes are detected (i.e. , drowsy). Since eye size varies from person to person, each person's initial settings will reduce this effect. Similarly, for the calculation of the MOR threshold, the mouth which cannot open to the maximum in the initial frame (localization stage), this threshold is obtained experimentally from  observations. If the test value is above this threshold, a yawn is detected (i.e. , asleep).

After calculating threshold , the system is used for testing. The system detects drowsiness if drowsiness is detected for at least one feature in the test frame.

To make this threshold more realistic, each frame decision is based on the last 75 frames. Gives a drowsiness detection indication and a warning if at least 70 images (out of 75 images) meet the drowsiness criteria for at least one attribute.  To make this threshold adaptive, another single threshold is initially calculated from the EAR threshold.

The average of the EAR values ??is calculated as the average of the 150 largest values ??over 300 frames during the configuration phase. The offset is then determined heuristically and subtracted from the mean to obtain the threshold.

When the EAR falls below this threshold, driver safety is compromised. This EAR threshold increases slightly with each yawn until  a certain limit is reached.  Because each yaw and head tilt is spread across multiple frames, the yaw and head tilt of consecutive frames are treated as a single yaw and added once to the adaptive threshold. As part of the test, if the EAR value falls below this adaptive threshold, drowsiness is detected and the driver is alerted.  uses a machine learning algorithm in addition to thresholding to detect drowsiness.

EAR and MOR values ??are stored with actual sleepiness records from synthetic test data. Prior to classification, a statistical analysis is performed on the features.  Initially, the characteristic space is transformed into an independent space. After transforming the feature values, use a Student's t-test to test whether the features are statistically significant for both classes. Since all three features were statistically significant at the 5% significance level, all three features were used for classification using a support vector machine [12].

Conclusion

After reviewing the results, we can conclude that we offer a low-cost, real-time driver drowsiness monitoring system based on visual behavior and machine learning. Here, visual behavioral characteristics such as eye aspect ratio and mouth aspect ratio are calculated from streaming video captured by a webcam. An adaptive threshold technique is developed to detect driver drowsiness in real time. The developed system can accurately process the generated synthetic data. Then store the feature values and use SVM for classification.

References

[1] W. L. Ou, M. H. Shih, C. W. Chang, X. H. Yu, C. P. Fan, \"Intelligent Video-Based Drowsy Driver Detection System under Various Illuminations and Embedded Software Implementation\", 2015 international Conf. on Consumer Electronics - Taiwan, 2015 [2] W. B. Horng, C. Y. Chen, Y. Chang, C. H. Fan, “Driver Fatigue Detection based on Eye Tracking and Dynamic Template Matching”, IEEE International Conference on Networking, Sensing, and Control, Taipei, Taiwan, March 21-23, 2004. [3] S. Singh, N. P. papanikolopoulos, “Monitoring Driver Fatigue using Facial Analysis Techniques”, IEEE Conference on Intelligent Transportation System, pp 314-318. [4] B. Alshaqaqi, A. S. Baquhaizel, M. E. A. Ouis, M. Bouumehed, A. Ouamri, M. Keche, “Driver Drowsiness Detection System”, IEEE International Workshop on Systems, Signal Processing and their Applications, 2013. [5] M. Karchani, A. Mazloumi, G. N. Saraji, A. Nahvi, K. S. Haghighi, B. M. Abadi, A. R. Foroshani, A. Niknezhad, “The Steps of Proposed Drowsiness Detection System Design based on Image Processing in Simulator Driving”, International Research Journal of Applied and Basic Sciences, vol. 9(6), pp 878-887, 2015. [6] R. Ahmad, and J. N. Borole, “Drowsy Driver Identification Using Eye Blink Detection,” IJISET - International Journal of Computer Science and Information Technologies, vol. 6, no. 1, pp. 270-274, Jan. 2015. [7] A. Abas, J. Mellor, and X. Chen, “Non-intrusive drowsiness detection by employing Support Vector Machine,” 2014 20th International Conference on Automation and Computing (ICAC), Bedfordshire, UK, 2014, pp. 188- 193. [8] A. Sengupta, A. Dasgupta, A. Chaudhuri, A. George, A. Routray, R. Guha; \"A Multimodal System for Assessing Alertness Levels Due to Cognitive Loading\", IEEE Trans. on Neural Systems and Rehabilitation Engg., vol. 25 (7), pp 1037-1046, 2017. [9] K. T. Chui, K. F. Tsang, H. R. Chi, B. W. K. Ling, and C. K. Wu, “An accurate ECG based transportation safety drowsiness detection scheme,” 343IEEE Transactions on Industrial Informatics, vol. 12, no. 4, pp. 1438- 1452, Aug. 2016. [10] N. Dalal and B. Triggs, “Histograms of Oriented Gradients for Human Detection”, IEEE conf. on CVPR, 2005. [11] V. Kazemi and J. Sullivan; \"One-millisecond face alignment with an ensemble of regression trees\", IEEE Conf. on Computer Vision and Pattern Recognition, 23-28 June 2014, Columbus, OH, USA. [12] Richard O. Duda, Peter E. Hart, David G. Stork, “Pattern Classification”, Wiley student edition.

Copyright

Copyright © 2023 Peddagoni Sai Pavan Goud, Nenavath Aravind. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.