Driver Drowsiness Detection Using Smartphone

Abstract

Transition state between being awake and asleep is called drowsiness. Driver drowsiness is the major cause of traffic crashes and financial losses. This abstract presents a mobile technology using smartphones to visual indicators of driver drowsiness, allowing the possibility of making drowsiness detection systems more affordable and portable. This technology uses the front camera of a smartphone to capture images of drivers, and then uses smartphone vision algorithms to detect and track the face and eye of the drivers. Eye blinks are then detected as indicators of driver drowsiness. A simulated driving study showed that drowsy drivers differed significantly in the frequency of and eye blinks, compared to when they were attentive. The smartphone-based Driver-Drowsiness detection technology may have important applications in reducing drowsiness-related improving driving safety. This abstract describes the steps involved in designing and implementing a driver drowsiness detection system based on smartphone. It combines off the-shelf smartphone components for eye state (open vs. closed) classification. Preliminary results show that the system is reliable and tolerant to many real-world constraints. Driver drowsiness is a highly problematic issue which impairs judgment and decision making among drivers resulting in fatal motor crashes. This describes a simple drowsiness detection approach for a smartphone with Android / IOS application using Android Studio 4.4.2. & Mobile Vision API for drowsiness detection before and while driving. Quick facial analysis were performed to check drowsiness before the driver starts driving. Facial analysis was undertaken by eye blinking duration. Blinking duration is used to indicator for drowsiness. A performance accuracy of drossiness detection proved to be around 90%.

Introduction

I. INTRODUCTION

Drowsiness is one of the significant reasons for road crashes that results in considerable damaging consequences to the individuals who suffer fatal or non-fatal injuries, property damage and economic losses to the nation. We are familiar with the hazards of drinking and driving or even texting and driving, but many people underestimate the dangers of drowsy driving. Each year, drowsy driving accounts for about 100,000 crashes, 71,000 injuries, and 1,550 fatalities, according to the National Safety Council (NSC). Drowsy driving contributes to an estimated 9.5% of all crashes, according to AAA. However, the actual number may be much higher as it is difficult to determine whether a driver was drowsy at the time of a crash. In many cases, drowsy driving is as dangerous as driving while impaired by alcohol.

Common symptoms that have been identified during drowsy driving include: constant nodding, difficulty opening eyes, missing road signs and turns, frequent lane drifting and difficulty in maintaining speed.

These traffic-related deaths and financial losses have encouraged the development of technologies to mitigate the risks of driver drowsiness.

Indicators of drowsiness include the movement of driver’s face, eye blink rate etc. Another approach is to use electrophysiological signals to detect driver drowsiness. A more technically feasible and financially practical solution is to build the drowsiness detection technology using smartphones. The computational powers of smartphones keep increasing, which allows the computer vision algorithms to run reasonably fast in smartphones.

A smartphone-based drowsiness detection technology would be more portable and affordable than many alternative drowsiness detection systems, which use devoted in vehicle cameras.

The proposed system in this paper uses a smartphone (either an Android smartphone or an iPhone) as a driver drowsiness detection system. The front camera of a smartphone captures images of the driver, and then feeds the images to the CPU of the smartphone for image processing. Intel’s open-source computer vision framework, the OpenCV 2.3 for the Android and iOS operating system, is used to develop the computer vision algorithms for face detection and eye detection.

The original contributions of this paper are develop a simple and user friendly and non-invasive android application for a smartphone to detect drowsiness to detect drowsiness before the driver enters the vehicle using physiological and facial analysis, to detect drowsiness while driving using facial analysis.

The paper is organized as follows: Methodology, System Requirements, Result, Conclusion

II. METHODOLOGY

A. System Setup

Figure 1 shows the setup of the proposed driver drowsiness detection system in real-world driving. The smartphone can be mounted on the dashboard of a vehicle. The smartphone is placed horizontally with the front camera aimed towards the driver’s face.

B. Algorithm framework

The drowsiness detection algorithms are carried out in five steps. The first step is image pre-processing. Figure 2 depicts the algorithm framework for the driver drowsiness detection system. Computer vision performance depends heavily on the image size, which is a great concern for smartphone applications considering the limited computational resources of smartphones. The second step is face detection. A face is located using Haar-like feature detectors. The third step is eye detection within the upper half of the detected face. The fourth step is blink detection by estimating the changes in black pixels within the eye regions. The fifth step is drowsiness judgment

C. Image Pre-Processing

The camera of the smartphone captures images at the resolution of 720×1280. The raw image is first reduced to a resolution of 180 × 320, a quarter of its original resolution. Then, the resized color image is converted to a grey image. Resized images and grey/color conversion can greatly reduce the amount of data that need to be processed, which makes the smartphone application run much faster.

D. Haar-Like Feature Detector

A Haar-like feature detector is used for face detection and eye detection. The Haar-like feature detector using the AdaBoost algorithm is commonly used in face detections because it is fast and accurate. A Haar-like feature detector considers adjacent rectangles at a specific region in a moving detection window. An image area can be described as the combination of different Haar-like features. The number and type of Haar-like features can be used to represent different objects. We define an accumulated sum of intensity from the origin as: ( ) i j x0 y0 S i, j I  (i, j) = = = ∑ ∑ where I (i,j) is the intensity at the location (i,j), S(i,j) is the accumulated sum of intensity from origin at the location (i,j).

The sum of intensity of a rectangle in the image, defined as two points at (Xleft, Yup) and (Xright, Ydown), can be easily calculated as the following equation, which greatly speeds up the computation:

Sacc (Xright ,Ydown) - Sacc (Xleft ,Ydown)- Sacc (Xright,Yup)+ Sacc (Xleft,Yup)

E. Face Detection

The face is detected using the Haar-like feature detector for faces. The centroid of the face is then calculated and used to determine head nod and head rotation. A head nod is defined as a large velocity of the head centroid, with vertical velocity larger than horizontal velocity. A head rotation is defined as a large velocity change of the head centroid, with the horizontal velocity larger than the vertical velocity. An empirical value of 100 pixels per second is set as the velocity threshold for the horizontal and vertical head movement.

F. Eye Detection

The commonly used algorithms for eye detection include Hough transform, template matching, Principle Component Analysis (PCA). In this paper, the eye is detected with the using the Haar-like feature detector for eyes. To improve computational efficiency, the search area for eyes is limited to the upper portion of the detected face. Figure 3 depicts the areas of interest for eye detection within a face. Horizontally, the eye area is between 1/6 to 5/6 of the face; vertically, the eye area is within 1/4 to 1/2 of the face. The width and height of the areas of interest for eye detection is determined by the following criterions:
 Waoi = Wface × 1/4  Haoi = Hface × 1/3

G. Blink Detection

 An eye blink is detected using the change of black pixels in the eye region. An open eye will have a larger visible pupil than a blinking eye. The pupil is often darker than the color of the rest of the eye. Thus, an open eye should have more black pixels. In the detected eye region, the eye image is converted to a binary image using image thresholding. Then, the ratio of black pixels in the binary image of the eye is calculated. The ratio of black pixels is used as the criterion for eye blinks. Figure 4 shows an example of an open eye and a closed eye, and their binary images after thresholding

H. Drowsiness Detection

Three criteria are used to determine the state of driver drowsiness, that is, the frequency of head nods and head rotationsand PERCLOS (Percent Eye Closed). A drowsy driver may exhibit frequent head nodsand eye blinks.

PERCLOS is another important and frequently used indicator of drowsiness. PERCLOS is defined as the percent of time when the eyes are closed in a short time window (often 30s). An eye is treated as closed if the height of the visible pupil is smaller than 30% of its maximum opening.

In this paper, PERCLOS is calculated using the following equation:

 PERCLOS = 30× / Nblink X S

Where Nblink is the number of eye blinks in the recent 30 second time window. S is the sampling rate. Figure 5 shows a screenshot of the Driver drowsiness application in detection mode. It shows the development version of the application running in an android. Faces of drivers were not shown in order minimize potential visual distractions. The spinning ring in the center indicates the application is running. Users can set the parameters such as alert sound and volume in the Setting tab. Their frequencies of   drowsiness report by time or by drives can be viewed in the Report tab.

III. SYSTEM REQUREMENTS

A. Built In Smartphone Camera

Use smartphone built-in camera to detect the driver's eyes. In order to make the camera works as a monitoring camera, the developers implemented the application used that camera to alert the driver

B. Stand For Mobile

This tool is purchased by the user, it’s used to stand the mobile in a suitable position to allow the application to detect driver's eye and to improve application performance

C. Software’s

1. Snapdragon SDK for Android: As mentioned earlier Snapdragon SDK is a package of software libraries, which is working in Android mobile platform. It has sample code, and documentation designed to facilitate the apps development and to integrate a host of next-gen technologies into apps that run on Snapdragon processor-powered the mobile devices. These features help Drowsiness detection app to include capabilities like facial processing, facial recognition, and more to transform the driver's experience.
2. Android OS: Android is a software stack for mobile devices that includes an OS, middleware and key applications. It’s also a free, open-source mobile platform. Android is not a device or a product and it is not even limited to a certain brand of mobile phones - you could build a DVR, a handheld GPS, an MP3 player.
3. Eclipse: Eclipse is a platform used for building integrated web and application development tooling. It helps in acceleration of development of integrated features based on a plug-in model. “It provides a common user interface (UI) model for working with tools”.
4. Java SDK: To build, test, and debug the Android application, we used the Android SDK which provides the API libraries and developer tools necessary. It consists of the essential Android SDK components and a version of the Eclipse IDE, with built-in ADT (Android Developer Tools).

IV. RESULTS

A. Self-Reported Drowsiness
At the beginning and ending of each session, subjects self-reported their drowsiness in two scales, the Stanford Sleepiness Scale (SSS) and Karolinska Sleepiness Scale (KSS). As shown in Figure 9, the self-reported drowsiness steadily increases across drive sessions in both scales.

B. Eye Movements

When the eye blinks the percentage of black pixels in the eye area decreases Thus, the standard deviation of percent of black pixels reflects the frequency of eye blinks. As shown in Figure12, the standard deviation of percent of black pixels in the eye area increases gradually as driving duration increases, producing a marginally significant main effect of drive sessions, Pair-wise comparisons show that the percent of black pixels in session 1 is significantly smaller than that of session 3 The percent of black pixels in the eye area does not differ between session 1 and session 2 , or session 2 and session 3 respectively.
The frequency of eye blinks is shown in Figure13. The blink frequency gradually increases as driving duration increases, producing a  marginally significant main effect of drive sessions. Pair-wise comparisons show that the blink frequency in session 1 i significantly smaller than that of session 2 and session 3 and respectively. The blink frequency does not differ between session 2 and session 3.

The average PERCLOS is shown in Figure14. The average PERCLOS increases numerically, but does not reach statistical significance. Pair-wise comparisons show that the average PERCLOS in session 3 is significantly larger than that of session 2 The average PERCLOS does not differ between session 1 and session 2, or session 1 and session 3 and respectively.

The increases in the standard deviation of percent black pixels in the eye area, the blink frequency and the average PERCLOS indicate that the eye movement behaviours change systematically as a result of long driving duration.

C. Discussion

 To reduce the risks of driver drowsiness, we developed a smartphone-based technology to monitor visual indicators of driver drowsiness, including head nods, head rotations, and eye blinks. This smartphonebased drowsiness detection technology provides a portable and affordable alternative to existing drowsiness detection systems.

Computer vision technology has a wide application in improving driving safety. Many safety technologies (such as drowsiness detection system, forward collision warning system and lane departure warning system) rely on computer vision algorithms. However, most of the computer vision-based technologies need devoted cameras and computer processors, thus, making the price of the technologies too high to be affordable for average drivers. The invention of smartphones makes the processors and cameras much smaller in its size and more portable. Smartphone-based computer vision technologies for driving safety does not require additional equipment than the smartphones drivers already have. Thus, smartphone-based technologies are more

likely to be commonly adopted by average drivers and gain a wide popularity. For example, researchers have developed lane departure warning systems using Android/iPhone and a vehicle detection system using an Android smartphone. To our best knowledge, the drowsiness detection algorithms described in this paper are the first time implemented in smartphones. Drowsiness detection using smartphones has the potentials to be more widely adopted than traditional camera based or EEG based drowsiness detection technology.

With the limited computational resources of smartphones, our algorithms still achieve a high sampling rate of about 7Hz, which is fast enough for practical applications. We strive to obtain this high sampling rate by grey color conversion, image resizing, and localization of the areas of interest. The rapid development of smartphone technology and its hardware will allow even better computational performance of the drowsiness detection algorithm.

Future study should include research to improve the quality of face and eye detection under low-lighting conditions, for example, during night time driving on the highway. Infrared illuminator may be able to improve the quality of face and eye detection and enhance the algorithm performance Besides eye blinks and head movements, other indicators of driver drowsiness, such as yawn detection and vehicle dynamics, should also be considered and incorporated for driver drowsiness detection.

This smartphone-based drowsiness detection is not without limitations. For example, the computer vision algorithms use battery quickly. Drivers are expected to use this technology with a car charger to power the smartphone. Future breakthroughs in battery technology may reduce this limitation. Another limitation is that eye detection is difficult for drivers wearing sunglasses. When eye detection is not possible, head nods and rotations can still be used to detect drowsiness. In these scenarios, yawn detection and vehicle dynamics should be considered to compensate for the reduction of drowsiness detection performance when eye blink information is not available.

This research contributes to the effort to detect driver drowsiness by providing a working prototype for real-time drowsiness detection using an Android smartphone or an iPhone. This technology has important applications for improving driving safety.

D. Stanford Sleepiness Scale

1. Feeling active, vital, alert, or wide awake
2. Functioning at high levels, but not at peak; able to concentrate
3. Awake, but relaxed; responsive but not fully alert
4. Somewhat foggy, let down
5. Foggy; losing interest in remaining awake; slowed down
6. Sleepy, woozy, fighting sleep; prefer to lie down

E. Karolinska Sleepiness Scale

Please circle the item which best describes your current sleepiness level.

1. Extremely alert
2. Very alert
3. Alert
4. Rather alert
5. Neither alert nor sleepy
6. Some signs of sleepiness
7. Sleepy, but no effort to keep alert
8. Sleepy, some effort to keep alert
9. Very sleepy, great effort to keep alert, fighting sleep

Conclusion

The aim of this work was to design and implement a user-friendly driver monitoring and drowsiness detection application. Android Studio 4.4.2 software was used for developing the application. The application mainly tested the facial analysis of driver and . Mobile vision Face API was used for face detection. The face orientation was computed through pose angle estimation in the Y and Z plane. Previous projects have used machine learning and AI algorithms for training data and data classification which ultimately provided very accurate results while driving. This work, however, aimed at providing ease of use, availability, reduced cost and privacy since data was stored in the user phone before and while driving. The driver could be tested for drowsiness even before driving. It could be concluded that physiological analysis for drowsiness detection yielded an accuracy of around 88.5% and is comparable to accuracies obtained from ECE and EEG sensors. The Drowsiness detection through facial features inspection before driving improved the system performance to approximately 90.8%.

References

[1] T.Covington,Thezebra.com,2020.URLhttps://www.thezebra.com/resources/research/drowsy-drivingstatistics/#:~:text=Drowsy%20driving%20statistics%20 in%202019&text=23.6%25%20of%20all%20respondents%20said,least%20once%20in%20thei %20lifetime.&text=Men%20(32.9%25)%20were%20more,compared%20to%20women%20(22.2%25) . [2] J. M. Owens, T. A. Dingus, F. Guo, Y. Fang, M. Perez, J. McClafferty and B. Tefft. Prevalence of drowsy- driving crashes: Estimates from a large-scale naturalistic driving study. Research Brief, AAA Foundation for Traffic Safety, Washington, DC, February 2018. URL, https://aaafoundation.org/prevalence-drowsy-driving-crashes-estimates-large-scale-naturalistic-driving-study/ [3] Drowsy driving. URL. https://www.nsc.org/road-safety/safety-topics/fatigued-driving [4] Facts about drowsy driving. URL, https://drowsydriving.org/wp-content/uploads/2009/10/DDPW-Drowsy-Driving-Facts.pdf , 2020. [5] Meet Android Studio. URL, https://developer.android.com/studio/intro. [6] Driver drowsiness detection. URL https://www.bosch-mobility-solutions.com/en/solutions/interior/driver-drowsiness-detection/ , 2020. [7] D. F. Dinges and R. Grace. PERCLOS: A valid psychophysiological measure of alertness as assessedby psychomotor vigilance. US Department of Transportation,FederalHighwayAdministration,Publication NumberFHWA-MCRT-98-006,1998.URL, https://rosap.ntl.bts.gov/view/dot/113/dot_113_DS1.pdf [8] J. Dai, J. Teng, X. Bai, Z. Shen and D. Xuan. Mobile phone based drunk driving detection. 4 th International Conference on Pervasive Computing Technologies for Healthcare, pages 1-8, March 22, 2010. URL, https://ieeexplore.ieee.org/abstract/document/5482295 [9] D. Sarkar and A. Chowdhury. A real time embedded system application for driver drowsiness and alcoholic intoxication detection. International Journal of Engineering Trends and Technology, 10(9):461-465, 2014. URL, http://www.ijettjournal.org/archive/ijett-v10p288 [10] H. Wang, C. Zhang, T. Shi, F. Wang and S. Ma. Real- time EEG-based detection of fatigue driving danger for accident prediction. International Journal of Neural Systems, 25(02):1550002, 2015. URL, https://pubmed.ncbi.nlm.nih.gov/25541095/ [11] D. M. Morris, J. J. Pilcher and F. S. Switzer III. Lane heading difference: An innovative model for drowsy driving detection using retrospective analysis around curves. Accident Analysis & Prevention. 80:117-24, 2015. URL, https://pubmed.ncbi.nlm.nih.gov/25899059/ [12] F. Li, H. Zhang, H. Che and X. Qiu. Dangerous driving behavior detection using smartphone sensors. 19th International Conference on Intelligent Transportation Systems (ITSC), pages. 1902-1907, 1 November, 2016. URL, https://www.semanticscholar.org/paper/Dangerous-driving-behaviordetection-using-sensors-Li-Zhang/f9d291d05185728b118fac5dc6fcac71d6fdc5fb [13] A. Dasgupta, D. Rahman and A. Routray. A smartphone- based drowsiness detection and warning system for automotive drivers. IEEE Transactions on Intelligent Transportation Systems, 20(11):4045-4054, 2018. URL, https://www.semanticscholar.org/paper/A-Smartphone-Based-Drowsiness-Detectionand-Warning-Dasgupta-Rahman/1d80267c3685e7cc87297fc3b23d2a40fee82f01 [14] I. Chatterjee and S. Roy, \"Smartphone-based drowsiness detection system for drivers in real-time,\" 2019 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), Goa, India, 2019, pp. 1-6,doi:10.1109/ANTS47819.2019.9117943.URL, http://www.scielo.org.za/scielo.php?script=sci_arttext&pid=S2309-89882021000100004 [15] J. He, S. Roberson, B. Fields, J. Peng, S. Cielocha and J. Coltea. Fatigue detection using smartphones. Journal of Ergonomics, 03:1-7, 2013. URL, https://www.longdom.org/abstract/fatigue-detection-using-smartphones-20707.html [16] W. Deng and R. Wu. Real-time driver-drowsiness detection system using facial features. IEEE Access, 7:118727-118738, 2019. URL, https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8808931 [17] S. Hachisuka, T. Kimura, K. Ishida, H. Nakatani and N. Ozaki. Drowsiness detection using facial expression features. SAE Technical Paper, 2010.URL, https://saemobilus.sae.org/content/2010-01-0466

Copyright

Copyright © 2022 Suraj Mali, Kiran Bangale, Rohan Walekar, Prof. P .T Shitole. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.