Advanced Driver Assistance System for Drivers using Machine Learning and Artificial Intelligence Techniques

Abstract

Machine learning techniques have been used in order to predict the condition and emotion of a driver to provide information that will improve safety on the road. It is an application of artificial intelligence. The face, an important part of the body, conveys a lot of information. When a driver is in a state of fatigue, the facial expressions, e.g., the frequency of blinking and yawning, are different from those in the normal state. In this paper, we propose a system called “Advanced Driver Assistant System”, which detects the drivers fatigue status, such as yawning, blinking, and duration of eye closure, using video images, without equipping their bodies with devices. Artificial Intelligence is a method by which systems can automatically learn as well as improve without being explicitly programmed. A driver’s condition can be estimated by bio-indicators, behavior while driving as well as the expressions on the face of a driver. In this paper we present an all-inclusive survey of recent works related to driver drowsiness detection and alert system. We also present the various machine learning techniques such as CNN algorithm, HAAR based cascade classifier, OpenCV which are used in order to determine the driver’s condition. Finally, we identify the challenges faced by the current systems and present the corresponding research opportunities.

Introduction

I. INTRODUCTION

Currently, transport systems are an essential part of human activities. We all can be victim of drowsiness while driving, simply after too short night sleep, altered physical condition or during long journeys. The sensation of sleep reduces the driver's level of vigilance producing dangerous situations and increases the probability of an occurrence of accidents. Driver drowsiness

and fatigue are among the important causes of

road accidents. Every year, they increase the number of deaths and fatalities injuries globally.

In this context, it is important to use new technologies to design and build systems that are able to monitor drivers and to measure their level of attention during the entire process of driving.

In this paper, a module for ADAS (Advanced driver assistance System) is presented in order to reduce the number of accidents caused by driver fatigue and thus improve road safety. This system treats the automatic detection of driver drowsiness based on visual information and artificial intelligence.

II.  RELATED WORK

In 2007, Arimitsu et al., developed the driving simulator with the seat belt motor retractor, which was used in a commercial vehicle, to provide the vibration stimulus to the drivers. The limitation of this paper was  ariation of the portions, which was stimulated by the seat belt. In 2008, Liang et al., proposed a novel brain computer interface (BCI)

system that can acquire and analyze electroencephalogram (EEG) signals in real-time to monitor human physiological as well as cognitive states, and in turn, provide warning signals to the users when needed. The accuracy of the BCI system is slightly less when compared to the existing systems to detect drowsiness. In 2010 Lin et al., proposed a system consists of a wireless physiological signal-acquisition module and an embedded signal-processing module. In case if defects in the EEG monitor then the detection of drowsiness may decrease.

Facial landmarks recognition. The purpose of facial keypoints recognition is that getting the crucial information about locations of eyebrows, eyes, lips and nose in the face. With the development of deep learning, it is the first time for Sun to introduced DCNN based on CNN to detect human facial keypoints. This algorithm only recognizes 5 facial keypoints, albeit its speed is very fast. To get a higher precision for facial key points recognition, Zhou [11] employed FACE++ which optimizes DCNN and it can recognize 68 facial keypoints, but this algorithm includes too much of a model and the operation of this algorithm is very complicated.Wu proposed Tweaked Convolutional Neural Networks (TCNN) which is based on Gaussian Mixture Model (GMM) to improve different layers of CNN. However, the robustness of TCNN depends on data excessively. Kowalski introduced Deep Alignment Network (DAN) to recognize the facial keypoints, which has better performance than other algorithms. Unfortunately, DAN needs vast models and calculation based on complicated functions. So in order to meet the requirement about real time performance, DriCare uses Dlib to recognize facial keypoints. DriCare, is built using a commercial camera automobile device, a cloud server that processes video data, and a commercial cell phone that stores the result.DriCare system, While driving, the automobile’s camera captures the driver’s portrait and uploads the video stream to the cloud server in realtime. Then, the cloud server analyzes the video and detects the driver’s degree of drowsiness. In this stage, three main parts are analyzed: the driver’s face tracking, facial key region recognition, and driver’s fatigue state. To meet the real-time performance of the system, we use the MC-KCF algorithm to track the driver’s face and recognize the facial key regions based on key-point detection. Then, the cloud server estimates the driver’s state when the states of the eyes and mouth change.

III. PROPOSED METHOD

A. Proposed System Algorithm

1. Viola-jones face detection algorithm is used to detect the face the images and given as input to Viola-jones eye detection algorithm.
2. Once the face is detected, Viola-jones eye detection algorithm is used to extract the eye region from the facial images and given as input to CNN.
3. CNN with four convolutional layers are used to extract the deep features and those features are passed to fully connected layer.
4. Soft max layer in CNN classifies the images in to sleepy or non-sleepy images.

The proposed architecture of Drowsiness detection system using Deep CNN. The proposed model has three phases:

a. Preprocessing stage,

b. Feature extraction,

c. CNN Classifier.

B. System Architecture

Following Figure shows the Architecture of the system

C. Face Detection and Eye Region Extraction

 Whole face region may not be required to detect the drowsiness but only eyes region is enough for detecting drowsiness. At first step by using the Viola-jones face detection algorithm face is detected from the images. Once the face is detected, Viola-jones eye detection algorithm is used to extract the eye region from the facial images. In 2001, P Viola and M Jones developed the Viola-Jones object detection algorithm [20, 21], it is the first algorithm used for face detection. For the face detection the Viola-Jones algorithm having three techniques those are Haar-like features, Ada boost and Cascade classifier. In this work, Viola-Jones object detection algorithm with Haar cascade classifier was used and implemented using OPEN CV with python. Haar cascade classifier uses Haar features for detecting the face from images.

D.  Feature Extraction and Classification

Feature extraction is one type of dimensionality reduction where useful parts of an image represented as a feature vector. In this paper features from the eye region images are extracted using a Convolutional Neural Network (CNN). 3.3.1 Convolutional neural network

Convolutional neural network (CNN) is used in the proposed system for detection of driver drowsiness. Since a feature vector is needed for each drowsy image to compare with existing features in a database to detect either drowsy or not. Usually, CNNs requires fixed size images as input so preprocessing is required. The preprocessing includes extracting the key frames from video based on temporal changes and store in database. From these stored images, feature vectors are generated in convolution layers of CNN. These feature vectors are then used for the detecting the driver drowsiness. CNN have layers like convolutional layers, pooling (max, min and average) layers, ReLU layer and fully-connected layer. Convolution layer is having kernels (filters) and each kernel having width, depth and height. This layer produces the feature maps as a result of calculating the scalar product between the kernels and local regions of image. CNN uses pooling layers (Max or Average) to minimize the size of the feature maps to speed up calculations. In this layer, input image is divided into different regions then operations are performed on each region. In Max Pooling, a maximum value is selected for each region and places it in the corresponding place in the output. ReLU (Rectified Linear Units) is anonlinear layer. The ReLU layer applies the max function on all the values in the input data and changes all the negative values to zero. The following equation shows the ReLU activation function.

f(x)=max(0,x) (1)

The fully-connected layers used to produce class scores from the activations which are used for classification.

E. Layers Design of Proposed Deep CNN Model

In our proposed work, a new Deep CNN model is designed for detection of driver drowsiness using deep learning based on Eye State. Figure shows the designed CNN model used in this work.

In the proposed method, 4 convolutional layers and one fully connected layer are used. Extracted key images with size of 128 X 128 are passed as input to the convolution layer-1 (Conv2d_1). In Conv2d_1 input image is convolved with 84 filters of size 3x3. After convolution, batch Normalization, non-linear transformation ReLU, Max pooling over 2 × 2 cells are included in the architecture, which is followed by dropout with 0.25%. Conv2d_1 required 840 parameters. Batch_normalization_1 is done with 336 parameters. The output of convolution layer-1 is fed in to the convolution layer-2(Conv2d_2). In Conv2d_2, input is convolved with 128 filters with size 5x5 each. After convolution, batch Normalization, non-linear transformation ReLU, MaxPooling over 2 × 2 cells with stride 2 followed by dropout with 0.25% applied. Conv2d_2 required 268928 parameters. Batch_normalization_2 required 512 parameters. The output of convolution layer-2 is fed in to the convolution layer-3(Conv2d_3). In Conv2d_3, input is convolved with 256 filters with size 5x5 each. After convolution, Batch Normalization, non-linear transformation ReLU, MaxPooling over 2 × 2 cells with stride 2 followed by dropout with 0.25% applied, Conv2d_3 required 819456 parameters. Batch_normalization_3 required 1024 parameters.

The output of convolution layer-3 is fed in to the convolution layer-4(Conv2d_4). In Conv2d_4 input is convolved with 512 filters with size 5x5 each. After convolution, Batch Normalization, non-linear transformation ReLU, Max Pooling over 2 × 2 cells with stride 2 followed by dropout with 0.25% applied. Conv2d_4 required 3277312 parameters. Batch_normalization_4 required 2048 parameters. Fully connected layer that is dense_1 required 8388864 parameters. Proposed CNN model required 12,757,874 trainable parameters. The output of classifier is two state, so output layer having only two outputs.Adam method is used for Optimization. Here softmax classifier is used for classification. In our proposed CNN framework, the 256 outputs of fully connected layer are the deep features retrieved from input eye images. The final 2 outputs can be the linear combinations of the deep features.

IV. ADVANTAGES

1. Its helps prevent accidents caused by the driver getting drowsy.
2. Driver drowsiness detection helps to avoid crashes caused by fatigue by advising drivers to take a break in time.
3. It’s possible to play music during driving.
4. Its help to search data on Google during driving using text to speech.
5. Traffic management can be maintained by reducing the accidents.
6.  

V. LIMITATIONS

1. Dependence on Ambient Light: With poor lighting conditions even though face is easily detected, sometimes the system is unable to detect the eyes. So it gives an erroneous result which must be taken care of. In real time scenario infrared backlights should be used to avoid poor lighting conditions.
2. Optimum Range Required:  When the distance between face and webcam is not at optimum range then certain problems are arising. When face is too close to webcam (less than 30 cm), then the system is unable to detect the face from the image. So, it only shows the video as output as algorithm is designed so as to detect eyes from the face region. This can be resolved by detecting eyes directly using haardetectobjects functions from the complete image instead of the face region. So, eyes can be monitored even if faces are not detected.
3. Hardware Requirements: Our system was run in a PC with a configuration of 1.6GHz and 1GB RAM Pentium dual core processor. Though the system runs fine on higher configurations, when a system has an inferior configuration, the system may not be smooth and drowsiness detection will be slow. The problem was resolved by using dedicated hardware in real time applications, so there are no issues of frame buffering or slower detection.

VI.  ANALYSIS RESULT

VII. FUTURE SCOPE

1. The future works may focus on the utilization of outer factors such as vehicle states, sleeping hours, weather conditions, mechanical data, etc., for fatigue measurement.
2. In Our system will add more function like chat bot, chat conversation, etc.
3. In future will add speed limit warning.
4. In future we will add Fuel alert and google map speech in our system.
5. We can also introduce smart lane driving assist.
6. It can provide service in various fields where, twenty-four-hour operations, high annual mileage, exposure to challenging environmental conditions, and demanding work schedules all contribute to this serious safety issue.

Conclusion

This paper provides a comparative study on papers related to river drowsiness detection and alert system. In order to provide a solution to the problem of detecting the state of drowsiness, an arithmetic-based method is used. This system uses eye movement in order to detect fatigue. Eye movement is detected using a camera. This is done to recognize the symptoms of fatigue in order to avoid accidents. It is based on the concept of eye-tracking. In order to obtain finer results, a hundred and fifty images of different people have been used. If the state of fatigue has been identified, an alarm system is turned on. In this paper, we presented the conception and implementation of a system for detecting driver drowsiness based on vision that aims to warn the driver if he is in drowsy state. Face and eye region are detected using Viola-Jones detection algorithm. Stacked deep convolution neural network is developed to extract features and used for learning phase. A SoftMax layer in CNN classifier is used to classify the driver as sleep or non-sleep. Proposed system achieved 96.42% accuracy. Proposed system effectively identifies the state of driver and alert with an alarm when the model predicts drowsy output state continuously.

References

REFERENCES [1] Driver Drowsiness Detection System Based On Visual Features: Fouzia, Roopalakshmi R, Jayantkumar A Rathod, Ashwitha S Shetty, Supriya K Alva’s Institute Of Engineering And Technology, Shobhavana Campus, Mijar, Moodbidri {Fouziya23, Drroopalakshmir, Ashwithashetty123, Supriya K189,}@Gmail.Com, Jayantkumar720@Yahoo.Co.In [2] Real-Time Driver-Drowsiness Detection System Using Facial Features Wanghua Deng,Ruoxue Wu 1)Beijing Engineering Research Center For Iot Software And Systems, Beijing University Of Technology, China 2)School Of Software, Yunnan University, China Corresponding Author: Ruoxue Wu (E-Mail: Rochelle.Wu820@Gmail.Com). [3] Driver Drowsiness Detection:International Journal of Engineering Research & Technology (IJERT) ISSN: 2278-0181. Published by, www.ijert.org NCAIT - 2020 Conference Proceedings Volume 8, Issue 15 1) V B Navya Kiran Dept. of Information Science and Engineering ,JSS Academy of Technical Education Bengaluru, India 2) Anisoor Rahman Dept. of Information Science and Engineering, JSS Academy of Technical Education Bengaluru, India 3) Raksha R Dept. of Information Science and Engineering,JSS Academy of Technical Education Bengaluru, India 4) Varsha K N Dept. of Information Science and Engineering, JSS Academy of Technical Education Bengaluru, India 5) Dr. Nagamani N P Assistant Professor Dept. of Information Science and Engineering, JSS Academy of Technical Education Bengaluru, India [4] DRIVER DROWSINESS DETECTION SYSTEM Belal ALSHAQAQI, Abdullah Salem BAQUHAIZEL, Mohamed El Amine OUIS, Meriem BOUMEHED, Abdelaziz OUAMRI ,Mokhtar KECHE Laboratory signals and images (LSI) ,University of Sciences and Technology of Oran Mohamed Boudiaf (USTO-MB) Oran, Algeria {alshaqaqi@ieee.org; abdullah.baquhaizel@gmail.com; ouis57@gmail.com; m_boumehed@yahoo.fr; ouamri@yahoo.com; m_keche@yahoo.com} [5] Deep CNN: A Machine Learning Approach for Driver Drowsiness Detection Based on Eye State :Venkata Rami Reddy Chirra, Srinivasulu Reddy Uyyala, Venkata Krishna Kishore Kolli 1 Department of Computer Applications, National Institute of Technology, Tiruchirappalli 620015, India 2 Machine Learning & Data Analytics Lab, Department of Computer Applications, National Institute of Technology, Tiruchirappalli 620015, India 3 Department of Computer Science & Engineering, VFSTR, Guntur 522213, India Corresponding Author Email: chvrr_cse@vignan.ac.in [6] Smart Driver Assistant :Abha Tewari, Sahil Khan, Aditya Krishnan, Tanmay Rauth, Jyoti Singh Dept. of Computer Engineering, VESIT, Chembur, Mumbai -400 074. Email: skhan28297@gmail.com [7] Real-Time Driver-Drowsiness Detection System Using Facial Features WANGHUA DENG1 AND RUOXUE WU : 1)Beijing Engineering Research Center for IoT Software and Systems, Beijing University of Technology, Beijing 100124, China 2)School of Software, Yunnan University, Kunming 650000, China Corresponding author: Ruoxue Wu (rochelle.wu820@gmail.com) [8] An Investigation of Early Detection of Driver Drowsiness Using Ensemble Machine Learning Based on Hybrid Sensing: Jongseong Gwak , Akinari Hirao and Motoki Shino 1 Institute of Industrial Science, The University of Tokyo, Tokyo 153-8505, Japan 2 Nissan Motor, Co., Ltd., Kanagawa 243-0192, Japan; a-hirao@mail.nissan.co.jp 3 Department of Human and Engineered Environment Studies, Graduate School of Frontier Sciences, The University of Tokyo, Chiba 277-8563, Japan; motoki@k.u-tokyo.ac.jp Correspondence: js-gwak@iis.u-tokyo.ac.jp; Tel.: +81-3-5452-6098 [9] Drowsiness Detection And Parking Assistance For Drivers : Alagu Meyyappan Kailasam, Devi K., Ayikumar Mohan, Benjamin Felix Prabhakar Department Of Computer Science And Engineering, Srm Valliammai Engineering College, Kattankulathur, Tamilnadu, India. [10] Smart driver monitoring system : Shubhi Shaily1 & Srikaran Krishnan1 & Saisriram Natarajan1 &ikumar Periyasamy

Copyright

Copyright © 2022 Shubham Salve, Tejal Jadhav, Siddhi Gajare, Prof. Hani Patil. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.