Mobile Phone Detection and Notification for The Prevention of Car Accidents

Abstract

The use of mobile devices can easily divert a driver\'s attention away from the road. Dangerous driving, such as texting and driving, can cause havoc in traffic and jeopardize safety. The goal of this project is to develop an accident-avoidance system that can detect the presence of a mobile phone in the driver\'s hand by installing an in-car camera facing the driver and running a YOLOv3-Tiny algorithm for mobile phone detection. In addition, the model will issue an audio alert to the driver and use face detection algorithms to determine the driver\'s identity. Twilio APIs are being used to send live messages to car owners about the actions taken using the car\'s location information.

Introduction

I. INTRODUCTION

As per the statistics presented by the World Health Organization (WHO), road injuries are one of the top 10 leading causes of death in upper-middle-income countries like India. Almost 80–90% of traffic accidents are caused by the driver's mistakes while operating the vehicle, as well as a lack of attention and distracted behavior [25]. Due to this, countries all around the world are giving it an extremely high priority to improve road safety and develop a better traffic management system. Even though there are multiple potential contributing factors, a lot of experiments indicate that the usage of a cellular phone while driving is likely to cause an accident as it delays the visual information processing by the driver [17].

Distracted driving, as it is well established in the literature, takes drivers' eyes off the road, switches their consciousness from driving to other tasks, and results in false perceptions [25]. Thus, in this project, we have proposed a vision-based solution that will warn the driver with an alerting sound when he or she is found with a mobile device in their hand [16]. We have used the YOLO algorithm to detect the mobile phone [26]. This detection will be achieved by installing a camera inside the car, behind the steering wheel, which will continuously monitor the driver in real time [12]. In case, the mobile phone is detected, an alerting sound will be made by the system and will be checked after some time once again. If the driver is found using the mobile phone again, then the location of the car will be fetched using the geo coordinates of the car[10]. Simultaneously, the image of the driver using the mobile phone will be captured. This image will then be compared with the existing images and registered users in the database. Using facial detection techniques, it will be checked if the driver is known to the system or not. If the image detection matches and the driver turns out to be known to the system, then a WhatsApp message will be sent to the registered car owner using the Twilio API, which will contain the location details as well as the name of the driver. If the person turns out to be unknown, then the owner will just receive the location of the car and a warning message saying that there is a possibility that the car might be stolen.   

II. LITERATURE REVIEW

Over the years, many techniques have been developed for the avoidance or prevention of car crashes through various means. Sensors detect changes in parameters or analyze weaving, swerving, sliding, sudden braking, fast U-turns, sudden acceleration, and other abnormal driving patterns to determine rash driving or accidents. Accelerometer sensors are used for getting the acceleration [1] ( difference in speeds between two consecutive time frames) and a gyroscopic sensor for the angular velocity. An increase or decrease in either of these two parameters triggers a warning, and a car crash emergency is evoked if the car suddenly comes to a halt, or else a warning for rash driving.  A GPS sensor and camera are also used to achieve more functionality and accurate results [2]. GPS sensors are used to send location information in emergency situations to contacts mentioned by the user. Additionally, the GPS sensors offer details about the type of road the car is being driven on. On highways, the vehicle speed increases, and the mobile accelerometer sensor picks up a higher G force than normal, resulting in a false positive. Hence, with the help of GPS and determining the kind of roads, the threshold values for the accelerometer are varied accordingly. The built-in microphone [2] listens for high-decibel acoustic events such as airbag deployment, impact noise, and car horns to detect an accident. Further, A mobile phone can be mounted on the steering wheel [6] to track driving behavior with the use of a magnetometer present inside mobile phones in addition to an accelerometer and gyroscope. With the use of sensors installed around the exterior of the car [26], car crashes were detected, and text alerts were sent to the emergency contacts. In computer vision-based approaches, a camera is placed beneath the rear-view mirror of a car [4] and is used to determine whether or not the driver is using a mobile device. It incorporates a Supervised Descent Method (SDM)-based facial landmark tracking algorithm to track the locations of facial landmarks in order to extract a crop of the region of interest. Object recognition was accomplished using the Histogram of Oriented Gradients (HOG) for feature extraction and various classifiers such as Random Forest Trees and AdaBoost. A cascade face detector is trained using the Adaboost learning algorithm [13]. Hand gesture recognition is accomplished with HOG and SVM, and lip recognition is accomplished with the Red Exclusion method, which recognizes lips because they are redder than skin color and is enhanced with the Fisher transformation. An efficient deep learning model, Mobile Net [23], combined with a single-shot multibox detector (Mobile NET-SSD), was used to identify the driver's mobile usage. If the driver was found using a mobile device, an alarm was triggered, and photos were taken to be sent to traffic authorities. The accelerometer sensor on a smartphone, which measures the acceleration the phone experiences while using particular applications, was used to detect reckless driving [8]. An ANN will later evaluate the gathered acceleration data to determine whether or not the current acceleration matches the data produced when the user was driving while using mobile. One of the CNN architectures, VGG-16, reinforces the learning model [9] [14] for mobile usage detection, yielding an accuracy of 95.54% on the test set.

Table 1.

III. METHODOLOGY

Our paper describes a method for determining whether a driver is using his phone by using an in-car camera. A Raspberry Pi module will be used in parallel with a compatible camera. The module will be installed on the dashboard in a suitable location where the driver can be clearly seen. When a driver makes a phone call or is seen using a mobile phone, his phone is usually held close to his ear or directly in front of his face. Given this, a detection model that looks for mobile phones in the ROI above the steering wheel and is focused on the driver will be run.

A. Object Detection

For real-time object classification, the YOLOv3-Tiny algorithm is used. YOLOv1 solves end-to-end recognition by transforming object detection into a regression problem, predicting bounding boxes, and classifying images directly from the full image. YOLOv2 was proposed as a compromise between accuracy and speed. Not only does YOLOv2 use a novel multi-scale training method, but it can also be run in a variety of sizes. The network structure, however, can be improved further. YOLOv3 enhances structure and accuracy. The prediction of this method is based on the multi-scale feature. YOLOv3 is a cross between a feature pyramid and a single shot multi box detector.  However, the YOLOv3 network structure is still being optimized in YOLOv3-Tiny. Yolo v3-Tiny employs a pooling layer, reduces the figure of the convolution layer, and reduces the depth of the convolutional layer. Convolution layers and max-pooling layers are used to extract features in the feed forward configuration of YOLO v3-Tiny. As a result, the running speed has increased significantly (it is roughly 442% faster than previous YOLO variants).

Tiny YOLOv3 extracts features and strengthens feature fusion by using upsampling in the prediction network. The 13x13 feature map in Figure 1 passes through the convolution and up samplingupsampling layers. This converts the 13x13x512 feature map to 26x26x256. The 26x26 feature map is also concatenated with the up sampling feature from earlier in the network. Finally, the output feature map of size 26x26 is generated [28].

When the driver is found to be using a mobile device, the geopy module in Python is used to get the exact location of the car. This location will also be critical in cases where the driver is not recognized and the vehicle is at risk of being stolen.

C. Face Recognition

After the driver is identified using a mobile device while driving a car, his face is then matched with the registered faces uploaded by the user for verification of the driver. We used DLIB's cutting-edge, deep learning-based facial recognition module for facial recognition. It supports single-shot learning, which means it only needs one image to train itself to detect people. which has proven to be 99.38% accurate on the Labelled Faces[13] in the Wild benchmark. To detect faces, the model employs the HOG (Histogram of Oriented Gradients) feature descriptor in combination with a linear SVM machine learning algorithm. HOG is a simple and effective feature descriptor. In addition to face detection, it is widely used for object detection, such as cars, pets, and fruits. HOG is robust for object detection because object shape is defined by the local intensity gradient distribution and edge direction. Dlib also consists of CNN (Convolutional Neural Network) based facial recognition, which is a class of deep networking. It works really well for non-frontal faces at unusual angles, where a HOG-based detector fails. In this project, we chose HOG and linear SVM-based face recognition over CNN because CNN needs more power and requires a        GPU for faster execution.

D. Messaging Module

The activity of the driver is reported to the car owner via messaging and email services. The Twilio API is used to send live messages to a registered mobile number via WhatsApp chat services. Twilio is a cloud-communications platform that offers a variety of services to help developers integrate voice, message, and video communications into apps, allowing them to achieve call center scale at a fraction of the time and expense. WhatsApp is used as a messaging platform for convenience. The SMT-PLIB module defines an SMTP client session object, which can be used to send email to any Internet machine running an SMTP or ESMTP listener daemon. We were able to notify the vehicle's owner via email that the driver had been using a mobile phone or any other prohibited device by using this module. We developed a specific email ID (acting as a service provider to the owner) to prevent any instances of abuse from using the email address, and this is done by turning on two-factor authentication of the email ID.

IV. RESULTS & DISCUSSIONS

As already discussed in the methodology above, the key factors that are expected out of this model are accurate detection of the mobile phone, identification of the driver, and sending an alert message to the registered owner of the vehicle. The first step is the detection of the mobile phone, which is done using in-car cameras installed behind the dashboard. With the help of localization techniques, the presence of mobile phones is detected in the driver's hand. This is specifically achieved using YOLOv3 tiny detection. If a mobile phone is detected, as illustrated in Fig. 4, then an audio alert is sent in the form of a buzzer sound. The significance of this alert is to send a warning for the driver to keep the mobile phone aside. The system waits for K seconds, and if the phone is detected again, the audio alert is repeated, and the geolocation of the car is fetched using the GeoPy module of Python. Then, with the help of a facial recognition algorithm, the face of the drivers gets identified and is compared with an existing database of registered drivers. If it matches with any of the profiles in the database, an email, a WhatsApp message, and the driver's details, along with the location, are immediately sent to the registered owner of the car. However, if it doesn't match with any image in the database, an email and message, along with the location and a warning saying that the car might be stolen, is sent to the owner.

V. LIMITATIONS AND FUTURE STUDIES

The camera is quite challenging to install inside an automobile. The camera must be quite compact in size. A fully functional camera that is connected to the internet can only be installed in high-functioning automobiles. Making in-car cameras necessary is a really challenging task. If installing cameras is not required, some people can try to avoid following the law. Installing a camera inside a car is an infringement on one's right to privacy, and hackers might misuse it. There is a slight possibility that the camera will misread a particular object. To prevent information misuse, the database must be connected to all significant government databases, even those that are not officially sanctioned by the government. In the upcoming days, our initiative will be very beneficial. The number of accidents caused by prolonged cell phone use while driving has dramatically climbed in recent years. Indeed, this has led to an increase in mortality across the globe. Our idea not only aims to address the issue of accidents, but it can also be used as a security system for vehicles and aid in identifying the driving style of the driver using face recognition software. With the aid of government regulations, we hope to implement this concept in automobiles, trucks, and other large vehicles in the next few years. If the government passes a law on this, it might completely alter the automotive industry.

VI. ACKNOWLEDGEMENT

We would like to convey our sincere gratitude to Ms. Shital Raut, our project guide, for her invaluable assistance with project-related tasks and guidance in writing the research paper. Without her efforts and those of our group, this could possibly be impossible. We appreciate the resources provided by our college for examining the project's implications in the daily lives of common people.

Conclusion

With the objective of preventing accidents, In this paper, we propose a mechanism for detecting cell phone use while driving. We have been able to achieve this with the help of various object detection and facial recognition algorithms in computer vision. Our model is able to correctly detect mobile phones and match the driver\'s face with the ones in the database. It is also sending WhatsApp messages with driver details and geolocation by making use of the Twilio API. It is a full-fledged system that detects mobile phones, alerts the drivers, and sends warning messages to the car owner. To expand the scope and potential of this project, an accelerometer could be added to measure the speed of the vehicle or use GPS location for the same.

References

[1] P. Adarsh, P. Rathi and M. Kumar, \"YOLO v3-Tiny: Object Detection and Recognition using one stage improved model,\" 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), 2020, pp. 687-694, doi: 10.1109/ICACCS48705.2020.9074315. [2] C. Liu, Y. Tao, J. Liang, K. Li and Y. Chen, \"Object Detection Based on YOLO Network,\" 2018 IEEE 4th Information Technology and Mechatronics Engineering Conference (ITOEC), 2018, pp. 799-803, doi: 10.1109/ITOEC.2018.8740604. [3] R. L. Galvez, A. A. Bandala, E. P. Dadios, R. R. P. Vicerra and J. M. Z. Maningo, \"Object Detection Using Convolutional Neural Networks,\" TENCON 2018 - 2018 IEEE Region 10 Conference, 2018, pp. 2023-2027, doi: 10.1109/TENCON.2018.8650517. [4] W. Boyuan and W. Muqing, \"Study on Pedestrian Detection Based on an Improved YOLOv4 Algorithm,\" 2020 IEEE 6th International Conference on Computer and Communications (ICCC), 2020, pp. 1198-1202, doi: 10.1109/ICCC51575.2020.9344983. [5] Lin, TY. et al. (2014). Microsoft COCO: Common Objects in Context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8693. Springer, Cham. https://doi.org/10.1007/978-3-319-10602-1_48 [6] M. B. Abbas and M. Khan, \"Sentiment Analysis for Automated Email Response System,\" 2019 International Conference on Communication Technologies (ComTech), 2019, pp. 65-70, doi: 10.1109/COMTECH.2019.8737827. [7] R. Kumari and C. Fancy, \"Analyzing the PyGameGUI modules available in python,\" 2017 International Conference on IoT and Application (ICIOT), 2017, pp. 1-6, doi: 10.1109/ICIOTA.2017.8073603. [8] M. Khan, S. Chakraborty, R. Astya and S. Khepra, \"Face Detection and Recognition Using OpenCV,\" 2019 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), 2019, pp. 116-119, doi: 10.1109/ICCCIS48478.2019.8974493. [9] L. Tianjiao and B. Hong, \"A optimized YOLO method for object detection,\" 2020 16th International Conference on Computational Intelligence and Security (CIS), 2020, pp. 30-34, doi: 10.1109/CIS52066.2020.00015. [10] Yordanov, B. E. \"Car Drivers\' Monitoring by Means of Mobile Phone Application.\" Bachelor\'s thesis, University of Twente, 2019. [11] Bagade, Abhay P. \"Cell phone usage while driving avoidance with GSM-RF based accident emergency alert system.\" Int. J. Adv. Res. Comput. Commun. Eng 2, no. 5 (2014) [12] Wang, Dan, Mingtao Pei, and Lan Zhu. \"Detecting driver use of mobile phone based on in-car camera.\" In 2014 Tenth International Conference on Computational Intelligence and Security, pp. 148-151. IEEE, 2014. [13] A. P. Jana, A. Biswas and Mohana, \"YOLO based Detection and Classification of Objects in video records,\" 2018 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), 2018, pp. 2448-2452, doi: 10.1109/RTEICT42901.2018.9012375. [14] Ali, Haimd M., and Zainab S. Alwan. Car accident detection and notification system using smartphone. Saarbrucken: LAP LAMBERT Academic Publishing, 2017. [15] Seshadri, Keshav, Felix Juefei-Xu, Dipan K. Pal, Marios Savvides, and Craig P. Thor. \"Driver cell phone usage detection on strategic highway research program (SHRP2) face view videos.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 35-43. 2015. [16] Hou, J. H., Xiaofeng Xie, Qian Cai, Zhengjie Deng, Houqun Yang, Hongnian Huang, Xun Wang, Lei Feng, and Yizhen Wang. \"Early warning system for drivers’ phone usage with deep learning network.\" EURASIP Journal on Wireless Communications and Networking 2022, no. 1 (2022): 1-11 [17] Ishida, Toshiro, and Tsuneo Matsuura. \"The effect of cellular phone use on driving performance.\" IATSS research 25, no. 2 (2001): 6-14. [18] Khurana, Rushil, and Mayank Goel. \"Eyes on the road: Detecting phone usage by drivers using on-device cameras.\" In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-11. 2020. [19] Mendez, Efrain, Alexandro Ortiz, Pedro Ponce, Juan Acosta, and Arturo Molina. \"Mobile phone usage detection by ann trained with a metaheuristic algorithm.\" Sensors 19, no. 14 (2019): 3110 [20] Woo, Yew Meng. \"Vision-based Detection of Mobile Device Use While Driving.\" PhD diss., Curtin University, 2019. [21] Ameya Kokate, Ayush Sinha, Dhesikan Chellappa, Hemant Bhatt, Wasim Sayyed, B.R. Quazi “Automatic Rash Driving Detection and Reporting Using Smartphones”. Student Department of Computer Engineering, AISSMS College of Engineering, Shivajinagar, Pune. [22] Jie Yang , Simon Sidhom , Gayathri Chandrasekaran, Tam Vu , Hongbo Liu, Nicolae Cecan , Yingying Chen , Marco Gruteser , Richard P. Martin “Detecting Driver Phone Use Leveraging Car Speakers”, Stevens Institute of Technology, Hoboken, USA [23] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, \"You Only Look Once: Unified, Real-Time Object Detection,\" 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788, doi: 10.1109/CVPR.2016.91. [24] M. Michalko, J. Onuška, A. Lavrín, D. Cymbalák, O. Kainz, \"Tracking the Object Features in Video Based on OpenCV\", Technical University of Košice/FEEI/Department of Computers and Informatics, Slovakia. [25] Ortega, Carlos A. Catalina, Miguel A. Mariscal, Wafa Boulagouas, Sixto Herrera, Juan M. Espinosa, and Susana García-Herrero. \"Effects of mobile phone use on driving performance: an experimental study of workload and traffic violations.\" International journal of environmental research and public health 18, no. 13 (2021): [26] X. Li and Y. Shi, \"Computer Vision Imaging Based on Artificial Intelligence,\" 2018 International Conference on Virtual Reality and Intelligent Systems (ICVRIS), 2018, pp. 22-25, doi: 10.1109/ICVRIS.2018.00014 [27] Jaswanth Reddy Rekkala, \"Mobile usage detection of driver Using CNN (Convolutional Neural Network)\", CALIFORNIA STATE UNIVERSITY, NORTHRIDGE. [28] Wendong Gai, Yakun Liu, Jing Zhang & Gang Jing (2021) An improved Tiny YOLOv3 for real-time object detection, Systems Science & Control Engineering, 9:1, 314-321, DOI: 10.1080/21642583.2021.190115 [29] Huang, G. (2018). Labeled faces in the wild home. LFW Face Database : Main. Retrieved November 22, 2022, from http://vis-www.cs.umass.edu/lfw/index.html

Copyright

Copyright © 2022 Hrishikesh Zope, Harshita Jain, Hruday Jain, Shital Raut. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.