Pothole Detection System for Automobiles using Machine Learning

Abstract

In recent years, rapid economic growth and technological progress have had a profound impact on traditional transportation systems. One notable concern in terms of traffic safety is the discomfort and risk posed by rough roads, particularly those marred by potholes. This research project seeks to address these issues by not only improving road safety but also enhancing the level of automation in driving. With the proliferation of self-driving vehicles equipped with advanced autopilot systems, there arises a pressing need for real-time sensing of road conditions. To achieve this goal, the project employs machine learning techniques and image processing to detect potholes in real-time and relay this information to other vehicles via an internet connection. The research phase involved a thorough evaluation of various software algorithms and hardware tools, with an emphasis on criteria such as efficiency, cost-effectiveness, accuracy, and practicality for assembly. In addition to these considerations, deep learning and machine learning methodologies were integrated with hardware tools to formulate a refined set of requirements for the pothole detection system.

Introduction

I. INTRODUCTION

Rapid economic growth and technological progress in recent years have significantly affected the quality of the traditional transportation system. The Intelligent Transportation System "ITS" which aims to improve the transportation system is becoming more and more popular. In terms of traffic safety, road users often feel uncomfortable when driving on uneven roads, especially potholes on the road. Potholes are caused by the wear and tear of road surfaces. They cause not only inconvenience to citizens but also death due to traffic accidents. By the way, there is a close relationship between the accident and the condition of the road, including potholes. Traffic accidents occur due to one or more of the following factors: human factors, vehicle factors, road factors, and environmental factors. Vogel and Bester presented risk factors (human, vehicle, and environmental factors) for 14 accident types that can be used as a reference point to determine the likely cause of a particular accident type. The US has more than 2,000 fatal accidents a year due to potholes and poor road conditions. Potholes on the road seriously affect the safety of vehicle drivers. With increased traffic, poor construction and climate change, potholes form on the street in time, leading to accidents and overall driving discomfort [1]. Relay systems for smart cars will help reduce this risk. Driver safety can therefore be improved by implementing a real-time pothole detection system to share pothole information. In addition, there are more and more sensors, including G-sensor, electronic compass, gyroscope, global positioning system "GPS", microphone and camera on mobile devices (such as smartphones and iPads).

II. HISTORICAL BACKGROUND

The research was done from a slightly different angle, where the researchers proposed a cost-effective solution to identify potholes and bumps on roads and alert drivers early to avoid accidents or vehicle damage. Several pothole detection methods have been proposed and can be divided into two groups: image recognition methods and mobile detection methods.

A. Image Recognition Method

Yu and Salari proposed a pothole detection method based on laser imaging to collect road information. An artificial neural network (ANN) algorithm is then used to analyze traffic information and detect potholes. However, this approach, which requires a lot of computing power for laser image recognition, is not suitable for mobile devices.

A low-cost model for the analysis of 3D roadway images has been proposed that uses a low-cost Kinect sensor that enables direct depth measurement, thus reducing computing costs. Lin and Liu proposed a pothole detection method based on advanced science and technology. This method distinguishes potholes from other defects such as cracks. Images are segmented using partial differential equations. In order to detect potholes, the method trains an SVM using a set of pavement images. However, the training model cannot detect road defects if the images are not properly lit.

B. Mobile Sensing Method

For the BusNet project, a G-sensor and GPS are equipped in the on-board unit (OBU) in the bus to collect accelerometer data and position information. This data can be sent to a data processing center via wireless networks, and the data processing center can analyze the data to check whether the accelerometer data vectors exceed pothole detection thresholds. However, this approach requires the accelerometer batch data to be sent when the bus enters the bus station. Therefore, this approach cannot provide real-time pothole detection information.

III. NEED FOR POTHOLE DETECTION SYSTEM

The total number of road accident deaths due to potholes in 2018, 2019 and 2020 were 2,015, 2,140 and 1,471 respectively. Road accidents caused by potholes resulted in the deaths of 5,626 people between 2018 and 2020, according to the latest government figures. According to data from the Ministry of Road Transport and Highways (MORTH), the total number of road accident deaths due to potholes in 2018, 2019 and 2020 were 2,015, 2,140 and 1,471 respectively. Assessing the condition of the road surface is essential for public safety and usability [2].

IV. RELATED WORK

Several pothole detection approaches have been proposed in the last few years. According to Kulkarni et al. (2014), existing pothole detection approaches can be divided into vibration-based methods, 3D reconstruction-based methods, and vision-based methods. The vibration-based approach involves the use of accelerometers, most often from a mobile device. This approach was used by Mednis et al. (2011); the authors compared the performance of multiple thresholding approaches based on z-axis mobile sensing data. The authors report a true positive rate of up to 90% for a small dataset. Also, Wang et al., 2015, Madli et al., 2015 applied a threshold approach.

Fox et al. (2015) used a pothole detection approach based on machine learning. This approach relies on extracted features from crowdsourcing undersampled data from simulated vehicle sensors using CarSim. The authors claimed a simulated accuracy of 99.6% and an empirical experimental accuracy of 88.9% based on the simulated model. The authors' choice to aggregate simulated data from 500 vehicles will be difficult to replicate in the real world. There are additional issues with GPS errors, missing data, different sensor configurations, and difficulties in generalizing the simulation platform, as shown by a drop in accuracy to 88.9% on one stretch of road. A comparison of two machine learning models (SVM and gradient boosting) was done by Bhatt et al. (2017). The authors collected 21,300 accelerometer and gyroscope observations of 96 labeled potholes from a single car using an iPhone 6Ss. They claim that SVM with RBF kernel and gradient boosting achieved the best accuracy of 92.9% and 92.02%, respectively. However, the achieved precision (0.78) and recall (0.42) are much lower.

An image/vision approach involves the use of cameras (images or videos) to collect pothole data. Authors such as Zhang et al., 2014 Li Shuai et al., 2016, Youquan et al., 2011 used various image processing approaches on a small sample of pothole detection data. Other authors, such as Anand et al. (2018), combined textural and spatial features of a camera image to train a deep neural network. The model was evaluated with 969 images. The resulting precision, recall, and F-score are 92.4, 93.8, and 93.0%, respectively. However, this approach is computationally complex, making it less practical for real-time detection [3].

Most of the work in this area uses an image-based approach to detect potholes. They rely on libraries of real potholes; any change in pothole size, road markings, or even the presence of dirt on the road can affect the accuracy of the model without machine learning. This approach requires high computing power due to its computational complexity. Therefore, it is not suitable for real-time pothole detection. The threshold model provides a simplified approach for pothole detection. However, the heuristic process of determining thresholds is tedious and prone to human error. In addition, there is a high probability that you will encounter difficulties in generalizing the model when it is applied to other kinds of road surfaces or cars or sizes of potholes. Most existing pothole detection approaches either rely on specialized and expensive hardware equipment, have lower pothole detection accuracy, or are not robust enough to detect all pothole types. Also, a performance metric that some models rely on is accuracy. This can often be confused with better model performance. Some of the existing work presented higher accuracy with much lower precision and recall scores, highlighting the challenges faced in effective pothole detection.

This study presents a comparative study of machine learning models for pothole detection. Data was collected from multiple Android devices/routes/cars and preprocessed using 2-second intervals of aggregated chunks of data to extract relevant statistical features for training the binary classifier.

V. TECHNICAL COMPONENTS OF THE SYSTEM

An easy way to comply with IJRASET paper formatting requirements is to use this document as a template and simply type your text into it.

A. Darknet Framework

We used YOLOv1, YOLOv2, YOLOv3, YOLOv4 and Tiny-YOLOv4 for training. YOLO is trained using an opensource neural network that is fast thanks to CUDA and C language providing real-time attribute for our detection.

B. PyTorch framework

An opensource deep learning framework bridges the gap between research and practical application. It is used for YOLOv5 training because the darknet framework does not support YOLOv5.

C. TensorFlow Framework

TensorFlow is an opensource deep learning and machine learning framework from Google that provides researchers with a wide range of tools and libraries for various machine learning and deep learning application development and deployment. We used the TensorFlow framework to train SSD-Mobilenetv2.

D. Python with Raspberry pi

The Raspberry Pi Foundation specifically chose Python as the main language because of its power, versatility, and ease of use.

E. GPIO (General Purpose Input/Output):

Raspberry-GPIO-python or RPi. GPIO is a Python module for controlling the GPIO interface on the Raspberry Pi.

F. Dataset Annotation

After collecting the dataset, the next step was to annotate them manually; so we used markup for annotation. It is a free graphic image annotation tool that generates labels in the YOLO darknet format. To train a YOLO model, annotations should be in YOLO format as object class x, y width, height, where object class is an integer value from 0 to the number of defined classes; in our case the object class will be 0 because we have only one class i.e. pothole and the remaining parameters are the coordinates, height and width of the marked bounding box of the object. The YOLOv5 annotation format is a bit different from the YOLO darknet format, so a conversion is needed. Since the YOLOv5 implementation is in PyTorch, its annotation format is class_id, center_x, center_y, width, height, where the class id is normalized to 1 from 0, and the remaining parameters are the same as the YOLO darknet.txt annotation format. Another thing needed to prepare the dataset is the "data.ymal" file, which contains the number of classes, the path to the train and validation folder, and finally the class names. After annotation, the dataset is split into train and test components with a ratio of 80% for training and 20% for testing. Each folder contained images corresponding to its annotation.txt file with the same filenames [4]. The implementation of SSD-Mobilenetv2 is in TensorFlow object detection, so dataset annotations must be converted to TensorflowTF record from Darknet txt for custom object training. The annotation format used for this model is a CSV file that contains the file name, image width and height, class name, and the x-min, y-min, x-max, y-max coordinates of the marked pothole. The data set split ratio remained the same as YOLOv5 and YOLO Darknet. The train and test folder contains images with a CSV annotation file. A label-map.pbtxt file is written for train and test, which contains the class name and id of the labeled objects in each folder.

G. Raspberry Pi

The Raspberry Pi is an inexpensive credit card-sized computer that plugs into a computer monitor or television and uses a standard keyboard and mouse. It's a capable little device that allows people of all ages to explore computing and learn to program in languages like Scratch and Python.

H. Raspberry Pi GPS module

The Raspberry Pi GPS module is built with CP2102 as a USB to UART Bridge chip, which is stable and faster. Also inside the chip is the L80-39 GPS chip. L80-39 has 66 search channels and 22 simultaneous tracking channels, which can help to communicate with the satellite by UART or USB.

VI. IMPLEMENTATION

The proposed system works with the help of a very simple and user-friendly application connected to a live feed camera and helps you achieve real-time pothole detection using the YOLO v4 algorithm. The camera continues to record while the YOLO v4 algorithm works simultaneously on the back. The required information needed for pothole detection is captured. Images are extracted during live recording and the detection process is performed accordingly. The YOLO v4 algorithm takes these frames one by one and processes them. It compares the information recorded from the images with the yol weights. When a pothole is detected, it predicts a bounding box around it. Once an object is detected, the coordinates are retrieved via the GPS module configured with the RaspberryPi. These GPS coordinates are immediately sent to the connected mobile device via Bluetooth. The mobile device stores this data along with timestamp and user details in temporary local storage [5].

As soon as an Internet connection is established on this mobile device, the database is updated. We rely on the Pi camera to collect real-time footage. The Pi Camera module is a camera that can be used to capture high-resolution images and videos. The Raspberry Pi Board has a CSI (Camera Serial Interface) interface to which we can directly connect the PiCamera module. This Pi Camera module can be connected to the Raspberry Pi's CSI port using a 15-pin ribbon cable. Data preprocessing is a key stage of the data mining process that involves manipulating or removing or adding data before it is used to ensure or improve performance. In data mining and machine learning initiatives, the phrase "garbage in, garbage out" is particularly apt. Since our YOLO v4 model is pre-trained, we had to input some images or videos as our data, while now with the real-time detection system, the images are automatically extracted from the live feed through the camera and further processed by the YOLO v4 algorithm.

The YOLO V4 algorithm is used in the proposed system. "You Only Look Once" is short for "You Only Look Once". It is a state-of-the-art real-time object identification system developed by Joseph Redmon that can distinguish multiple items in a single image. YOLO uses a completely different detection technique than earlier technologies. It uses a single neural network to process the entire image. The image is partitioned into regions by this network, which predicts bounding boxes and probabilities for each region. Projected probabilities are used to consider these bounding boxes.

YOLOv4 is good with its AP and FPS improvements. On a single CPU, YOLOv4 optimizes real-time object detection and training. On the COCO dataset, YOLOv4 achieved state-of-the-art performance of 43.5 percent (AP) at 65 frames per second (FPS). YOLO divides the input image into an S x S grid, with each grid cell anticipating the object centered in that cell. Coordinate data is recorded using the GPS module.

A successful detection signals the GPS module to store the retrieved coordinates. These coordinates are obtained by sending signals to nearby radio towers and therefore do not require an internet connection. This is a crucial feature of the GPS module. These coordinates can be sent to a mobile device via a Bluetooth connection. Most raspberryPi modules have built-in Bluetooth technology, but raspberryPi modules prior to 2016 lack such a built-in feature. They may require an additional Bluetooth module [6].

VII. COMPARATIVE ANALYSIS

A comparative analysis of two Algorithms was done. CNN and YOLOv4 were tested and the table 1 below, explains the results.

TABLE I
Comparative analysis of Different Algorithms

VIII. ADVANTAGES

A. Storing Location of all the Potholes

Our system helps in detection and localization of roadway potholes by collecting data about their coordinates using accelerometer and GPS embedded in the car itself. This data can be shared with government authorities to help them have a clear picture of pothole locations and fix them quickly and efficiently. As most government authorities ignore potholes by saying that they will repair them after the rainy season stops, our system can give them an idea of where the majority of potholes are present. Thus, before the rainy season begins, they can be prepared to fix places which are more prone to bad weather rather than doing shallow repair work on all roads which eventually get damaged.

B. Provide Customized Car Designs

The data that we collect can be used by car manufacturers to customize cars based on the location of sale to provide better services to customers. Car companies undergo a heavy study on the geography of the location before building the design and dynamics of the car where all safety matters are considered.

At this phase, they require a ton of data to draw conclusions from so this is where we come in and provide them with all coordinates where potholes are present thus giving them an upper edge over their competitors. They can build better cars based on geography and pothole configurations rather than building cars which are not efficient.

For example, cars like Wagner are not at all suitable for off-road places and deep pothole locations so manufacturers can avoid making cars like this for areas which have dirt roads and rather make a car like Tata Punch which has a good wheelbase and ground clearance.

C. Avoiding Tragic Accidents

This system helps us to avoid dreadful potholes and hence prevent tragic accidents due to bad road conditions. Cameras installed on moving vehicles can detect potholes in real-time and help drivers avoid them. Tire blowouts and wheel damage are common when a tire plunges into a deep pothole, causing it to split due to sharp edges of the hole. This can cause your vehicle to lose control and collide with ongoing traffic. The force of hitting a pothole can cause misalignment in the steering components, leading to potential control problems and increased accident risks.

IX. LIMITATIONS

A. Lower Range Scanning by Night Vision Cameras

Cameras have two modes of working: night vision mode and daylight mode. Cameras have a long-range capturing power in daylight mode, but it decreases when operated in night vision mode. Thus, the pothole detection range decreases at night due to the low range provided by the camera.

B. Predefined Functionality

Although self-driving vehicles use several sophisticated forms of technology such as radars, lasers, and high-definition cameras, most budget-friendly cars do not provide them. Providing equipment such as Raspberry Pi is not cost-efficient and durable for the long run. There are many self-driving cars that provide inbuilt data processing hardware which can be used by our systems to process data and furnish efficient results for evaluating potholes on the road.

C. Unpredictable Weather

The unknowing weather changes cause bad predictions, and our systems operate using cameras which are rendered virtually useless in blizzards, blinding rain, fog, or other conditions where visibility is zero. In fact, this has proven to be such a challenge for technology companies working to develop self-driving cars that Google and several other companies have reported to regulatory authorities in several states that human drivers were forced to take over control of their prototype self-driving vehicles during these exact types of weather conditions.

D. Camera Limitations

A common issue that one may face with a camera at nighttime is the screen turning white or certain white areas on the screen. The reason for this is infra-red light reflecting back into the camera lens. This especially remains the cause if the angle of view is very wide. During night, we often experience the problem of general image deterioration. What happens is that when infra-red switches on camera, it draws more amount of current [7]. But if the power is not powerful enough then this extra current drawn could lead to some problems. There can appear moving spots in front of the camera which may seem like spider webs. This phenomenon is more noticeable during fog type weather conditions where water droplets suspended in the air reflect the infra-red. The warm currents present in the air lead particles to rise and fall, and these appear as moving spots in front of the camera.

X. Applications

Poor nature and design and development of roads along with natural calamities like rainfall cause undesirable changes in road structure resulting in potholes. Additionally, the lack of a proper road maintenance system results in a lot of potholes.Both underdeveloped and developed countries could use the software if it were implemented on a large scale and easily available. It was possible to analyze pothole data such as the location of the pothole and the number of vehicles on the road indicating the road's busyness. Smart cars use this technology to automatically slow down and alert passengers.

The system is also used to provide an overall comfortable experience to users. These systems can also be used for the ongoing inspection of roads by the government. Cameras mounted on road signals can analyze road conditions and send an automatic warning message to the relevant road authorities.The same algorithms and technologies used in software are used by food delivery robots to avoid obstacles and potholes that stand in the way. Pedestrians and everyday users.

Conclusion

Aging roads and poor road maintenance systems result in a large number of potholes, the number of which increases over time. Potholes threaten the safety of road traffic and the efficiency of transport. In addition, they are often a contributing factor in traffic accidents. To solve problems associated with potholes, the location and size of the potholes must be determined quickly. Sophisticated road maintenance strategies can be developed using a pothole database, which requires a specific pothole detection system that can collect pothole information at low cost and over a large area. However, repairing potholes has long relied on manual detection. Current automatic detection systems such as vibration-based or laser-scanning systems are insufficient for correct and inexpensive pothole detection due to the unstable detection of vibration-based methods and the high cost of laser-scanning-based methods. The proposed system detects potholes over a wide area and at a low cost. We have developed a new pothole detection algorithm specifically designed to work with embedded computing environments using cameras.

References

[1] C. Ng, T. Law, F. Jakarni, and S. Kulanthayan, \"Road infrastructure development and economic growth,\" in IOP Conference Series: Materials Science and Engineering, vol. 512, no. 1, Article ID 012045, 2019. [Online]. Available: https://iopscience.iop.org/article/10.1088/1757-899X/512/1/012045/meta [2] S. Rahman and A. Patel, \"Pothole Image Dataset,\" Kaggle, 2020. [Online]. Available: https://www.kaggle.com/sachinpatel21/pothole-image-dataset/. [3] Z. Hou, K. C. Wang, and W. Gong, \"Experimentation of 3D pavement imaging through stereovision,\" in Proceedings of the International Conference on Transportation Engineering 2007, pp. 376–381, Chengdu, China, July 2007. [Online]. Available: https://ascelibrary.org/doi/abs/10.1061/40932(246)62 [4] J. Eriksson, L. Girod, B. Hull, R. Newton, S. Madden, and H. Balakrishnan, \"The pothole patrol: using a mobile sensor network for road surface monitoring,\" in Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services, pp. 29–39, Breckenridge, CO, USA, June 2008. [Online]. Available: https://dl.acm.org/doi/abs/10.1145/1378600.1378605 [5] G. Jog, C. Koch, M. Golparvar-Fard, and I. Brilakis, \"Pothole properties measurement through visual 2D recognition and 3D reconstruction,\" in Proceedings of the Computing in Civil Engineering (2012), pp. 553–560, Clearwater Beach, FL, USA, June 2012. [Online]. Available: https://scholar.google.com/scholar_lookup?title=Pothole%20properties%20measurement%20through%20visual%202D%20recognition%20and%203D%20reconstruction&author=G.%20Jog&author=C.%20Koch&author=M.%20Golparvar-Fard&author=&author=I.%20Brilakis [6] S. Lee, S. Kim, K. E. An, S.-K. Ryu, and D. Seo, \"Image processing-based pothole detecting system for driving environment,\" in Proceedings of the 2018 IEEE International Conference on Consumer Electronics (ICCE), pp. 1-2, IEEE, Las Vegas, NV, USA, January 2018. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/8326141/ [7] O. A. Egaji, G. Evans, M. G. Griffiths, and G. Islas, \"Real-time machine learning-based approach for pothole detection,\" Expert Systems with Applications, vol. 184, p. 115562, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417421009684

Copyright

Copyright © 2023 Priyansh Salian, Krish Naik. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.