Damage Detection and Classification of Road Surfaces

Abstract

Road damage refers to a condition where a road is not able to serve the traffic up to an optimal level. For understanding the causes and the process of identifying road damages, a thorough literature survey of various international and national journal, articles have been done and presented in the current study. The same thing has also been done to assess and understand the current deep learning models used in road damage detection. Various common factors are identified that cause road damage. The dataset is extracted from the AWS server which has a separate.tar.gz train and test dataset. The dataset is in .jpg format which consists of images taken from three countries India, Japan, Czech Republic. The dataset of mixed images is used to reduce the biases of the model and to increase its accuracy. The cleaning and analysis of the data are done where different types of damages were classified and the categories having the least images are removed and not considered for further analysis. Then the dimensions of the bounding box of train datasets are identified and the area is calculated to find the damages which are taking more area. Two models are developed using MobilenetSSD and YOLOV5 and both are compared to find the best model.

Introduction

I. INTRODUCTION

Throughout the globe, roads are used by cars, buses, trucks, motorcycles, mopeds, pedestrians, animals, taxis, and other travelers. Travel made possible by motor vehicles supports economic and social development in many countries. Yet each year, vehicles are involved in crashes responsible for millions of deaths and injuries. Each year, 1.35 million people lost their lives on roadways around the world. Every day, almost 3,700 people lost their lives globally in crashes involving cars, buses, motorcycles, bicycles, trucks, or pedestrians. More than half of those killed are pedestrians, motorcyclists, or cyclists.

Roads are intended to make driving easier. However almost every vehicle also put a lot of pressure on the surface  itself. This consistent pressure can cause stresses to emerge in the road surface, which as a result causes cracking. Vehicle crash are estimated to be the eighth leading cause of death globally for all age groups and the leading cause of death for children and young people 5–29 years of age.

The motivation for this project is to produce new and innovative solutions to the problems that have been recognized in the company. Road infrastructure is considered one of the most crucial and valuable public assets because of its contribution to economic and social development.  It connects various businesses and provides access to various public and private sectors. There are three ways by which the conditions of the roads are inspected: manual, semi-automated, and fully automated. Manual and semi-automated methods are traditionally used, in the manual approach the inspection person walks or drives through the pavement to inspect the condition of the road whereas in the semi-automated process road images are collected by a fast-moving vehicle, and then the images are inspected by a person in the office.

II. LITERATURE REVIEW

The literature concerned with road damage caused by heavy commercial vehicles is reviewed. The main types of vehicle-generated road damage are described and the methods that can be used to analyze them are presented. Attention is given to the principal features of the response of road surfaces to vehicle loads and mathematical models that have been developed to predict road response.

According to the authors Yachao Yuan, Yali Yuan, Thar Baker, Lutz Maria Kolbe, Dieter Hogrefe [1], Existing Road damage detection systems mainly process data on clouds, however, they are not able to warn users timely due to the long latency. In this paper, they propose FedRD: a novel privacy-preserving edge-cloud and Federated learning-based framework for intelligent hazardous Road Damage detection and warning.

The authors Taoran Wei, Danhua Cao, Caiyun Zheng, Qun Yang [2] propose a surface defect segmentation method based on defect sample simulation, which only needs a few defects training samples. The simulation method requires one single defect reference sample for training and can generate the same type of defect in the specified area.

According to the authors, Rodrigo Rill-García, Eva Dokladalova, Petr Dokládal [3] Recent Road crack detection methods obtain appealing scores but typically allow a few pixels tolerance margin. This is acceptable for locating cracks, but not for measuring their width. U-VGG19, obtains an F-score of 71.77% on CrackForest, which is superior to other approaches when no tolerance is admitted.

According to the authors Rana Khallaf, Mohamed Khallaf [4], six application-based topics were identified: equipment tracking, crack detection, construction work management, sewer assessment, 3D point cloud enhancement, and miscellaneous topics. Analysis shows that deep learning has been beneficial in leveraging data in areas such as crack detection and segmentation of infrastructure and sewers; equipment and worker detection and analysis and reporting on construction-related operations.

According to the authors, Jie Fang, Bo Qu, and Yuan Yuan [5], Visual-based Road crack detection becomes a hot research topic over the last decade because of its huge application demands. Road crack detection is a special form of salient object detection task, whose objects are small and distributed randomly in the image compared to the traditional ones, which increases the difficulty of detecting.

III. OBJECTIVES

The primary objective of this paper is to create a model which can detect road damage and classify it into different categories. To compare and calculate the efficiency of deep learning  algorithms (Mobile Net SSD and YOLOV5). To identify various factors that cause road damage. To replace traditional methods of identifying damages with efficient and reliable deep learning models. The secondary objectives include Automation of Road Damage Detection using Deep Learning Object Detection Models. To conduct market research to expand Aurigo’s domain. Using state of Art Object Detection Models and Computer Vision Algorithms, process images of damaged roads can be used and detect various forms of Damage like Potholes, Ruts, Bumps, Wheel Marks, Pavement Damage. Damage to Crosswalks, Lines, etc. Once Damage has been Identified using this Data, then inform relevant Stakeholders (Like Transportation Departments / Road Agencies) so that they may plan maintenance/rebuilding activities.

IV. PROBLEM STATEMENT

The problem statement for this project may be stated as Classification and damage detection of road surfaces. Roads are an important asset to any country and like one of the economic backbones of any country. Typical causes for the damages are the materials used during construction, climate conditions, and traffic volume. damages need to be identified and patched as soon as possible. The process starts from the identification of the damage to the patching of the damage. The manual inspection is very time-consuming and can be biased. Therefore, there is a need to come up with a solution that can automate the inspection and classification process.

Manual inspection is also a very expensive method that requires the use of a vehicle. There’s small damage or cracks on the road. People can complain about it in the respective department, and this makes the process longer and more time-consuming. So, it can be thought of as something where the person can take the image and upload it to the server of the responsible department and can take action accordingly and the process can be integrated with the GIS (Geographical Information System) which can also provide the location of the damage on the road.

V. METHODOLOGY

As the first step, there will be a studying of the various Existing methods that are used to detect road damage. Some of which include visual observations by humans and quantitative analysis using expensive machines. Quantitative determination based on large-scale inspection, such as using a mobile measurement system (MMS) (KOKUSAI KOGYO CO., 2016) or laser-scanning method (Yu and Salari, 2011) is widely conducted. Image processing etc.

The next step is to collect data from various sources. The dataset will consist of damage categories, comprising cracks and potholes. The next step is to generate the DL algorithm and model. python and deep learning models will be used. In validating step, help will be taken from the technical team to check for bugs, errors, or any other improvements that can be done. There will also be a demo of the model that will be conducted to validate the model.

The result is then generated by running the model. As a part of the model, object detection is used Object detection is a computer vision technique for locating instances of objects in images or videos.

Object detection algorithms typically leverage machine learning or deep learning to produce meaningful results. Given an image or a video sequence, they localize instances of objects of interest. As in our case, they localize various damages on the road.

General Concepts: Evolved into an effective tool for analyzing big data, deep learning uses complex algorithms and artificial neural networks to train machines / computers, learn from experience, and data / like the human brain. Classify and recognize images. In deep learning, a convolutional neural network (CNN) is a type of artificial neural network commonly used for image / object detection and classification.

The Artificial Neural Network (ANN) is a fully connected multi-layer neural network, as shown in the following figure. These consist of an input layer, some hidden layers, and an output layer. All nodes in one layer are connected to all other nodes in the next layer. By increasing the number of hidden layers, the network becomes deeper.

Python is dynamically typed, and garbage collected. It supports multiple programming paradigms, including structured (especially procedural), object oriented, and functional programming. Due to its extensive standard library, it is often referred to as the "battery-included" language. Programmers often fall in love with Python because of the productivity gains that Python offers.

YOLO is an acronym for You Only Look Once. Version 5 is used. It was introduced by Ultralytics in June 2020 and is the most advanced object identification algorithm currently available. This is a new convolutional neural network (CNN) that detects objects with high accuracy in real time

Exploratory Data Analysis: In statistics, exploratory data analysis is an approach to analyze data sets to summarize their main characteristics, often using statistical graphical approach and other data visualization methods. A statistical model can be used or not, but primarily EDA is for seeing what the data can tell us beyond the formal modeling and thereby contrasts traditional hypothesis testing.

Table 1: Concatenated Data

A. Comparison

The next step of the project is the comparison of the deep learning algorithms namely MobilenetSSD and YOLOV5. MobileNet is a small, low-latency, and low-power convolutional feature extractor that can be built to perform classification, detection, or segmentation like popular large-scale models, such as Inception SSD [20]. It is based on a depth-wise separable convolution. YOLO is an acronym for 'You only look once', is an object detection algorithm that divides images into a grid system. Each cell in the grid is responsible for detecting objects within itself. For the comparison of both the algorithms in this project, their performance is evaluated based on three indices namely, precision, recall, and F1 score. Precision is defined as the percentage of correctly predicted features (true positive) out of the total predictive features (true positive and false positive). Whereas the recall is the percentage of correctly predicted features (true positive) out of the total features present in the class (true positive and false negative). Since both precision and recall counter each other so to balance the F1 score is used.

F1 = 2 × (Precision × Recall) / (Precision + Recall). The goal of the comparison is to identify the algorithm, which is more efficient, fast, reliable, and more accuracy.

B. Deployment

After training and validating the model the test phase comes where the model has been deployed. From the comparison phase, the best model is identified and has been chosen for the deployment using pytorch. After executing the deployment part at the backend an URL for the local system is generated at the end which can be opened in the browser. Then the user can upload the image from their system and the output will be generated with the image containing a bounding box specifying the damage along with the classification specifying the type of damage.

To display the data

df_czech = pd.read_csv('labels_Czech.csv')

df_czech.head()

D20     8381

D00     6592

D40     5627

D44     5057

D10     4446

D50     3581

D43      793

D01      179

D11       45

D0w0       1

Name: class, dtype: int64

There can be seen 10 different classes of damage as mentioned in the beginning. The data is not balanced as class D43 (Cross Walk Blur), D01 (Longitudinal Construction Joint Part Crack), D11 (Lateral Construction Joint Part Crack) and D0w0 are scarce whereas D20 (Alligator Crack) is in abundance. So, entries with D43, D01, D11, and D0w0 can be dropped.

Plotting the area Graph

This plot tells us about the average area covered by the bounding box. D20 (Alligator Crack) has the highest mean area. D50 (Utility Hole) has a smaller average area. This tells us that Alligator cracks are usually spread on more areas of the road followed by D44 (White line blur).

Most of the damage classes have a very small spread or variance. Though the green one, D20 (Alligator crack), has a widespread. Widespread signifies that the damage is propagated more on the road rather than being concentrated at a single small point.

VI. RESULTS

Objectness is essentially a measure of the probability that an object exists in the proposed region of interest if the objectivity is high, it means that the image window is likely to contain an object. As seen in the graph the objectness loss is decreasing with more test runs and loss is less in validation than in the training model, which means the model is underfitting and moves towards optimal. Classification loss gives an idea of how well the algorithm can predict the correct class of a given object.

As seen in the graph the classification loss is decreasing with more test runs and loss is less in validation than in the training model, which means the model is underfitting and moves towards optimal.

Model Loss: as seen in the graph with the increase in several epochs the model loss is decreasing. The optimal number of epochs will be 7 as at that point both the train and test curve are intersecting each other.

GIoU: Generalized intersection over union maximize the overlap area of ground truth and predicts boundary box. It increases the predicted box size by moving slowly towards the target box for non-overlapping cases. As seen in the graph the GIoU loss is decreasing with more test runs and loss is less in validation than in the training model, which means the model is underfitting and moves towards optimal.

From the above model accuracy comparison graph YOLO is better than MobileNet. Yolo is having an accuracy of 95% whereas the Mobile net’s accuracy is 78%.

The results of the deployment platform in terms of Image classification and Identification are

IoT devices such as traffic camera footage, collected images / videos, are key data used to identify the presence of damage, classify damage types, and gain deeper insights into road problems and imperfections. Can be used as. Various technologies used in the project, such as Python, Deep Learning Models, PyTorch Framework, Streamlit, OpenCV, and Jupyter Notebooks, extend the depth and scope of the project and make it easier to understand and apply. You can use the drone to collect video / image data for this project and use it as basic data for various classifications and identifications. It's a fully automated cloud-based software solution that allows you to use drones to upload and process video / image data and identify road damage. The drone's data is geographically tagged to help authorities be informed of the exact location and nature of the damage and to plan and schedule maintenance activities.

Conclusion

The solution is a cost-cutting initiative that government agencies can use to schedule maintenance and other related activities through automated cloud software. This is a great benefit to the customer (transportation/department / institution) as it reduces the manual and time-consuming data collection process. This has a significant impact on the public as it helps governments plan and maintain road infrastructure through concrete data-driven strategies. The developed platform can be a standalone application for users to upload images / videos, a front end that allows users to upload images / videos and return results based on the images and defects present on the road. In the future, the same prototype can be extended to propose a single standardized model that can be applied worldwide, or at least in many countries with the same road conditions. In addition, this work can be used as a baseline and the experiment can be repeated by collecting more images from different countries including India, Japan and Czech in different seasonal conditions, making each damage class more appropriate. It expresses and improves the robustness of the entire image detection system. All damage categories. Use a variety of algorithms to develop models that improve the accuracy of damage detection.

References

[1] Yachao Yuan, Yali Yuan, Thar Baker, Lutz Maria Kolbe, Dieter Hogrefe FedRD: Privacy-preserving adaptive Federated learning framework for intelligent hazardous Road Damage detection and warning Elsevier 2021 [2] Taoran Wei, Danhua Cao, Caiyun Zheng, Qun Yang A simulation-based few samples learning method for surface defect segmentation Elsevier 2020 [3] Rodrigo Rill-García, Eva Dokladalova, Petr Dokládal Pixel-accurate Road crack detection in presence of inaccurate annotations Elsevier 2022 [4] Rana Khallaf, Mohamed Khallaf Classification, and analysis of deep learning applications in construction: A systematic literature review. Elsevier 2021 [5] Je Fang, Bo Qu, Yuan Yuan Distribution equalization learning mechanism for road crack detection Elsevier 2019

Copyright

Copyright © 2022 Jeevitha D, Himanshu Sharma, Vivekanand S Gogi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.