Speed Breaker and Potholes Detection Using DL and IoT

Abstract

This project introduces a novel approach to enhance road safety by implementing a real-time pothole detection system. The system utilizes a camera module to capture video input, which is then processed using a Python file for Image detection. Upon detecting a pothole or a speed breaker, the system triggers a buzzer to alert nearby vehicles and pedestrians. This alert mechanism aims to reduce the risk of accidents and vehicle damage caused by potholes, thereby improving overall road safety. Additionally, the project can contribute to the efficient maintenance of roads by enabling timely repair of identified potholes. The system\'s effectiveness in alerting road users and facilitating prompt repairs underscores its potential to enhance the quality and safety of transportation infrastructure. Also a concept of RQI (Road Quality Index) will improve the transport and travelling over the road. And to make a database which contain coordinates where speed breakers and potholes are present and this database will be real time.

Introduction

I. INTRODUCTION

Road safety is an issue of great concern not only nationally or regionally, but also across the globe. Poor road conditions rank as the top cause of incidents and vehicle damage. Potholes are particularly dangerous to vehicle traffic and pedestrians alike, causing immense traffic congestion, increased cost of repair, and even loss of lives. Traditional means of maintaining the road rely heavily on physical inspection and reporting, which is time-consuming and inefficient. There is scope for a very high demand for automated real-time systems that can detect road anomalies like potholes and speed breakers to enhance road safety, followed by a due and timely maintenance.

The objective is to make a database that contains the coordinates in which the speed breakers and potholes are present, and this database would be real-time.

A. Project Overview

This paper introduces a new real-time pothole detection mechanism to enhance road safety and reduce the efficiency of road maintenance. In this system, the installed camera module keeps taking the input blocks of videos from the surface of the road. Advanced techniques in Image detection in Python are used to process this video feed and identify or classify both potholes and speed breakers.

1. Camera Module: The camera module is the key component for data collection. It captures the video of the road surface at high resolution. Since video input is continuous, this facilitates the monitoring of the roads for the complete duration.
2. Image Detection using Python: The core of this system is an Image detection algorithm implemented using Python, which processes the video input. The potholes and speed breakers are detected and segmented from the video feed using state-of-the-art computer vision techniques.
3. Alert Mechanism: The system triggers a buzzer to alert vehicles and pedestrians nearby in case of any pothole or speed breaker. This immediate audio warning helps avoid accidents and vehicle damage by alerting drivers to take necessary driving precautions.
4. RQI- Road Quality Index: It's the additional application of the system as the output of the RQI will produce an index worthiness, indicating road conditions. This index will assist the authorities in decision-making on the prioritization of maintenance work and efficient way of using resources.

B. Primary Objectives of the Project

Creating a real time road anomalies detection system which contains a device to detect road anomalies, a database where the coordinates of the anomalies are stored, a machine learning model to detect anomalies through video feed. And quantifying quality of a road in numbers so that one can know how damaged the road or how good the road is just by this number.

This real-time detection system for potholes can be applied to drastically improve the quality and safety of the roads. Proactive deadling with the problem of road anomalies, the system ensures less risk related to bad road conditions and rights the efficient service and management of the road infrastructure, thereby providing all road users with a considerably smoother and safer ride.

II. LITERATURE REVIEW

A. Literature Review

This literature survey focuses on the technological advancements in road safety through the integration of Lidar and radar sensors. The prototype described aims to enhance real-time detection of speed breakers and potholes, providing early warnings to drivers. By incorporating a two-step confirmation mechanism, where GPS location is logged only upon consecutive detections, the system aims for accuracy and reduces false positives. The continuous updating of a centralized database contributes to improved signal delivery times, thereby enhancing overall road safety. Moreover, the device utilizes proximity data to analyze traffic density, offering valuable insights for efficient traffic management. This comprehensive approach not only addresses immediate safety concerns but also contributes to the broader goal of optimizing transportation systems.

1. Proposing an innovative early warning system using a smartphone app to alert drivers in advance of approaching speed-breakers, crucial for safety, especially in low visibility conditions. The system, based on accelerometer data, detects speed-breakers without requiring costly computations for reorientation, a significant improvement. Evaluated with 678 km of drive data involving diverse drivers, vehicle types, and smartphones, the results are promising. Further enhancements are anticipated through aggregating detection reports from multiple smartphones.
2. This study introduces Pothole Patrol (P2), a mobile sensing system leveraging sensor-equipped vehicles to monitor road surface conditions. Deployed on 7 taxis in Boston, P2 utilizes GPS and vibration sensors, employing a machine-learning approach to identify potholes and anomalies with high accuracy (misidentification rate < 0.2%). The system, evaluated on extensive taxi drive data, successfully detects real potholes, with over 90% confirmed as road anomalies requiring repair after clustering to reduce spurious detections.
3. Highlighting the societal importance of road infrastructure, akin to blood vessels for humans, necessitating continuous monitoring and timely repairs. The paper introduces a mobile sensing system for road irregularity detection via Android OS smartphones, emphasizing participatory sensing for comprehensive real-time data. Discussed algorithms achieve a high true positive rate of 90% with real-world data, determining optimal parameters and offering practical recommendations for application.
4. Addressing the global challenge of maintaining road quality, especially critical for motorcycle safety, this research employs mobile phones with tri-axial accelerometers to collect and analyze acceleration data. With data spanning 60 kilometers, supervised and unsupervised machine learning methods, including SVM, achieve 78.5% precision in detecting road anomalies. Unsupervised learning constructs a model for smooth roads, ranking road segments based on learned thresholds. Comparison with human evaluators shows excellent performance in road quality ranking, emphasizing the effectiveness of mobile sensing for road assessment.

In conclusion studies present innovative approaches to road infrastructure maintenance. The Pothole Patrol (P2) system, implemented on Boston taxis, combines mobile sensing and machine learning for accurate road anomaly detection. Another study emphasizes the societal importance of roads, utilizing Android smartphones to achieve a 90% true positive rate in detecting irregularities. In Taiwan, motorcycle-based acceleration data, coupled with machine learning, addresses poor road surfaces, achieving a 78.5% precision in ranking road quality. Lastly, a participatory sensing study using Android smartphones underscores the importance of continuous road monitoring, achieving a 90% true positive rate for real-time data collection and optimal repairs.

III. METHODOLOGY

A. Requirements

1) Software Requirements

1. CNN architecture for Image detection
2. Python for taking the image/video
3. Firebase for database of data regarding tilt sensor
4. Firebase for storing the coordinates of the road anomalies
5. Arduino ide for tilt sensor working and pushing   the coordinates of road to the database
6. HTML, CSS, Javascript for interface
7. Mapbox API for map integration with interface
8. Google Colab for training the model
9. Google API for fetching coordinates of that location

2) Hardware Requirements

1. Node MCU
2. Tilt Sensor
3. Buzzer
4. ESP32 camera module

B. System Workflow and Data Processing

In addition to the components previously outlined, this project incorporates multiple data sources and processing steps to ensure accurate and reliable road anomalies detection. The system uses a combination of video input from a camera module and vibration data from a tilt sensor to enhance the detection accuracy. The results from both data sources are then combined to produce a final detection outcome. Here’s a detailed explanation of each step:

1) Video Input and Image Detection (Result 1)

Video Capture: The camera module continuously captures video footage of the road surface.

Image Detection Model: The captured video is fed into a pre-trained Image detection model. This model, trained using a platform like Google Colab, identifies and segments road anomalies such as potholes and speed breakers. The Image detection process involves detecting objects within each frame of the video and classifying them accordingly.

Output Generation: The model generates a segmented output, highlighting the detected anomalies. This output forms the first result (Result 1) of the system.

2) Vibration Data from Tilt Sensor (Result 2)

Tilt Sensor Integration: A tilt sensor mounted on the vehicle detects vibrations caused by road anomalies. This sensor measures the degree of tilt or vibration experienced, providing a quantitative measure of road roughness.

Data Storage and Retrieval: The vibration data is continuously recorded and sent to a cloud database, such as Firebase.

The data stored in Firebase includes timestamped readings of the tilt sensor, indicating the presence and severity of road anomalies.

Python Integration: A Python script retrieves the stored vibration data from Firebase. This script processes the retrieved data to identify significant vibration events that correlate with potential road anomalies. The processed vibration data forms the second result (Result 2) of the system.

3) Data Fusion and Final Detection (Final Result)

Data Synchronization: The system synchronizes the video-based Image detection results (Result 1) with the vibration data from the tilt sensor (Result 2). This synchronization ensures that both data sources are analyzed within the same timeframe.

Logical Operation and Analysis: The system performs a logical AND operation between Result 1 and Result 2. This operation combines the video-based detections with the vibration-based detections, enhancing the overall accuracy of the system. Only those road anomalies detected by both the Image detection model and the tilt sensor are considered in the final result. This reduces false positives and improves reliability.

Final Output: The final result is generated, highlighting the confirmed road anomalies that have been validated by both the video and vibration data. This final output triggers the alert mechanism (buzzer) to warn nearby vehicles and pedestrians about the detected anomalies.

4) Enhancing Accuracy and Reliability

By combining video and vibration data, the system achieves a higher level of accuracy in detecting road anomalies. This multi-sensor approach ensures that detections are validated through multiple sources, reducing the likelihood of false positives and negatives. The integration of these two data streams allows for a more comprehensive understanding of road conditions, improving the system’s effectiveness in real-world scenarios.

a) How is result 1 generated

Video Capture: The ESP32 Camera module takes the video feed and transfer this video feed through it’s Wifi capability to our laptop and then the image on which the module is sending it’s image is fetched using Python code and sent for Image detection.

Image Detection Model : We have used YOLOv8 for we have took the `yolov8s-seg.pt` and finetuned it for our problem.

Dataset: We took the dataset of potholes and speed breaker from Roboflow Universe. Each image in the dataset are segmented and we imported the dataset in format suitable to YOLOv8 using Roboflow API.

Training: The training was done on Google Colab using GPU available on colab and the weights of the model was saved locally.

Detection : The weights of the model was imported on local python file and used for detection.

If the image passed to the model has been segmented and if it shows in the output then this is result 1 and this will get uploaded to firebase along with the coordinates of that location using Google API for coordinates and it will show 1 in Result 1.

b) How is result 2 generated

Tilt Sensor : The Tilt sensor is caliberated so that it will turn the buzzer on for 1 sec once it gets a shock. This shock will be when we hit speed breaker or potholes. The code is uploaded on Node MCU using Arduino IDE and the Node MCU and our Laptop is connected on same Wifi it works like : Once the Tilt sensor hits any road anomaly then the coordinates of that location will be uploaded to firebase using libraries in Arduino IDE which supports integration of firebase and Arduino.

If the Tilt sensor hits a road anomaly it then the coordinates of that location will be fetched using Google API and uploaded to the database and it will also show 1 in Result 2.

C. Final Result

There will be AND operation between Result 1 and Result 2 and then the final result will be generated. If the final result is 1 then there is a road anomaly and if it’s 0 then it’s not a road anomaly.

Fig.1. System Design

Integrating this database with the interface

Using Mapbox API we can mark the coordinates where on the Map. This data from the firebase is fetched and on these coordinates, there will be a image displayed

With this integration on a large scale one can get information on how many road anomalies are present between Starting location and Destination.

And on basis of this RQI (Road Quality Index) will be made.

????????????=∑(???????????? )/6

1. ???????????? -> Harmfulness factor (It will be rated from 0-5) on basis of the intensity of that factor on that location

If the intensity of that factor is more then it will be rated high near 5. Otherwise low

Number of Factors

Factors represents physical conditions such as:

• Rain
• Temperature
• Snow
• Storm
• Pothole rate (number of potholes per 1 km)
• Speed breaker rate (number of speed breakers per 1 km)

The Harmfulness factor are assigned to the depending on how much intensity that factor has at that location.

If the RQI (Road Quality Index) increases, the quality of the road decreases. This is because the RQI is calculated based on the Harmfulness Factors (HF_i). Higher values of HF_i indicate more severe and harmful conditions, leading to a higher overall RQI. Therefore, a higher RQI reflects poorer road quality with more significant issues such as potholes, speed breakers, adverse weather conditions, etc.

IV. RESULTS AND DISCUSSIONS

The project successfully implemented a real-time pothole detection system using machine learning and IoT.

A. Real-time Pothole and Speed Breaker Detection

The model was successfully able to develop a real-time detection system for road anomalies by the fusion of machine learning, computer vision, and IoT. This became feasible only because it had an ESP32 camera module and a tilt sensor that provide robust data for anomaly detection.

1) Video-based detection (Result 1)

 The YOLOv8 Image detection model gave the best results in detecting potholes and speed breakers because it was trained on a custom dataset containing images of such road anomalies.

The video feed from the ESP32 camera module was processed in real-time, and the segmented output reliably told about the presence of potholes and speed breakers. This video-based detection mechanism provided a visual confirmation of anomalies, which were then uploaded to Firebase along with their coordinates.

Fig.2. Output of Detection model

Fig.3. Speed breaker detected

2) Vibration-based Detection Result 2

Vibration due to road anomalies was detected with the help of a tilt sensor mounted on the vehicle. This data is continuously recorded and processed, and all events of large vibration magnitude were logged into the cloud database.

The ability of the tilt sensor in capturing the intensity of vibrations correlated well with the presence of potholes and speed breakers, thus providing an additional layer of verification.

Fig. 4. Tilt sensor Hardware

3) Data Fusion

This leads to an enhanced accuracy of the system through the results from video-based detection combined with those from vibration data. The logical AND operation ensured that only anomalies detected by both methods were taken into consideration for the final result, hence reducing false positives.

Fig.5. Real time database for storing coordinates

B. Alert Mechanism

The alert mechanism was designed to be adequate by dispensing real-time warnings to drivers and pedestrians. The buzzer is an instantaneous audio alert triggered by the confirmed detections, which reduces the risk of accidents and vehicle damage.

C. Road Quality Index (RQI)

The RQI added value to the project at an important stage by providing a numerical measure of road conditions. The RQI denoted the quality of roads, and it would be an effective measure in guiding such decisions on the maintenance of roads and resource allocation.

Sample RQI Calculation:

Assuming the following Harmfulness Factors (HF_i) for each condition:

Rain (HF_rain): 3

Temperature (HF_temp): 2

Snow (HF_snow): 4

Storm (HF_storm): 3

Pothole rate (HF_pothole): 5

Speed breaker rate (HF_speedbreaker): 2

The RQI calculation is as follows:

RQI = ((3)+(2)+(4)+(3)+(5)+(2))/(6)?

RQI = (19)/(6)?

RQI = 3.16666

?RQI≈3.16

This example calculation demonstrates the effectiveness of the RQI in providing a comprehensive overview of road conditions.

D. Database and Interface Integration

This has made the visualization and monitoring of road anomalies efficient by integrating a real-time database with an interactive map interface powered by the Mapbox API, making it user-friendly and giving a view for monitoring the road conditions available for use not only by the authorities but also by the general public.

Fig.6. The red locations show either there is pothole or speed breaker

Conclusion

The project \"Speed Breaker and Potholes Detection Using ML and IoT\" has successfully implemented a real-time pothole detection system that utilizes machine learning and IoT technologies. This was coupled with the successful development of a system for real-time detection and monitoring of road anomalies, which improved road safety and the efficiency of road maintenance. These have been the major contributions of the project: Real-time Detection: The integration of camera and tilt sensor data provided a robust mechanism for detecting potholes and speed breakers with high accuracy and reliability. Alert System: In this regard, immediate audio alerts have been useful in controlling risks related to road anomalies for the protection of drivers and pedestrians. RQI thus provided a quantitative measure for road conditions, which allowed the prioritization of maintenance and efficient use of resources. Database and Interface: This is the real-time database and the interactive map interface that have supplied convenience of monitoring, management of road conditions easily, raising the user experience in general. It showed that machine learning, computer vision, and IoT can be combined with a lot of potential for road safety and infrastructure management. Further work could include checking scalability of the system, integration with other smart city applications, and further improvement in detection algorithms for handling diversified environmental scenarios. No funding was raised for this research project.

References

[1] S. K. R., P. V. P. P., and A. K. V., \"Study on Road Condition Monitoring using IoT Sensors,\" International Journal of Engineering Research & Technology (IJERT), vol. 8, no. 10, pp. 123-130, Oct. 2019. [2] M. S. S., P. R., and S. D., \"Pothole Detection Systems Using Accelerometers and GPS,\" IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 6, pp. 1795-1805, June 2016. [3] H. W., and L. Z., \"Image Processing Techniques for Road Anomaly Detection,\" IEEE Access, vol. 7, pp. 165-175, Jan. 2019. [4] \"Firebase Documentation,\" [Online]. Available: https://firebase.google.com/docs. [Accessed: May 27, 2024]. [5] \"ESP32 Camera Module Datasheet,\" [Online]. Available: https://www.espressif.com/en/products/hardware/esp32/overview. [Accessed: May 27, 2024]. [6] \"Tilt Sensor Specifications,\" [Online]. Available: https://www.sparkfun.com/datasheets/Sensors/Accelerometer/Tilt_Sensor_Datasheet.pdf. [Accessed: May 27, 2024]. [7] \"NodeMCU Documentation,\" [Online]. Available: https://nodemcu.readthedocs.io/en/release/. [Accessed: May 27, 2024]. [8] \"Arduino IDE Reference,\" [Online]. Available: https://www.arduino.cc/reference/en/. [Accessed: May 27, 2024]. [9] \"GPS Module Datasheet,\" [Online]. Available: https://www.u-blox.com/en/product/neo-6-series. [Accessed: May 27, 2024]. [10] D. Mohan, “Road safety in less-motorized environments: Future con cerns,” International Journal of Epidemiology, vol. 31 (3), pp. 527–532, 2002. [11] D. M. Chan, “Who global status report on road safety 2013,” World Health Organization, Tech. Rep., 2013. [12] D. A. Aderamo, “Spatial pattern of road traffic accident casualties in nigeria,” Mediterranean Journal of Social Science, vol. 3 (2), May 2012. [13] D. Bowrey, R. Thomas, R. Evans, and P. Richmond, “Road humps: accident prevention or hazard?” Journal of accident & emergency medicine, vol. 13, no. 4, p. 288289, July 1996. [Online]. Available: http://europepmc.org/articles/PMC1342738 [14] “Road able: accident report.” [Online]. Avail http://timesofindia.indiatimes.com/india/ Over-11000-people-killed-by-potholes-speed-breakers-last-year/ articleshow/48950267.cms [15] R. Bhoraskar, N. Vankadhara, B. Raman, and P. Kulkarni, “Wolverine: Traffic and road condition estimation using smartphone sensors,” in COMSNETS’12, 2012, pp. 1–6. [16] M. Jain, A. P. Singh, S. Bali, and S. Kaul, “Speed-Breaker Early Warning System,” in Proceedings of the 6th USENIX/ACM Workshop on Networked Systems for Developing Regions. New York, NY, USA: ACM, 2012. [17] P. Mohan, V. N. Padmanabhan, and R. Ramjee, “Nericell: Rich mon itoring of road and traffic conditions using mobile smartphones,” in Proceedings of the 6th ACM Conference on Embedded Network Sensor Systems, ser. SenSys ’08. New York, NY, USA: ACM, 2008, pp. 323 336. [18] J. Eriksson, L. Girod, B. Hull, R. Newton, S. Madden, and H. Balakr ishnan, “The pothole patrol: Using a mobile sensor network for road surface monitoring,” in Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services, ser. MobiSys ’08. New York, NY, USA: ACM, 2008, pp. 29–39. [19] D. Kim and Y. I. Lee, “Characteristics of driving behavior according to accumulative task loading time of intra-city bus driver,” in Proceedings of the Eastern Asia Society for Transportation Studies, vol. 9, 2013. [20] R. C. Nitin and K. Mangesh, “Landmark based shortest path detection by using a* and haversine formula,” International Journal of Innovative Research in Computer and Communication Engineering, April 2013. [21] “Garmin glo bluetooth gps.” [Online]. Available: https://buy.garmin. com/en-US/US/oem/sensors-and-boards/glo-/prod109827.html [22] A. Mednis, G. Strazdins, R. Zviedris, G. Kanonirs and L. Selavo, \"Real time pothole detection using Android smartphones with accelerometers,\" 2011 International Conference on Distributed Computing in Sensor Systems and Workshops (DCOSS), Barcelona, 2011, pp. 1-6. [23] Rajeshwari S., Santhosh Hebbar, Varaprasad G., “Implementing Intelligent Traffic Control System for Congestion Control, Ambulance Clearance and Stolen Vehicle Detection”, IEEE Sensors Journal, Vol.15, No.2, pp.1109-1113, 2015. [24] I. S. Jacobs and C. P. Bean, “Fine particles, thin films and exchange anisotropy,” in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271–350. [25] Sudarshan S. Rode, Shonil Vijay, Prakhar Goyal, Purushottam Kulkarni, Kavi Arya, “Pothole Detection and Warning System”, In Proceedings of International Conference on Electronic Computer Technology, pp.286- 290, 2009. [26] Faith Orhan, P. Erhan Eren, “Road Hazard Detection and Sharing with Multimodal Sensor Analysis on Smartphones”, In Proceedings ofInternational Conference on Next Generation Mobile Apps, Services and Technologies, pp. 56-61, 2013. [27] Moazzam, K. Kamal, S. Mathavan, S. Usman, M. Rahman, “Metrology and Visualization of Potholes using the Microsoft Kinect Sensor”, In Proceedings of IEEE Conference on Intelligent Transport System, pp.1284-1291, 2013. [28] He Youquan, Wang Jian, Qiu Hanxing, Zhang Wei, Xie Jianfang, “A Research of Pavement Potholes Detection Based on Three-Dimensional Project Transformation”, In Proceedings of International Congress on Image and Signal Processing, pp.1805-1808, 2011.

Copyright

Copyright © 2024 Prasad Khambadkar, Aditya Kharat, Aashish Kapase , Prathamesh Karande , Swarup Kekan , Ganesh Dongre. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.