Social Distancing Detection Using Computer Vision

Abstract

SARS-CoV-19 is one of the deadliest pandemics the world has witnessed, taking around 6 crore lives till now across worldwide and about 6 lakhs in India. To limit its spread numerous countries have issued many safety measures. Due to the absence of the vaccine against the covid-19 the world has suffered a lot. Though scientists have developed several vaccines then also the pandemic is still out of control so therefore the only feasible option available to us is social distancing. We are implementing a Deep-learning based solution proposed for the above-stated problem. The distance between people can be estimated and the pair of people in the display will be indicated with red or green bounding boxes over it. The video frame from the camera was used as input, and the open-source object detection pre-trained model based on the YOLOv4 / YOLOv5 algorithm was employed for pedestrian detection. Later, the video frame was transformed into a top-down view for distance measurement from the 2D plane. The connection of CCTV cameras in public areas, public transportation, and hospitals is useful for gathering information. The proposed method was validated on a pre-recorded video of pedestrians walking on the street.

Introduction

I. INTRODUCTION

When the novel coronavirus (Covid-19) pandemic emerges, the spread of the virus has left the public keep anxiety if they do not have any effective cure. The World Health Organization (WHO) has declared Covid-19 as a pandemic due to the increase in the number of cases reported around the world. To contain the pandemic, many countries have implemented a lockdown where the government enforced that the citizens to stay at home during this critical period. The public health bodies such as the Centers for Disease Control and Prevention (CDC) had to make it clear that the most effective way to slow down the spread of Covid-19 is by avoiding close contact with other people . To flatten the curve on the Covid-19 pandemic, the citizens around the world are practicing physical distancing. To implement social distancing, group activities and congregations such as travel, meetings, gatherings, workshops, and praying had been banned during the quarantine period. The people are encouraged to use phone and email to manage and conduct events as much as possible to minimize person-to-person contact. The world has not yet fully recovered from this pandemic and the vaccine that can effectively treat Covid-19 is yet to be discovered. In reducing the impact of this coronavirus pandemic, practicing social distancing and self-isolation have been deemed as the most effective ways to break the chain of infections after restarting the economic activities. In fact, it has been observed that there are many people who are ignoring public health measures, especially with respect to social distancing. It is understandable that given the people’s excitement to start working again, they sometimes tend to forget or neglect the implementation of social distancing. Hence, this work aims to facilitate the enforcement of social distancing by providing automated detection of social distance violations in workplaces and public areas using a deep learning model. In the area of machine learning and computer vision, there are different methods that can be used for object detection. These methods can also be applied to detect the social distance between people. Deep learning has gained more attention in object detection and was used for human detection purposes. Develop a social distancing detection tool that can detect the distance between people to keep them safe. Evaluation of the classification results by analyzing  real-time video streams from the camera.

II. LITERATURE SURVEY

In [1] The author proposes an approach to estimating the distance between people to analyze whether social distancing is maintained. After obtaining the bounding box of people using YOLOv3, a width threshold is set for objects among which the distance is measured. By measuring the ratio of pixels to metres, the distance between two people in a given frame is approximated. However, this approach only calculates the distance between people without taking the individual distances of each person from camera into account. The major assumption is that each person is the same distance from the camera.

In [2] The author uses the YOLOv3 object detection framework to identify people in a given frame. After computing a pairwise distance between the centroids of the detected bounding boxes of people, this value is compared to a predefined minimum pixel threshold. By mapping these pixels to measurable units, detection of violations in social distancing norms are identified.

In [3] The MobileNet Single Shot Multibox Detector (SSD) object tracking model and the Open-CV library for image processing are used  to detect people in regions of interest. The distance between the humans detected in the video footage will be calculated and compared to a set of predetermined pixel values. The distance between the central points and the overlapping boundary between persons is measured. When harmful distances between persons are detected, notifications or cautions can be sent out to maintain the distance safe.

In [4] Computer Vision and MobileNet V2 architecture is used to  maintain a secure environment and ensure individuals protection by automatically monitoring public places to avoid the spread of the COVID-19 virus and assist police by minimizing their physical surveillance work in containment zones and public areas where surveillance is required by means of camera feeds with raspberry pi4 in real-time.

The proposed system will operate in an efficient manner in the current situation when the lockout is eased and helps to track public places easily in an automated manner. We have addressed in depth the tracking of social distancing and the identification of face masks that help to ensure human health.

In [5] Via using real-time deep learning based framework to automate the process of monitoring the social distancing via object detection and tracking approaches, where each individual is identified in the real-time with the 9 help of bounding boxes. The generated bounding boxes aid in identifying the clusters or groups of people satisfying the closeness property computed with the help of pairwise vectorised  approach. The number of violations are confirmed by computing the number of groups formed and violation index term computed as the ratio of the number of people to the number of groups.

In [6] The aurthor has developed a smart wearable device that can detact social distancing between the group of peoples. The system consists of an ultrasonic sensor, a microcontroller, a buzzer. In this work, the HC-SR04 ultrasonic sensor was used to detect the distance of any obstacle behind the person wearing the device. The function of the microcontroller is to read the value from the sensor. The program/ code is already uploaded in the microcontroller. The condition implemented in the code is that when the distance is greater than 1 meter the condition goes true and the buzzer starts ON. If the distance is less than 1meter the other condition goes true and buzzer will not ON this time.

In [7] The transfer learning approach for performance optimization with a deep learning algorithm is used to automatically monitor people in public places with a camera integrated with a local machine and to detect people with mask or no mask. We also do fine tuning, which is another form of transfer learning, more powerful than just the feature extraction. In this process camera video feeds from the Network Video Recorder (NVR) are streamed using RTSP and then these frames are converted to grayscale to improve speed and accuracy and are send to the model for further processing inside machine. We have used the RestNet50 architecture as the core model for detection as RestNet50 provides a huge cost advantage compared to the normal 2D CNN model. The process also involves the YOLOV3 Detector, a neural network architecture that has already been trained on a large collection of images such as Image Net and Pascal for high quality image classification.

In [8] The proposed system uses a transfer learning approach to performance optimization with a deep learning algorithm and a computer vision to automatically monitor people in public places with a camera integrated with a local machine and to detect people with mask or no mask. We also do fine tuning, which is another form of transfer learning, more powerful than just the feature extraction. In this process camera video feeds from the Network Video Recorder (NVR) are streamed using RTSP and then these frames are person's distance from the camera. After calculating the depth of the person in the camera, we calculate the distance between two people in the video. A number of people can be detected in a video. Thus, the Euclidean distance is measured between the mid-point of the bounding boxes of all detected individuals.. In the proposed system transfer learning is used on top of the high performing pretrained YOLOV3 model for face detection with RestNet50 architecture as backbone to create a lightweight model that is accurate and computationally efficient, making it easier to deploy the model to machine. The result of the YOLOV3model extracts a person mask and displays a bounding box.

In [9] First the input from video or camera is being given to the system. In this input by using image processing the person are detected using Yolo Algorithm which have been trained by training dataset. The detected person Centroid will be calculated and by Euclidean distance formula a pairwise distance between centroids will be calculated. By this we will get the distance between person and the person who are Violating Social Distancing will be counted and the person who are maintaining the social distance will also be counted.

In [10] First input is passed to the proposed system .It can be in form of photo snap, live streaming videos and recorded videos. Then system  reads  the input frame by frame if it reaches to EOF (End OF Frame) then the system will stop, else continues using algorithm called YOLOv3 (You Only Look Once) it gets the bounding boxes for face and person in the frame. Then using different algorithm for face mask detection and social distance detection can be computed, if the person in frame is wearing mask and maintaining distance then bounding box will be displayed in green colour if the person is not maintaining social distance and not wore face mask then bounding box will be in red colour.

In [11] Author proposed a social distance monitoring system for university campuses. The proposed system estimates social distances by using BLE packets among dedicated mobile nodes, collecting their data on a monitoring server via a campus-wide wireless LAN. We partially implemented the proposed system using M5StickC devices and conducted fundamental evaluations for between-node BLE communications. The results confirmed that it is possible to roughly estimate distances by using average or median RSSI values, that there are variations in RSSI depending on the orientations of persons wearing the monitor, and that sender-node battery power does not affect RSSI.

In [12]  Here CNN algorithm is used for Face-Mask detection and YOLOv3 algorithm is mainly used for calculating the social distance between every individual. The working of the module is described in the paper which has been divided into a few sub-modules such as the Admin Module, Pre-processing Module, Segmentation Module, Feature Extraction Module, CNN Module, Classify Dataset Module, and End User. To get more accurate results here splitting technique is used which is the Training and Testing part.As the camera gets on it will start taking records of video then this video will be going to split into an image. These split images are Pre-Processed for cleaning and organizing image later Segmentation is done on this image in which the image is divided as per object wise then next comes Feature Extraction which performs to remove all unusable objects and take only important objects from the processed image. On this processed image CNN and YOLOv3 algorithms operations are performed to detect whether a person is wearing a mask or not and also calculate social distance. If the person is not wearing a mask and/or if a person is not following social distance the alert voice command is passed from the system.

In [13]  To slow the spread of the COVID-19 virus via airborne transmission, although a “social distancing” policy of approximately 1.83 m (6 feet) is recommended, it needs to be scrutinized. By visualizing the trajectories and fates of SARS-CoV-2-laden droplets with various ambient conditions using the CFPD method with modeling the condensation/evaporation effects between the ambient water vapour and the droplets, this study finds that the 1.83 m (6 feet) social distancing policy is the “minimum requirement” and is not sufficient to avoid the contact of SARS-CoV-2 due to the complexity of environmental wind conditions. These conclusions do not consider the sneeze droplet cloud that can span approximately up to 8 m (27 feet). Therefore, a safe social distance is highly dependent on different factors, so it is highly recommended that people wear masks and other face coverings when in public.

In [14] Here mathematical model is used to find  the spread of infection in a population that structured by age and social contact between ages. Since contagion spreads through the structure of social contacts and the latter varies with age, it is necessary to resolve both these aspects of a population in any model that attempts to understand and predict how the modification of the social contact structure through social distancing impacts the spread of disease.

In [15] A deep learning based framework is proposed that utilizes object detection and tracking models to aid in the social distancing remedy for dealing with the escalation of COVID-19 cases. In order to maintain the balance of speed and accuracy, YOLO v3  alongside the Deepsort  are utilized as object detection and tracking approaches while surrounding each detected object with the bounding boxes. Later, these bounding boxes are utilized to compute the pairwise L2 norm with computationally efficient vectorized representation for identifying the clusters of people not obeying the order of social distancing. Furthermore, to visualize the clusters in the live stream, each bounding box is color-coded based on its association with the group where people belonging to the same group are represented with the same color. Each surveillance frame is also accompanied with the streamline plot depicting the statistical count of the number of social groups and an index term (violation index) representing the ratio of the number of people to the number of groups. Furthermore, estimated violations can be computed by multiplying the violation index with the total number of social groups.

In [16] In this paper, we developed a new social distance system that limits the spread of COVID-19 in crowded places. The main contribution of the proposed work is that it efficiently and cost-effectively guarantees safe social distancing between people indoors. The developed SD-Tag has been validated through several experiments, and achieved reasonable accuracy and user acceptability. We aim to employ wireless sensor networks to guarantee high localization accuracy, minimum cost, and low power consumption for future works.

In [17] The proposed technique was examined with two testing datasets to evaluate the capability of detection and localization of persons in the thermal images.YOLOv2 model detects people and provides the bounding box information. After people detection, the Euclidean distance between each detected centroid pair is computed using the detected bounding box and its centroid information based on dimensions of (x, y) for each bounding box.

As a further step, we designed and trained R-CNN and Fast R-CNN models for people detection with the same training datasets. Then compared these R-CNN and Fast R-CNN architectures with the technique proposed  using the same testing datasets of thermal images. YOLOv2 neural network looks the entire image at one time, unlike R-CNN and Fast R-CNN methods which see only the generated region proposals. Therefore, the proposed technique reduces the problem of background mistakes and improves the localization of detected persons in the image. In addition to that, the proposed approach shows better accuracy in comparison to other methodologies. The methodology proposed  is a promising one for people detection and social distancing classification on thermal images.

In  [18] This work proposed an AI- and monocular-camera-based real-time system to detect and monitor social distancing. In addition, our system utilized the proposed critical social density value to avoid overcrowding by modulating inflow to the ROI. The proposed approach was demonstrated using three different pedestrian crowd datasets. Quantitative validation was conducted over the Oxford Town Center Dataset that provides ground truth

pedestrian detections. There were some missed detections in the Mall Dataset and Train Station Dataset, as in some areas the pedestrian density is extremely high and occlusions occur.

However, after our qualitative and quantitative analysis, most pedestrians were successfully captured and the missed detections have an minor effect on the proposed method.

In [19] Social Distancing is the key to stop spread zoonotic diseases and prevent them from affecting a larger population. Considering this  a digital solution to enforce social distancing is developed using  PoseNet architecture which  has two variants, MobileNetV1 and Resnet50 as its backbone architecture.

We tune the hyper-parameter output-stride, which results into the output stride of the PoseNet model. Lower the value of output stride, higher is the resolution of the output. Lower output stride results into higher accuracy but with a trade-off in speed. MobileNetV1 is faster than ResNet50 but it is less accurate.

In  [20] The proposed work distinguishes the social distancing pattern and classifies them as a violation of social distancing or maintaining the social distancing norm. Additionally, it also displays labels as per the object detection. The classifier was then implemented for live video streams and images also.

This system can be used in CCTV for surveillance of people during pandemics. Mass screening is possible and hence can be used in crowded places like railway stations, bus stops, markets, streets, mall entrances, schools, colleges, etc. By monitoring the distance between two individuals, we can make sure that an individual is maintaining social distancing in the right way which will enable us to curb the virus.

III. PROPOSED METHODOLOGY

This Social distancing detection tool was developed to detect the safety distance between two people. This particular project is based on three other projects  (1)  Object detection (2) Object tracking and (3) Distance measurement between the detected objects. In object detection, we are using YOLO (You only look once) transfer learning process. There are other transfer learning methods for detecting objects like mobile net SSD etc but Here we are using Yolov4.

Yolov4 can detect 9000 classes. We are using the coco dataset here which is trained on 80 layers but we are using person class here so from 80 layers we are using only person class. After that, we will move onto object tracking. After the detection of the person class, we need to track them So we will assign a new id to every detected person and draw a box over them which is the centroid of the box.

We have detected two persons over here . we have drawn boxes over them and measured the centroid of the box and in the second frame purple one is the old centroid and yellow is the new centroid same in the third frame and there is a new person detected over here so how can we know that the person has moved from one point to another for this we will calculate euclidean distance from old centroids and new centroids and the close pairs will be detected as the same person. After we have tracked the person we need to measure the distance between the two.

A. Algorithm

// Input: Video Frame

//Output: Violation Detection

//Operation:

            1: Start

               2: Load Camera.

               3: Detect the number of pedestrian in frame.

               4: Transform perspective view into top-down view.

               5: Measure the distance between pedestrians using Euclidean Distance formula:

                              d = √ [(x2 – x1)2 + (y2 – y1)2]

                     Where,

                     “d” is the Euclidean distance

                     (x1, y1) is the coordinate of the first point

                     (x2, y2) is the coordinate of the second point.

6. Detect the social distancing measurement.

7. Indicate the result in output frame.

8. Stop.

B. Mathematical Model

The mathematical model helps us to follow the course of action we are going to take into our project module with the help of certain mathematical variables and formulas. When you tell someone about the project which follows certain method, whatever you are saying should be justified so based on your reasoning of a mathematical model, it can provide you the basis that your method of working is justified. The mathematical model uses language of the real and business world.

Let S be the Whole system

S = {D1, D2}; Where, D1, D2 are the distances between the pair of pedestrians.

S= {I, P, O}

I-input

P-procedure

O-output

• Input(I)

I = {Video frame, Detecting object, class probabilities}

              Where,

Checker -> Video frame, Detecting object, class probabilities

Model ->Analyze class probabilities

• Procedure (P)

P = {I, YOLO algorithm}

1. I – Video frame
2. YOLO Algorithm – For detecting the pedestrians
3. Detect: Safe Distance

• Output(O)

O = {Maintaining safe distance or not}

IV. RESULTS AND DISCUSSIONS

The video shows the pedestrian walking on a public street. In this work, the video frame is fixed at a specified angle to the street. The perspective view of the video frame is transformed into a top-down view for a more accurate estimation of distance measurement. shows the social distancing detection in a video frame and the results of the top-down view. The points represent each pedestrian for social distancing detection. The red points represent the pedestrians whose distance with another pedestrian is below the acceptable threshold and the green points represent the pedestrians who keep a safe distance from other pedestrians. However, there are also some detection errors shown. These errors are possibly due to the pedestrians walking too near to another pedestrian until they are overlaid on the camera view. The precision of the distance measurement between pedestrians is also affected by the pedestrian detection algorithm. The YOLO algorithm is also able to detect the half body of the pedestrian as an object by showing the bounding box, the position of the pedestrian corresponding to the middle-point of the bottom line is estimated based on the bounding box will be less precise. To overcome the detection errors, the proposed methodology had been improved by adding a quadrilateral box to observe the appointed region in an image as shown in figure 3. Hence, only the pedestrians walking within the specified space will be counted for people density measurement.

V. FUTURE SCOPE

Furthermore, the work can be further improved by optimizing the pedestrian detection algorithm, integrating other detection algorithms such as mask detection and human body temperature detection, improving the computing power of the hardware, and calibrating the camera perspective view. In the future, we will utilize this methodology on mobile cameras, e.g., mounted on an autonomous drone system, and hence drones are simpler to operate and more effective to capture fast actions of the detected objects from different angles. In addition to that, the newly released YOLOv5 detector will be also considered.

Conclusion

Social distancing is one of the important precautions in reducing physical contact that may lead to the spread of coronavirus. The consequences of non-compliance with these guidelines will be causing higher rates of virus transmission. This research presented an intelligent surveillance system for people tracking and social distancing classification. The proposed technique achieved promising results for people detection in terms of evaluation the accuracy and precision of the detector comparable to the other deep learning models. A specific algorithm was implemented on bounding boxes to distinguish between safe and unsafe conditions, respectively, marking as green and red the bounding box for detected persons. The proposed technique showed better results for real-time performance vs other object detectors. The proposed approach can be implemented in a distributed video surveillance system; indeed, it is a suitable solution for the authorities to visualize the compliance of people with social distancing.

References

[1] Anand Kumar, Bhawna Patle, Swara Hirwani,“Social Distancing Detector Using Deep Learning”,Novateur Publications,International Journal Of Innovations In Engineering Research And Technology [Ijiert] VOLUME 8, June. -2021. [2] Abdul hakeem Q. Albayati, Farah F. Alkhalid, Rafah K. Hussain, “Online Multi-Layers Social Distance Detection in Iraqi Schools”, Turkish Journal of Computer and Mathematics Education, Vol.12 No.6 , 5 April 2021 [3] Steffy Francis, Jensy V ,”A Survey on Deep Learning - Based COVID-19 Safety Monitoring”, Vol-7 2021. [4] Shashi Yadav, ”Deep Learning Based Safe Social Distancing and Face Mask Detection in Public Areas for COVID-19 Safety Guidelines Adherence ”,International Journal for Research in Applied Science & Engineering Technology (IJRASET),Volume 8,July2020. [5] Anis Sheikh, Asad Sheikh, Afnan Danish, Sayyed Faizan Ali, Mohammad Ubaid,”Social Distancing Detection Using Open-Cv ”,International Journal of Advanced Research in Computer and Communication Engineering ,Vol. 10, May 2021. [6] Rinisha C .P , Aruna B,”Wearable Social Distancing Detection System: A Review” 2021 IJCRT | Volume 9, Issue 10 October 2021. [7] Bhavik Limbasiya , Chinmay Raut,”COVID-19 Face Mask and Social Distancing Detector using Machine Learning ”,International Research Journal of Engineering and Technology (IRJET),Volume: 08. [8] Saurabh Yadav, Pawan Singh, ”Social Distancing Detector ”,Journal of Management and Service Science, 2021, Vol. 01. [9] Aman Kumar Sao, Harish Khedekar, Chirag Panpaliya, Shantanu Korde, Ms. Kavita S. Kumavat,“ Social Distancing Detector Using Image Processing”,International Journal of Advanced Research in Science, Communication and Technology (IJARSCT) Volume 4, April 2021. [10] Vishwesh M S, Nikhil D K, “Social Distancing and Face Mask Detection From CCTV Camera”,International Journal of Engineering Research & Technology (IJERT),Vol. 10 ,August-2021. [11] Yutaro Kobayashi, Yoshiaki Taniguchi, Youji Ochi, and Nobukazu Iguchi,” A System for Monitoring Social Distancing Using Microcomputer Modules on University Campuses”, May 15,2021, IEEE. [12] Anjali Sanjivanrao More, Shruti Kailas Yendhe, Priyanka Deepak Chandane, Bhavana Dnyaneshwar Sinalkar “Detection Of Face Mask And Social Distancing By Using Machine Learning Algorithm”,2022 IJCRT, Volume 10, Issue 1 January 2022 . [13] Yu Feng, Thierry Marchal, Ted Sperry, Hang Yi, “Influence Of Wind And Relative Humidity On The Social Distancing Effectiveness To Prevent COVID-19 Airborne Transmission”, A numerical study Journal of Aerosol Science. [14] Rajesh Singh, and R. Adhikari, ”Age-structured Impact Of Social Distancing On The COVID-19 Epidemic In India”, 26 Mar 2020 [15] Narinder Singh Punn, Sanjay Kumar Sonbhadra, Sonali Agarwal and Gaurav Rai, “Monitoring COVID-19 Social Distancing With Person Detection And Tracking Via Fine-Tuned YOLOv3 and Deepsort Techniques ”,27 Apr 2021. [16] Alhmiedat, T, “Social Distance Monitoring Approach UsingWearable Smart Tags”,Aborokbah,Electronics 2021. [17] Sergio Saponara, Abdussalam Elhadi Elhanashi, Alessio Gagliardi, “Implementing a Real-Time, AI-Based People Detection And Social Distancing Measuring System For Covid-19”, Journal of Real-Time Image Processing · December 2021. [18] Yang D, Yurtsever, E, Renganathan, V, Redmill, K.A., Ozgüner, Ü. “A Vision-Based Social Distancing and Critical Density Detection System for COVID-19”,2021. [19] Arnab Ghoraia, Sarah Gawdea, Dhananjay Kalband, “Digital Solution For Enforcing Social Distancing”, International Conference on Innovative Computing and Communication. [20] Rinkal Keniya, Ninad Mehendale , “Real-Time Social Distancing Detector Using Social Distancing Net19 Deep Learning Network”.

Copyright

Copyright © 2022 Hrishikesh Narayan, Ayush Pawar, Ankit Kumar, Avinash Narayan, Pooja Waware. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.