IoT Based Surveillance System Using DNN Model

Abstract

Localization, Visibility, Proximity, Detection, Recognition has always been a challenge for surveillance system. These challenges can be felt in the industries where surveillance systems are used like armed forces, technical-agriculture and other such fields. Most of the Smart system available are just for the surveillance of Human intervention but there is a need for a system which can be used for animals as well because with the outburst of human population and symbiotic relationship with wild animals results in life loss and damage to agriculture. In this paper we are designing to overcome these above-mentioned challenges for human and animal-based surveillance system in real time application. The system setup is done on a Raspberry pi integrated with deep-learning models which performs the classification of objects on the frames, then the classified objects is given to a face detection model for further processing. The detected face is relayed to the back-end for feature mapping with the saved log files with containing features of familiar face IDs. Four models were tested for face detection out of which the DNN model performed the best giving an accuracy of 94.88%.The system is also able to send alerts to the admin if any threat is detected with the help of a communication module

Introduction

I. INTRODUCTION

Peace is state of tranquility which is important to be organized in a society to live and for its social development, but the outburst of human population has resulted in increase in crime and burglary cases. One way to deal with this is by using CCTV (Closed-Circuit Television) but it can only help in giving a trail to the law enforcers that how the crime was committed or providing evidences [1].We need a system which can help in preventing the crime.

The Security system are mostly designed for the human, against the human but the human interference with the wildlife is also a major concern, we are cutting down jungles to develop new infrastructure, pushing habitat of animals, restricting them to live in a smaller space resulting in the increase cases of contact in-between. The wild animals sometimes enter into cities and villages which gets lethal to the life of either of the species so the need of a system for surveillance and protection has become a requirement for both. With the advancement in technology IoT based Surveillance system are very becoming very popular. IoT is basically a network in which a number of devices are always interconnected via internet[2]. Analyst has predicted a rapid growth in IoT based product and services [3].

In this paper, A Surveillance cum security system is developed for human as well as animals which can not only help in security but can also prevent crimes, burglary and human animal interaction. The system is developed on Raspberry pi 4 which has functionalities like object detection, classification and recognition using a camera for input. The most popular method for image processing is Computer Vision, with help of which machine can understand about the data in Images [4]. First step for face recognition is to detect a face in an image using Deep Learning Techniques, Deep Learning helps in improving the system time to time [5]. Haar Cascade Classifier, LBP Cascade Classifier are some of the regularly used facial detection algorithms [6]. We are using Telegram to send an alert whenever a threat is detected, from raspberry-pi4 to a user. The paper will now be formulated into sections as follows:

Section 2 Discuss the assessment of the substantial studies on which this paper focuses. Section 3 discuss about the proposed system. Section 4 Include the discussion related to results obtained and finally concludes with future scope.

II. RELATED WORK

(Amrutkar, Mistari et al. 2020) [7] Studied Multi-featured intrusion and hazard detection alarm system using IOT and object detection, the adopted methodology for the study was Face recognition and object detection , Sensor Module, Controller module, User Communication module, the results from the study were that security system can be divided into two parts hardware architecture and software features.

(Lumaban and Battung 2020) [8] The aim of this research paper is to develop a system to help detect the presence of criminals using descriptive and system development methods. For Software Rapid Application Development(RAD) methodology. Three face recognition algorithms, namely Local Binary Pattern Histogram, Eigenface, and Fisherface were tested and as a result LBPH algorithm got the highest face recognition accuracy rate with 95.92%. While Fisherface has 81.63% and Eigenface 51.02% accuracy rate.

(Prathaban, Thean et al. 2019) [9] The research aim to develop a system using OpenCV ,raspberry pi and a PIR sensor for surveillance the sensor was used to improve the efficacy of motion recognition the Haar-Cascade algorithm was applied in the initial stage. The detection rate for OpenCV was 100% whereas for PIR based motion recognition it was 76%.

(Khodadin, Pudaruth et al. 2020) [10] The system developed after the research has functionalities such as motion detection, object detection, face recognition , counting people and object displacement detection feature using Dlib toolkit for face recognition and a Twilio account was set up for alerting the admin or the user. The system developed was found to be reliable.

(Kaundanya, Pathak et al. 2017) [11]  The objective of this research was to overcome the drawbacks of the CCTV system and providing a cheaper and user friendly model the recognition was done when motion was detected using Raspberry pi integrated with camera and a PIR sensor working on Python Programming language and its libraries. For face detection Local Binary Pattern was used with which they were able to achieve results with 80% accuracy.

(Singh, Kaur et al. 2015) [12] The research results that LBP features are effective and efficient for facial expression recognition. In this a facial representation is predicated on statistical local features using, Local Binary Patterns, for facial expression. A face image is first split into small regions that LBP histograms are extracted and then concatenated in to a single feature vector.

(Han and Bhanu 2005) [13] the study objective is to investigate human repetitive activity properties from thermal infrared imagery for which a statistical approach is used to extract features for activity recognition, the results obtained by using statistical approach were decent enough and the performance for repetitive human activity was remarkable.

(Kadir, Kamaruddin et al. 2014) [14] The aim of this research was to find a better real-time face detection algorithm which evaluates two methods of face detection, Haar features and Local Binary Pattern features based on detection hit rate and detection speed tested on Microsoft Visual C++ 2010 Express with OpenCV library. The finding of the research is LBP gives more efficient and reliable results on real-time face detection system LBP detected 4.3% more faces than Haar on hit rate aspect and LBP detection speed is 140% faster.

(Patil, Ambatkar et al. 2017) [15] The objective was to develop an alert device using Raspberry Pi which connected to the internet. It can observe using Open-Source Computer Vision (OpenCV) for face recognition LBPH Recognition algorithm is used. The devices alert when gesture or motion is detected, the images are shown straight to a cloud where the observer can further take actions.

(Vadivukarasi, Krithiga et al. 2018) [16] The research aim’s to come up with a security system which is integrated with a face detection and recognition technique, the algorithm used for face recognition is Haar algorithm in OpenCV implemented using a Raspberry Pi 3 device which is used as a controlling unit coded in Python language. The system developed was user friendly, has high latency and low cost and dependable.

(Kumbhar, Singh et al. 2018)  [17] In this research paper an IoT enabled System is developed which can send security alerts to registered members through e-mail every time a human intrusion is detected. The system developed with the Raspberry pi-3 which has other components connected like PIR sensor, Microphone, Ultrasonic sensor, Buzzer. For Object detection YOLO algorithm is used which is performs remarkably as compared to other object detection techniques in terms of speed.

(Dahake and Mandaogade 2017) [18] Studied hardware implementation of face detection system using Raspberry Pi. Haar like feature is used for face detection which gives digital image features it has good calculation speed, for making a database of know Faces Haar Classifier is used over the detected face as training, and finally for Face Recognition is done using Local Binary Pattern (LBP). The system was capable for face detection even from poor quality images and shows excellent performance efficiency.

(Sayem and Chowdhury 2018) [19] The research paper objective is to develop a system which can perform face recognition over poor quality images. Raspberry Pi and camera modules are the main components, the algorithm used for face recognition is Local Binary Pattern Histograms. The system sends notification is an unknown subject is detected SMTP (Simple mail transfer protocol) is used for sending and receiving emails. After reviewing all these papers, we observed that some of the techniques and algorithms were common for developing a surveillance system like the use of python programming language for coding, use of Raspberry Pi as a microprocessor for processing the information such as image processing, send alerts to the users, taking input from sensors etc. LBH was the most common algorithm used for face detection as it is more efficient than other algorithms. Use of OpenCV for object detection which is one of the best open-source libraries to perform computer vision tasks.

III. PROPOSED SYSTEM

This section is about the design and implementation of our proposed system. As the input system, a camera is used, which is connected to a Raspberry pi 4. The camera sends data in from of image which are sent to the Raspberry pi as an input. These images are processed to get information about the objects in the camera frame. When an object comes in front of the camera the image data is processed by an algorithm which detects the object in the image and also classifies the object in a specific category for which is it pre-trained. From here the system developed system has 2 modes one for animal and another for human. General System Flow is described in Figure 1

Whenever a person is detected face data from the frame is matched with the data of faces in the database. The Database consists of face data of registered users. The System is provided with 400 images per user to train and learn on for face recognition .The information about the classified object is then looked upon in the database for processing. It will then verify about the object that if, the detected object can be a threat or not. If it is a threat then it will to notify the user about it by sending a message to all the registered users and then users can take necessary action to dissolve the threat.

A. Algorithm

The modules used are further explained in brief.

1. Camera: It is the input component. The camera gives stream of images as an input to the Raspberry pi for processing. For over system we are using a USB camera of 5Mega Pixel which is capable of recording full HD videos @ 30fps (frames per sec).
2. Raspberry Pi: It is a single board small credit card size microprocessor that can be used as robot brain, smart home hub and much more. We are using the latest model 4 in our system it is faster, more powerful, energy –efficient with 2GB LPDDR4-3200 SDRAM , USB 3.0 port and broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz processor. Brief documentation of this module is done [20]. Raspberry pi is being used in from a long period of time as the reason behind using it is because they are cheap, easy to maintain and provide high functionality.
3. YOLOv4: It is an state of the art object detection model [21] providing with faster object detection. Object detection models are trained to look at an image and search for a subset of object classes. When found, these object classes are enclosed in a bounding box and their class is identified [22] CNN (Convolutional Neural Networks) used for image processing, classification, segmentation are the backbone of the object detection. The basic idea behind the CNN is to map the image data to an output variable. YOLO (You Only Look Once[23] was written by Joseph Redmon. After this several other versions like YOLOv2, YOLOv3, YOLOv4 and now YOLOv5 have come out. Talking about YOLOv4, 3 models are the backbone of the YOLOv4 which are CSPResNet50, CSPDarkNet53, EfficientNet-B3[21] also it works faster than any other object detection model at the cost of accuracy. 
4. OpenCV: For image processing OpenCV(Open Source Computer Vision Library) is a very popular software library for computer vision related task. It is a huge open-source library. It has more than 2500 algorithms which can be used to detect and recognize faces, identify objects, classify human objects and other computer vision tasks. Originally it is written in C and C++ and now available in Python. Face Recognition is done after the face is detected in an image. Face detection involves identification and extraction[7]. The detected face data is then compared with the faces data in the database. There are many face detection algorithms available like LBP(Local Binary Patterns), Eigenface, Fisherface, Haar-Like and out of which LBP perform the best as observed in [8, 14]. It works very well with feature extraction from an image which is very important for an image processing model [12]. Data of the registered faces are saved in a .npy file. During the time of face recognition the data of the new face detected is matched from the face data stored in .yml file then the decision is made weather the face detected is a known face or not.
5. User Communication: In this module the system notifies the owner by sending customized text message and a picture. There are various methods to do this either by using an IoT app, SMTP or service like [5] Blynk, smartliving.io[24], Whatsapp, Telegram etc.  We are using the Telegram [6] app as it free, secure, fast and can have a 2 way communication. Python provides python-telegram-bot library making the integration with the python very easily. It also encrypts the message with AES-256 (Advanced Encryption standard).

IV. RESULT

The developed system can successfully recognize various daily use objects, in the (a) picture it can clearly identify a chair with 88% accuracy and a person with 87% accuracy within a single frame and in picture (b) it can detect dogs and cats as object with an average accuracy of 84%. It can also identify several other objects like a cell-phone, computer peripherals etc. on which the model is trained in it can be custom trained for other objects as well.

The Accuracy rate of the model to detect the face depends on various parameters one of which is the image resolution. The models were tested on this parameter and as result the DDN model performed the best out of all the four models is shown in Table I. and the variation is shown in Graph 1.

All four models were tested for 100 images as a result of which the DNN Model was able to detect the faces in the frame for most of the time giving an accuracy of 94.88%.

The proposed system was also successfully able to recognize different persons, given that the facial data should be stored in the system. The model was trained on a dataset of 100 images each of with 4 labels identified as ‘Known’ faces and any other face would be classified as ‘Unknown’. In (a) the system was successfully able to identify between a Unknown person by 80.50% accuracy and a Known person was detected in (b) with 87.30% accuracy. Hence, it can easily identify different members from same household.

References

Porter, G. (2009). CCTV images as evidence. Australian Journal of Forensic Sciences, 41(1), 11-25. [2] S. A. I. Quadri and P. Sathish, \\\"IoT based home automation and surveillance system,\\\" in 2017 International Conference on Intelligent Computing and Control Systems (ICICCS), 2017, pp. 861-866: IEEE. [3] Vyas, D. A., Bhatt, D., & Jha, D. (2015). IoT: trends, challenges and future scope. IJCSC, 7(1), 186-197. [4] X. Zhang, W.-J. Yi, and J. Saniie, \\\"Home surveillance system using computer vision and convolutional neural network,\\\" in 2019 IEEE International Conference on Electro Information Technology (EIT), 2019, pp. 266-270: IEEE. [5] S. A. Radzi et al., \\\"IoT based facial recognition door access control home security system using raspberry pi,\\\" vol. 11, no. 1, p. 417, 2020. [6] D. S. S. Mahesh, T. M. Reddy, A. S. Yaswanth, C. Joshitha, and S. S. Reddy, \\\"Facial detection and recognition system on Raspberry Pi with enhanced security,\\\" in 2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE), 2020, pp. 1-5: IEEE. [7] Amrutkar, A., Mistari, S., Thambave, K., & Pandhare, R. 2020. Home Security Using IOT and Machine Learning. EasyChair. International Journal of Computing and Digital Systems., 9(6):1251-1261. [8] Lumaban, M. B. P., & Battung, G. T. (2020). CCTV-Based Surveillance System with Face Recognition Feature. International Journal, 9(1.3). [9] T. Prathaban, W. Thean, and M. I. S. M. Sazali, \\\"A vision-based home security system using OpenCV on Raspberry Pi 3,\\\" In: AIP Conference Proceedings, 2019, vol. 2173, no. 1, p. 020013: AIP Publishing LLC. [10] F. Khodadin, S. J. I. J. o. C. Pudaruth, and D. Systems, \\\"An Intelligent Camera Surveillance System with Effective Notification Features,\\\" vol. 9, no. 6, pp. 1251-1261, 2020. [11] Kaundanya, C., Pathak, O., Nalawade, A. and Parode, S. 2017. Smart surveillance system using Raspberry pi and face recognition. International Journal of Advanced Research in Computer and Communication Engineering ,6(4): 621-624. [12] Singh, S., Kaur, A. and Taqdir. 2015. A face recognition technique using local binary pattern method. International Journal of Advanced Research in Computer and Communication Engineering. 4(3):165-168. [13] J. Han and B. Bhanu, \\\"Human activity recognition in thermal infrared imagery,\\\" in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\\\'05)-Workshops, 2005, pp. 17-17: IEEE. [14] K. Kadir, M. K. Kamaruddin, H. Nasir, S. I. Safie, and Z. A. K. Bakti, \\\"A comparative study between LBP and Haar-like features for Face Detection using OpenCV,\\\" In: 2014 4th International Conference on Engineering Technology and Technopreneuship (ICE2T), 2014, pp. 335-339: IEEE. [15] N. Patil, S. Ambatkar, and S. Kakde, \\\"IoT based smart surveillance security system using raspberry Pi,\\\" In:2017 International Conference on Communication and Signal Processing (ICCSP), 2017, pp. 0344-0348: IEEE. [16] Vadivukarasi, K., & Krithiga, S. (2018). Home security system using IOT. International Journal of Pure and Applied Mathematics, 119(15), 1863-1868. [17] Kumbhar, D., Chaudhari, H. , Taur, S. and Bhatambrekar, S. 2019. IoT Based Home Security System Using Raspberry Pi-3. International Journal of Research and Analytical Reviews (IJRAR). 6(4): 1-8. [18] M. Dahake and N. J. T. R. P. C. S. Mandaogade, \\\"Implementation of Raspberry Pi for Human Face Detection & Recognition,\\\" pp. 2-5, 2017. [19] I. M. Sayem and M. S. Chowdhury, \\\"Integrating face recognition security system with the internet of things,\\\" in 2018 International Conference on Machine Learning and Data Engineering (iCMLDE), 2018, pp. 14-18: IEEE. [20] RaspberryPi4 https://www.raspberrypi.org/products/raspberry-pi-4-model-b/specifications/ Last accesed on 01/06/2021/ [21] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. J. a. p. a. Liao, \\\"Yolov4: Optimal speed and accuracy of object detection,\\\" 2020. [22] Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934. [23] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, \\\"You only look once: Unified, real-time object detection,\\\" in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779-788. [24] S. N. Jyothi and K. V. Vardhan, \\\"Design and implementation of real time security surveillance system using IoT,\\\" in 2016 international conference on communication and electronics systems (ICCES), 2016, pp. 1-5: IEEE.

Copyright

Copyright © 2022 Rahul Rawat, Ratnesh Pd Srivastava. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.