Drowning Detection System using LRCN Approach

Abstract

This project provides the insights of a real-time video surveillance system capable of automatically detecting drowning incidents in a swimming pool. Drowning is the 3rd reason for the highest unintentional deaths, and that’s why it is necessary to create trustable security mechanisms. Currently, most of the swimming pool\'s security mechanisms include CCTV surveillance and lifeguards to help in drowning situations. But this method is not enough for huge swimming pools like in amusement parks. Nowadays, some of the security systems are using AI for drowning detection using cameras situated underwater at a fixed location and also by using floating boards having a camera mounted on the bottom side so that underwater view can be captured. But the main problems in these systems arise when the pool is crowded and vision of cameras is blocked by people. In this project, rather than using underwater cameras, we are using cameras situated on top of the swimming pool to get an upper view of the swimming pool so that entire swimming pool will be under surveillance all time.

Introduction

I. BACKGROUND AND MOTIVATION

Drowning is the 3rd reason for the highest unintentional deaths, and that’s why it is necessary to create trustable security mechanisms. Currently, most of the swimming pool's security mechanisms include CCTV surveillance and lifeguards to help in drowning situations. But this method is not enough for huge swimming pools like in amusement parks.

Some of security systems are using AI for drowning detection using cameras situated underwater at a fixed location and also by using floating boards having a camera mounted on the bottom side so that underwater view can be captured. But the main problems in these system arises when the pool is crowded and vision of cameras are blocked by people. In this project, rather than using underwater cameras, we are supposed to use cameras situated on top of the swimming pool to get an upper view of the swimming pool.

II. INTRODUCTION

Drowning is the 3rd reason for the highest unintentional deaths, and that‘s why it is necessary to create trustable security mechanisms. This project aims to create a system that will be able to automatically detect drowning incidents in the swimming pool using human action detection. The drowning detection model will be used to process and classify video that will be given to the system which will be recorded using live surveillance cameras. The system will break this video in image frames and apply model over it and if the early actions of drowning like hand waving, water splashing or diving is detected then the system will set the alarm so that the lifeguards can initiate their rescue operations. The classifier model is trained using a Long-term Recurrent Convolutional Network which is a combination of convolutional neural network and recurrent neural network which is suitable for large-scale visual understanding tasks such as activity recognition and image captioning.

III. LITERATURE REVIEW

This section aims to identify and discuss the lacunae and similarities respectively, in some of the previous works related to drowning detection systems.

Lei Fei, Wang Xueli, Chen Dongsheng, proposed a background subtraction method for drowning detection and swimmer identification using visual surveillance in their research paper. This method fails to reflect real background accurately thus restricting model accurate shape detection of moving objects. It also fails to reflect sudden background changes.

Ajil Roy, Dr. K. Srinivasan, proposed drowning detection using RFID-based swimming goggles, however, this model also fails to overcome the limitation of accuracy since the water sensor is not placed very close to the mouth and nose. But this model successfully overcomes limitations of video surveillance-based drowning detection systems like the need for high power computing devices.

Chi Zhang, Xiaoguang Li, Fei Lei, proposed ?A Novel Camera-Based Drowning Detection Algorithm? using input video sequences obtained from underwater cameras. In this case, to detect drowning swimmers an implementable real-time detection system with high accuracy will be needed.

IV. PROPOSED METHODOLOGY

In particular, a specific type of neural networks called Convolutional Neural Networks (CNNs) is best suited for the task of image recognition. So implementation of Long Term Recurrent Convolution Network (LRCN) approach suitable for Video Classification & Action Recognition.

The Long Term Recurrent Convolution Network methodology is a combination of Convolutional Neural Network (CNN) & Recurrent Neural Network (RNN). LRCN is end-to-end trainable and appropriate for vast visual understanding tasks such as video description, activity recognition and image captioning. The main idea is to learn visual features from video frames with the help of CNN & then use LSTM layers to transform a sequence of image embeddings into a class label, sentences, probabilities, etc. 

In this segment, we empirically propose LRCN approach for the implementation of drowning detection as CNN extracts the features from the input provided to the model and then the LSTM layers predict the action of the human whether one is drowning, swimming or diving.

A. Convolutional Neural Networks

Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm that can take in an input image and assign learnable weights to various features in the image. As compared to other classification algorithms, Convnet requires less preprocessing. Filters are hand-engineered, with enough training in primitive methods and ConvNets can learn these filters/characteristics.

B. Long Short-Term Memory Networks

LSTM Networks is a supplement of recurrent neural networks (RNNs) mainly introduced to handle situations where RNNs fail. RNN is a network that works on the present input by taking into account the previous output (feedback) and storing it in its memory for a short period of time (short-term memory). Long Short-Term Memory (LSTM) overcomes the vanishing gradient problem as the training model is left unaltered. Long time lags in certain problems are crossed using LSTMs which also handle noise, distributed representations, and continuous values. LSTMs provide us with an immense range of parameters such as learning rates, and input and output biases. Hence, no need for fine adjustments. The update complexity weight is reduced by O(1) in LSTM’s.

Thus, we suggest two approaches for implementing the drowning detection-

1. Using ConvLSTM2D layers in the  model
2. Using LRCN approach

As shown in the figures below, both of this approaches provide convincing results and proper detection if someone’s drowning in the swimming pool.

The project has three phases:

1. Preprocessing of Data: The Video Dataset used for training the LRCN model is a blend of the UCF50 dataset & videos that are downloaded from Youtube. The video dataset of swimming and drowning is not available on any websites so we had to download related videos for the same, one by one from youtube. But these videos had extra activities like in swimming videos there was a person approaching the swimming competition, taking their stance and handshakes after the race. In drowning-related videos, there was a rescue operation as well in video. So we cropped such extra activities and kept only the required part of the video which were actual swimming and drowning actions.  Finally, the videos of diving, breaststroke swimming from the UCF50 video dataset were combined with the data that was created by us for further training of the models. For training, the model image frames were extracted from each video and used for model training, and the same for testing.
2. Creating Deep Learning Model: In this project, we have used two models which are the ConvLstm2D model and the LRCN model. In this phase, we used training set videos to train the model by passing them through multiple layers of the ConvLstm2D network and also implemented the LRCN network to compare the better results provided by the respective models. The architecture of both models are as follows:

A. ConvLstm2D Model

B. LRCN Model

V. SOFTWARE DESIGN

A. Incremental Model

The incremental model is a process of software development where requirements are broken down into multiple standalone modules of the software development cycle. Incremental development is done in various stages such as analysis design, implementation, testing/verification, and maintenance.

Each iteration passes through the analysis, design, coding, and testing phases, and each subsequent release of the system adds function to the previous release until all designed functionality has been implemented and accepted by the user.

The first increment is often a core product where the basic functionalities are addressed, and supplementary features are added in the next increments. After completion of the first iteration, the product is handed over to the end-users for their feedback. Once the core product is analyzed by the client, further improvements are done in the next increment.

VI. RESULT AND DISCUSSION

The outputs achieved will predict the class names for a batch of frames of the videos given as input. The predicted class name having the highest probabilities will be detected as the action being performed in the swimming pool.

1. The predicted class name having maximum probability can be displayed as the confidence variable.
2. After the successful completion of the project, one can:
3. Observe the video surveillance and rely on the drowning detection system.
4. An alarm will be raised if someone is detected as drowning.
5. Drowning preventive measures can be performed due to early alerts raised by the system.
6. The project should examine the actions performed by swimmers to detect drowning more accurately.

VII. FUTURE SCOPE

Availability of better dataset, modern methodologies, and technologies with high computational power accompanied by high-quality surveillance cameras, will help to improve the accuracy of drowning detection & even can be used in adverse conditions. 

After the implementation of all these essentials, this system also can be used on sea beaches for drowning detection

Conclusion

Once we have the working drowning detection model we can feed live video footage of the swimming pool to it so that it can keep detecting continuously for any drowning activities. If drowning is detected it will be highlighted on the system screen as well as alarms will be raised to alert security guards so that they can initiate rescue

References

[1] https://link.springer.com/content/pdf/10.1007/3-540-47979-1_20.pdf [2] https://scholar.archive.org/work/k6feepj5z5dfxmwar32flg2jqq/access/wayback/http://www.es.sdu.edu.cn/project/doc/papers/icess09-papers/15-1.pdf [3] https://www.angeleye.tech/en/ [4] Long-term Recurrent Convolutional Network (LRCN) - Home (kobiso.github.io) [5] https://cseweb.ucsd.edu//classes/wi19/cse291-g/student_presentations/Image_Caption_LRCN.pdf

Copyright

Copyright © 2022 Shardul Sanjay Chavan, Sanket Tukaram Dhake, Shubham Virendra Jadhav, Prof. Johnson Mathew. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.