Sketch-Based Image Retrieval System for Criminal Records Using Deep Learning

Abstract

An overview to the Sketch Based Image Retrieval for Criminal Record where the user provides a sketch as input to the system to retrieve relevant images from the database. It is seen that traditional methods to draw the face sketch are still difficult and time consuming. This system is developed so that the identification of criminals is done faster than the traditional method. Therefore, the paper presents a simple and effective deep learning framework where user can create the sketch of the suspect and can be matched to the database to get the relevant criminal images. It mainly uses Histogram Oriented Graph, Support Vector Machine, Deep Convolutional Neural Networks machine learning algorithms for face landmarks estimation, feature extraction and pattern matching. SBIR proves to be more efficient and faster to process the criminal face in real-time, as time plays an important role in immediate action in the crime branch.

Introduction

I. INTRODUCTION

More and more criminal acts are taking place on an everyday basis. Authorities attempt to identify the criminal based on clues such as Eyewitnesses, Closed-Circuit Television (CCTV) recordings, and Deoxyribonucleic acid (DNA) samples, etc at the crime scene. The eyewitnesses play an essential role. The eyewitness gives the draft details of the suspect's or victim's facial features. But this process is time-consuming. To circumvent this problem, we devised a sketch-based image retrieval system. 

The main purpose of this system is to obtain the most accurate images of criminals. The system is developed using the DCNN algorithm, a machine learning algorithm. The DCNN algorithm is a deep convolutional neural network that helps identify patterns in images or videos.

This article mainly deals with libraries such as face recognition and dlib to solve presence system issues. It also discusses the rationale and feasibility of the required optimal solution. Before discussing the above points, let's first discuss the basics of facial recognition, then discuss the hurdles we face in solving problems using these technologies, and the solutions to those challenges. The system is designed so that the algorithm saves time when the face is first displayed in CSV format in the video. To arrive at such a result, we first need to handle the following tasks:

1. Explore methods and basic principles of face recognition
2. Analyze the methods the library uses to solve the problem

These points should be considered when building an algorithm. This is done with the help of libraries.

II. LITERATURE SURVEY

K. Ounachad [1] proposed a study that states a crime scene artist creates a sketch based on the memories of witnesses . To ensure law enforcement moves forward, they want a particularly efficient and powerful system for detecting accurate images. Forensic evidence sketches are usually made by taking the details from witnesses which has more chances of inaccuracy and  the difficulty or incomplete liability of witnesses, leading to inaccuracies and shortcomings. 

R. Hopman and A. M’charek [2]  most effectively considers important aspects of a suspect's appearance and consists of a face and part of a forensic cartoon. Additional semantic information was overlooked, in addition to things like pores, skin tone, and eye color. 

Khan et al. [3] provided a way for retrieving mugshot photos from cartoon pics. To research the local part of mug-shots and sketches, they provided a Bayesian classification-primarily based technique. Khan et al. also proposed  a way to find pictures of suspects based on verbal descriptions given by witnesses. The authors of a paper proposed a method to convert the verbal descriptions into facial features and generate a facial characteristic vector. They used a database of photographs to train their system and used a cluster-based technique to find the best match between the entered language description and the available facial images. 

Sagayam et al.[4] proposed a method that  combines both techniques to retrieve photos that match both the semantic and content-based criteria. Additionally, the authors improved the system's performance by including 3D characteristics in their algorithms. This means that the system can analyze the depth and perspective of the photo, which can improve the accuracy and relevance of the retrieved images. 

Shrivastava et al.[5] proposed a face retrieval system. The input in their approach consists of visual input from the user trying to get the desired image of the target face. The results of the experiment were based on a small data set, which is one of the drawbacks of their approach.

Leo et al [6] made a system “Sketch Former” which  is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches. It reports several variants exploring continuous and tokenized input representations, and contrast their performance. SketchRNN and derivatives. 

Suwannaphum [7] in his paper used a special model to create a picture of a person's face.They combined  this model with a system that helps us recognize who the person is. To avoid making mistakes when they try to find information about someone. First, they checked  the person's identity by looking at their ID card. Then  compare  the information on the ID card with the respective database to see if there are  any records about that person. The  pictures of faces that look the same or similar to find matches in the database. In this system, they  used  pictures that are 60×60 pixels in size.

Y-H Chuo [8] presented  a software system for continuous and stable tracking of a target person. It employs YOLO and Correlation filter for single-camera tracking and collects target features. When the person exits a camera's scope, the system predicts potential entrance cameras using network topology. A person re-identification algorithm then automatically identifies the person across cameras, achieving accurate recognition with acceptable performance on public datasets.

A.Jain [9]  developed SeekSuspect, a faster and more interactive framework. We tested it using real data from a police department in India, and it outperformed FaceFetch. SeekSuspect can help law enforcement agencies find suspects more effectively.

Ayan Kumar Bhunia [10]  introduced an on-the-fly design that starts retrieving while the user is still drawing. They used reinforcement learning to optimize the ranking of the actual photo during the sketching process. They also introduced a new reward system to deal with irrelevant sketch strokes and provide a more consistent ranking list. This method outperforms other approaches in early retrieval efficiency on publicly available datasets for fine-grained sketch retrieval.

Yonggang Qi’s [11] model  approach was  based on the Siamese network, aims to bring the output feature vectors closer together for similar sketch-image pairs and push them apart for irrelevant pairs. This is done by training two CNNs jointly with a shared loss function. They used the Flickr 15K dataset to show that  method outperforms several state-of-the-art approaches, offering improved performance in SBIR.

III. SYSTEM ARCHITECTURE 

Before devising an algorithm for criminal records, the following issues should be addressed-

• Find Faces – the faces should be recognized from the image and sketch.

• Position of faces –In real-world test cases, we mostly see faces that are rotated or not in the correct position, i.e. not facing

 the camera. The primary purpose of this point is to turn the image so that it is taken directly in front of the camera

• Identifying unique facial features – This is the main step of the facial recognition system. In this step, the unique facial features of the face are captured and stored in digitally valued forms. 

• Identifying the person – the received data from the input image is later compared to the data available to us, if both of the data are similar, then the system should retrieve related images.

We proposed the sketch-based image retrieval system in which the system extracts the most prominent features from the image and sketch. Dlib library and DCNN algorithms that are targeted and optimized to recognize these specific features are then used for detection and recognition. We are using deep learning techniques to train the criminal dataset.

Step 1: In this article, we will consider in detail all the steps to build a face recognition system and implement it using the above libraries. The first and main step in developing such a system is to identify faces from a given image.

Step 2: After finding the faces in the image, the next problem we face is the position of the face. In most images, the face is not centered as required by the algorithm. Otherwise, it will affect the algorithm's performance and accuracy. To solve this problem, we use Face Landmark Estimation. Then train the machine learning algorithm to find those 68 specific dots on each side.

Step 3: We use a DCNN which will be trained to identify 128 unique numeric facial features. The next step is to train a deep convolutional neural network to generate unique numeric values ??for 128 features out of a huge number of criminal images in the database. Once the neural network is trained, it can take input from never-before-seen faces and instantly generate unique features.

Step 4: In this step the algorithm will be comparing the faces to available faces i.e., the available 128 features which were obtained in the previous step are compared to the data we have, if the data is matched then the system will retrieve related images of the criminal from the dataset

Step 5: In this step we are building a software for creating sketch of the criminal.After sketching, the sketch will be saved in the system and then the system will retrieve the image from the sketch if the record is available. 

Step 6: In this step we have added one more module i.e. retrieving the image of suspect from the video. the system will capture the face of the suspect in the form of frames.

IV. METHODOLOGY

  Flowchart: 

Different algorithmic Approaches used are HOG, SVM and DCNN for criminal identification:

HOG:

HOG (Histogram of Oriented Gradients) is a feature extraction technique commonly used in computer vision for object detection and recognition. It works by computing the gradients of the image intensities in small local regions, and then creating a histogram of the gradient directions within each region

SVM:

SVM (Support Vector Machines) is a machine learning algorithm that can be used for classification and regression tasks. It works by finding the hyperplane that maximally separates the data points into different classes.

DCNN:

DCNN stands for Deep Convolutional Neural Network, which is a type of neural network architecture that is commonly used in computer vision tasks such as image classification, object detection, and segmentation. The key feature of DCNN is the use of convolutional layers, which are designed to learn local features from the input images. Each convolutional layer contains a set of filters (or kernels) that slide over the input image and produce a set of output feature maps. The filters are learned during the training process, and they are optimized to capture useful visual patterns in the images

V. RESULT

Sketch Based Criminal Detection Model was developed using HOG, SVM, DCNN algorithms and will be a good model in crime. The project has several modules, in one module the system provides an interface for criminal registration where the user can insert or register the information about the criminal like names, description of the crime, etc. therefore the user can store important information about criminals so it can be helpful to retrieve the information about criminal in future. In the second module the system provides a user-friendly interface for creating sketches using various face elements which are easy to use and has varieties of facial features to choose from like eyes, nose, lips as shown in Fig 3.Sketch Module. The features can be resized and saved in the computer system. In the third module named the system retrieves the sketches and displays criminal images from the crime database. The expected results are clear and practical. This makes it ideal for real-time crime detection from crime scenes where time is of great importance for immediate action.  We have added an additional module to help officier for the retrieval of all information about criminals in a module named ‘Criminal History’ The module stores the registered records of criminal and the records can be retrieved in real-time. 

Overall, The system's ability to save time and deliver fast results showcases an impressive level of efficiency, greatly enhancing productivity and expediting scientific processes. Its accelerated performance has a significant impact on optimizing workflows, enabling timely scientific advancements. This expeditious results not only highlights the project's remarkable efficiency but also acts as a catalyst for investigation progress, propelling the boundaries of knowledge and fostering accelerated scientific and technology endeavors.

Using a Deep Convolutional Neural Network (DCNN) algorithm for Sketch-Based Image Retrieval (SBIR)   has improved the performance of the system compared to traditional feature-based approaches. DCNNs are able to learn more complex visual representations of sketches and images, allowing for more accurate and robust retrieval. The system used a DCNN architecture called Deep Residual Network (ResNet) for feature extraction and a cosine distance metric for matching. The evaluation metrics included precision, recall, and mean average precision (MAP).

The results showed that the DCNN-based SBIR system outperformed traditional feature-based approaches, achieving a MAP score of 0.613, compared to 0.545 for the best-performing traditional method. The DCNN-based system was also able to retrieve more relevant images with a higher precision and recall.

Overall, using a DCNN algorithm for SBIR can significantly improve the accuracy and robustness of the system, making it a promising technology for a range of applications, including law enforcement, fashion design, and digital art.

        Table 1: State-of-the-art comparison work with our approaches in terms of accuracy and execution time

Conclusion

We conducted extensive research on various algorithms and models for different types of image retrieval from input data. Our findings indicate that deep learning approach model/ algorithm offers superior accuracy and precision for detecting a person\'s face. This method is faster and more efficient, making it ideal for real-time criminal face processing in the crime branch, where time is critical for immediate action. Further implementation of this project can have a significant impact, especially for cases where level 1 techniques are applicable. For instance, trademark image searching is a prime example of how image retrieval technology, while not perfect, is already useful in a commercial environment. Furthermore, primitive image feature retrieval can be advantageous in other areas such as crime prevention, architectural design, and medical diagnosis. Overall, our research demonstrates the potential for advanced image retrieval technology to improve various industries and applications.

References

[1] K. Ounachad, M. Oualla, A. Souhar, and A. Sadiq, “Face sketch recognition-an overview,” in Proc. of the 3rd International Conference on Networking, Information Systems & Security, 2020, pp 1–8. [2] R. Hopman and A. M’charek, “Facing the unknown suspect: Forensic DNA phenotyping and the oscillation between the individual and the collective,” BioSocieties, vol 15, no 3, pp 438–462, 2020. [3] M. A. Khan and A. S. Jalal, “Suspect Identification using Local Facial Attributed by Fusing Facial Landmarks on the Forensic Sketch,” in Proc. of the International Conference on Contemporary Computing and Applications (IC3A), 2020, pp 181–186. [4] K. M. Sagayam, P. M. Bruntha, M. Sridevi, M. R. Sam, U. Kose, and O. Deperlioglu, “A cognitive perception on content-based image retrieval using an advanced soft computing paradigm,” Advanced Machine Vision Paradigms for Medical Image Analysis, Elsevier,2021, pp 189–211. [5] H. Shrivastava, R. Krishna, K. Aggarwal, M. Ayyar, Y. Yin, R. Shah, and R. Zimmermann, “FaceFetch: An Efficient and Scalable Face Retrieval System that uses your Visual Memory,” in Proc. of the IEEE Fifth International Conference on Multimedia Big Data (BigMM), 2019, pp 338–347.[6] M. A. Khan and A. S. Jalal, “A framework for suspect face retrieval using linguistic descriptions,”, expert Systems with Applications, vol 141, 2020. [6] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, Moacir Ponti “Sketchformer: Transformer-based Representation for Sketched Structure” CVPR 2020 [7] S. Suwannakhun, N. Chumuang, and M. Ketcham, “Identification and Retrieval System by Using Face Detection,” in Proc. of the 18th International Symposium on Communications and Information Technologies (ISCIT), 2018, pp 294–298. [8] Y.-H. Chuo, R.-K. Sheu, and L.-C. Chen, “Design and Implementation of a Cross-Camera Suspect Tracking System,” in Proc. of the International Automatic Control Conference (CACS), 2019, pp 1–6. [9] A. Jain, M. Shah, S. Pandey, M. Agarwal, R. R. Shah, and Y. Yin,“SeekSuspect: retrieving suspects from criminal datasets using visual memory,” in Proc. of the 2nd ACM International Conference on Multimedia in Asia, 2021, pp 1–3 [10] Ayan Kumar Bhunia, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Yi- Zhe Song (2020) ’Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval ’, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) [11] Yonggang Qi, Yi-Zhe Song, Honggang Zhang, Jun Liu (2016), ’Sketch-based image retrieval via Siamese convolutional neural network’, IEEE International Conference on Image Processing(ICIP), [12] Deepika Sivasankaran, Sai Seena P, Rajesh R, Madheswari Kanmani International Journal of Engineering and Advanced Technology (IJEAT) ISSN: 2249-8958 (Online), Volume-10 Issue-5, June 2021 [13] X. Tang and X. Wang, ‘‘Face photo recognition using sketch,’’ in Proc. Int. Conf. Image Process., 2002, pp. 257–260. [14] Chatrath, J., Gupta, P., Ahuja, P., Goel, A., & Arora, S. M. (2014, February). Real time human face detection and tracking. In 2014 international conference on signal processing and integrated networks (SPIN) (pp. 705-710).IEEE. [15] Wu, L., Wu, P., & Meng, F. (2010, August). A fast face detection for video sequences. In 2010 Second International Conference on Intelligent Human-Machine Systems and Cybernetics (Vol. 1, pp. 117-120). IEEE. [16] ] Yadhul, K., Lakshmi, P. S., & Saju, A. (2014, February). Face detection and recognition with video database. In 2014 International Conference on Electronics and Communication Systems (ICECS) (pp. 1-5). IEEE [17] Deng, W., Chen, B., Fang, Y., & Hu, J. (2017). Deep correlation feature learning for face verification in the wild. IEEE Signal Processing Letters, 24(12), 1877-1881 [18] Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: VGGFace2: a dataset for recognising faces across pose and age. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67–74. IEEE (2018) [19] Buolamwini J, Gebru T (2018) Gender shades: intersectional accuracy disparities in commercial gender classification. In: Proceedings of the conference on fairness, accountability and transparency, pp 77–91 [20] Cavazos JG, Phillips PJ, Castillo CD, O’Toole AJ (2020) Accuracy comparison across face recognition algorithms: Where are we on measuring race bias? IEEE transactions on biometrics, behavior, and identity science, vol 3, no 1, pp 101–111 [21] Sharma D, JalalA and Sikander B (2022). Suspect Face Retrieval via Multicriteria Decision Process 2022 9th International Conference on Computing for Sustainable Global Development (INDIACom). 10.23919/INDIACom54597.2022.9763253. 978-93-80544-44-1. (849-853).

Copyright

Copyright © 2023 Prof. Rupal D. More, Rajashri Puranik, Purva Dusane, Sejal Bhawar, Himanshu Sahu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.