Forensic Face Sketch Construction and Recognition

Abstract

In forensic science, it is seen that hand-drawn face sketches are still very limited and time consuming when it comes to using them with the latest technologies used for recognition and identification of criminals. In this paper, we present a standalone application which would allow users to create composites face sketch of the suspect without the help of forensic artists using drag and drop feature in the application and can automatically match the drawn composite face sketch with the police database much faster and efficiently using deep learning and cloud infrastructure.

Introduction

I. INTRODUCTION

A criminal can be easily identified and brought to justice using a face sketch drawn based on the description been provided by the eye-witness, however in this world of modernization the traditional way of hand drawing a sketch is not found to be that effective and time saving when used for matching and identifying from the already available database or real-time databases.

During the past there were several techniques been proposed to convert hand-drawn face sketches and use them to automatically identify and recognize the suspect from the police database, but these techniques could not provide the desired precise results. Application to create a composite face sketches were even introduced which too had various limitations like limited facial features kit, cartoonistic feel to the created suspect face which made it much harder to use these applications and get the desired results and efficiency. The above applications and needs motivated us into thinking of creating an application which would not just provide a set of individual features like eyes, ears, mouth, etc. to be selected to create a face sketch but also would allow user to upload hand-drawn individual features on the platform which would then be converted in to the applications component set. This in turn would make the created sketch  much more similar to the hand-drawn sketch and would be much easier for the law enforcement departments to adapt the application.

Our application would even allow the law enforcement team to upload previous hand-drawn sketch in order to use the platform to identify and recognize the suspect using the much more efficient deep learning algorithm and cloud  infrastructure provided by the application. The machine learning algorithm would learn from the sketches and the database in order to suggest the user all the relatable facial features that could be used with a single selected feature in order the decrease the time frame and increase the efficiency of the platform.

II. LITERATURE SURVEY

There are lot of studies on face sketch construction and recognition using various approaches. Dr. Charlie Frowd along with Yasmeen Bashir, Kamran Nawaz and Anna Petkovic designed a standalone application for constructing and identifying the facial composites, the initial system was found to be time consuming and confusing as the traditional method, later switching to a new approach in which the victim was given option of faces and was made to selected similar face resembling the suspect and at the end the system would combine all the selected face and try to predict automatically the criminal’s facial composite. The Results where promising and 10 out of 12 composite faces where named correctly out of which the results 21.3% when the witness was helped by the department person to construct the faces and 17.1% when the witness tried constructing faces by themselves.

Xiaoou Tang and Xiaogang Wang proposed a recognition method of photo-sketch synthesized using a Multiscale Markov Random Field Model the project could synthesis a give sketch into photo or a given photo in to sketch and then search the database for a relevant match for this the model divided the face sketch in to patches.

In this they first synthesized the available photos in to sketch and then trained the model making the model to decrease the difference between photos and sketch this enhanced the overall efficiency of the recognition model. For testing this they took few samples in which the photos where synthesized in to sketch and the same faces where drawn from sketch artist.

Another proposed method was sketch to photo matching proposed by Anil K Jain and Brendan Klare which used SIFT Descriptor, the method proposed displayed result based on the measured SIFT Descriptor distance between the face photos in the database and the sketches. 

The algorithm first  converts the face photos using linear transformation which was based on Tang and Wang proposed model and then the sketch was used to measure the SIFT descriptor distance compared to the face photo and in some cases distance between images in the databases too where measured for better accuracy. The experimental result shows that the dataset used where very similar to the those used by Tang in their experiment and the addition in the algorithm was the measurement of the descriptor which gave a better result and accuracy from the model proposed by Tang and Wang.

P. C. Yuen and C. H. Man too proposed a method to search human faces using sketches, this method converted sketches to mug shots and then matched those mugshots to faces using some local and global variables been declared by the face matching algorithms. However, in some cases the mugshots where hard to be matched with the human faces in the databases like FERET Database and Japanese Database. The proposed method showed an accuracy of about 70% in the experimental results, which was fair decent but still lacked the accuracy needed by the law enforcement department.

The common issue with all the proposed algorithm where that they compared the face sketches with human face which were usually front facing making it easier to be mapped both in drawn sketch and human face photograph, but when a photograph or sketch collected had their faces in different direction the algorithms were less likely to map it and match with a face from the database which is front facing. There are even system been proposed for composite face construction but most system used facial features which where been taken from photographs and then been selected by the operator as per described by the witness and at last complied to form a single human face making it much more complicated for human as well as any algorithm to match it with a criminal face as every facial feature was been taken from the separate face photograph having various dissimilarity and when combined together made it harder to recognize. Thus, all the previous approaches proved either inefficient or time consuming and complicated. Our application as mentioned above would not only overcome the limitations of the mentioned proposed techniques but would also fill in the gap between the traditional hand-drawn face sketch technique and new modernized composite face sketch technique by letting user to upload the hand-drawn face sketches and facial features.

III. METHODOLOGY

A. System Flow

The Fig. 6. Illustrates the overall flow of the system starting with the login section which ensuring the two-step verification process. Further the application can either be used with a hand-drawn sketch or a composite face sketch can be created using the drag and drop feature, Either of the images would then go under features extraction process which would help the application to apply image processing and computer vision algorithm and finally match the sketch with the database and then display the ratio of similarities between the sketch and the data base photograph.

In this application, Operations are performed in two stages.

1. Face Sketch Construction: The flowchart illustrates the users flow been followed by the platform to provide an construct accurate face sketch based on the description, the dashboard is designed simple in order to encourage no professional training to go through before using this platform already saving the timeframe which would have been taken a lot time and resources of the Department.

The dashboard consists of Five main modules, First the important module is the Canvas been shown at the middle of the dashboard which would house the face sketch components and the elements of the face sketches helping in the construction of the face sketch.

Creating the face sketch would be a complicated thing if all the face elements are given all together and in an unordered manner making the process difficult for the user and complicated to construct an accurate face which would be against the agenda aimed in the proposed system. So, to overcome this issue we planned on ordering the face elements based on the face category it belongs to like head, nose, hair, eyes, etc. making it much easier for the user to interact with the platform and construct the face sketch. This is available in the column in the left on Canvas on the dashboard click on a face category allows user to get various other face structure.

Coming to the various face elements in a particular face category we could have multiple and n number of elements for a single category, so to solve this our platform would use machine learning in future to predict the similar face elements or predict an suggest the elements to be selected in the face sketch but this would only work once we have appropriate data to train the model on this algorithm and work to enhance the platform.

So, now when the user clicks on a particular face category and then a new module to the right of the canvas opens and lets user to select an element from the option of face elements to construct a face sketch. This option can be selected be selected based on the description provided by the eye witness.

The elements when selected are shown on the canvas and can be moved and placed as per the description of the eye witness to get a better and accurate sketch and the elements have a fixed location and order to be placed on the canvas like the eye elements would be placed over the head element irrespective of the order the were selected. Same for every face element.

The final module is the options to enhance the use of the dashboard, suppose in cases the user selects an element which is not to be selected so that could be rectified using the option to erase that particular element which would be seen when selecting the face category from the left panel. The major important buttons are placed in the panel on the right which has a button to completely erase anything on the canvas of the dashboard making it totally blank.

Then we have a button to save the constructed face sketch, saving the face sketch as a PNG file for better future access. This could be any location on the host pc or on the server depending on the Law Enforcement Department.

2. Face Sketch Recognition: The flowchart illustrates the users flow been followed by the platform to provide an recognize accurate face sketch based on the description, the dashboard is designed simple in order to encourage no professional training to go through before using this platform already saving the timeframe which would have been taken a lot time and resources of the Department.     

The above image demonstrates the first part before using the platform to recognize faces is making the existing records in with the law enforcement department suitable for our platform by training and making the platforms algorithm recognize and assign ids to the face photo to the user in the existing records in with the law enforcement department. For this the platforms algorithms gets connected to the records and breaks each face photo in to various smaller feature and assign an ID to the multiple features generated for a single face photo.

Now, the Module which is majorly designed to be run on the Law enforcements server for security protocols, is been executed where in the user first opens either the hand drawn sketch or the face sketch constructed on our platform saved in the host machine, after which the opened face sketch is been uploaded to the Law enforcements server housing the recognition module so that the process or the data of the record are not tampered and are secure and accurate.

Once the sketch is uploaded on to the server the algorithm first traces the sketch image in order to learn the features in the sketch and map the features as shown in the below figure in order to match those with the features of the face photos in the records.

After mapping the sketch and matching the face sketch with the records and finding a match the platform displays the matched face along with the similarity percentage and other details of the person from the records. The platform displaying all this and the matched person is shown in the below figure.

IV. FUTURE SCOPE

The Project ‘Forensic Face Sketch Construction and Recognition’ is currently designed to work on very few scenarios like on face sketches and matching those sketches with the face photos in the law enforcement records.

The platform can be much enhanced in the future to work with various technologies and scenarios enabling it to explore various media and surveillances medium and get a much wider spread and outputs, The platform can be modified to match the Face sketch with the human faces from the video feeds by using the 3D mapping and imaging techniques and same can be implemented to the CCTV surveillances to perform face recognition on the Live CCTV footage using the Face Sketch.

The platform can further be connected to social media has social media platforms acts has a rich source for data in today’s world, this technique of connecting this platform with the social media platform would enhance the ability of the platform to find a much more accurate match for the face sketch and making the process much more accurate and speeding up the process.

In all the platform could have features which could be different and unique too and easy to upgrade, when compared to related studies on this field, enhancing the overall security and accuracy by standing out among all the related studies and proposed systems in this field.

Conclusion

The Project ‘Forensic Face Sketch Construction and Recognition’ is been designed, developed and finally tested keeping the real-world scenarios from the very first splash screen to the final screen to fetch data from the records keeping security, privacy and accuracy as the key factor in every scenario. The platform displayed a tremendous result on Security point of view by blocking the platform use if the MAC Address and IP Address on load didn’t match the credentials associated with the user in the database and later the OTP system proved its ability to restrict the use of previously generated OTP and even generating the new OTP every time the OTP page is reloaded or the user tries to relog in the platform. The platform even showed good accuracy and speed while face sketch construction and recognition process, provided an average accuracy of more than 90% with a confidence level of 100% when tested with various test cases, test scenario and data sets, which means a very good rate according to related studies on this field. The platform even has features which are different and unique too when compared to related studies on this field, enhancing the overall security and accuracy by standing out among all the related studies and proposed systems in this field.

References

[1] Hamed Kiani Galoogahi and Terence Sim, “Face Sketch Recognition By Local Radon Binary Pattern: LRBP”, 19th IEEE International Conference on Image Processing, 2012. [2] Charlie Frowd, Anna Petkovic, Kamran Nawaz and Yasmeen Bashir, “Automating the Processes Involved in Facial Composite Production and Identification” Symposium on Bio-inspired Learning and Intelligent Systems for Security, 2009. [3] W. Zhang, X. Wang and X. Tang, “Coupled information theoretic encoding for face photo-sketch recognition”, in Proc. of CVPR, pp. 513-520, 2011. [4] X. Tang and X. Wang, “Face sketch recognition”, IEEE Trans. Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 50-57, 2004. [5] B. Klare and A. Jain, “Sketch to photo matching: a featurebased approach”, SPIE Conference on Biometric Technology for Human Identification, 2010. [6] P. Yuen and C. Man, “Human face image searching system using sketches,” IEEE Trans. SMC, Part A: Systems and Humans, vol. 37, pp. 493–504, July 2007. [7] H. Han, B. Klare, K. Bonnen, and A. Jain, “Matching composite sketches to face photos: A component-based approach,” IEEE Trans. on Information Forensics and Security, vol. 8, pp. 191–204, January 2013.

Copyright

Copyright © 2022 Ashutosh Patil, Pranav Tambe, Hrushikesh Dinkar, Prof. Diksha Bhave. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.