Forensic Face Sketch Artist System

Abstract

In this modern age, the general crime rate is increasing day-by-day and to cope up with this the enforcement departments too should find ways in which would speed up the overall process and help them in bringing one to justice. One such way can be using face recognition technology for identifying and verifying the criminal. The traditional approach here is to use the hand-drawn face sketches drawn by forensic sketch artist to spot the criminal, modernizing this is able to mean using the hand-drawn sketch and then matching them with the enforcement departments database to spot the criminal. Using this approach would result in the various limitations with latest technologies and even would be time consuming as there are very few forensic sketch artists available when compared to the increasing crime ratio. Our project is aimed on decreasing the time span and speeding up this process by providing a standalone platform to the law enforcement department which would allow users to create accurate face sketch of the suspect without the help of forensic sketch artist and no special training or artistic skills. The sketch can be created using drag and drop feature in the application with variety of face elements and can automatically match the drawn composite face sketch with the law enforcement departments database much faster and efficiently using deep learning and cloud infrastructure.

Introduction

I. INTRODUCTION

A criminal can be easily identified and brought to justice using a face sketch drawn based on the information been provided by the eye-witness, however in this world of modernization the traditional way of hand drawing a sketch is not found to be that effective and time saving when used for matching and identifying from the already available database or real-time databases.

During the past there were several techniques been proposed to convert hand-drawn face sketches and use them to automatically identify and recognize the suspect from the police database, but these techniques could not provide the desired precise results. Application to make a composite face sketches were even introduced which too had various limitations like limited facial features kit, cartoonistic feel to the created suspect face which made it much harder to use these applications and get the specified results and efficiency.

The above applications and needs motivated us into thinking of creating an application which would not just provide a set of individual features like eyes, ears, mouth, etc. to be selected to create a face sketch but also would allow user to upload hand-drawn individual features on the platform which would then be converted in to the applications component set. This in turn would make the created sketch much more similar to the hand-drawn sketch and would be much easier for the law enforcement departments to adapt the application.

Our application would even allow the law enforcement team to upload a previous handdrawn sketch in order to use the platform to identify and recognize the suspect using the much more efficient deep learning algorithm and cloud infrastructure provided by the application.

The machine learning algorithm would learn from the sketches and the database in order to suggest the user all the relatable facial features that could be used with a single selected feature in order the decrease the time frame and increase the efficiency of the platform.

Overview

This is a Forensic application, allowing user to construct accurate composite face sketch using the predefined facial feature sets provided as tools that can be resized and repositioned as per requirement/described by the eye-witness.

Moreover, the constructed composite face sketch can then be matched with the law enforcement departments database using deep learning and the speed and efficiency of cloud infrastructure to identify and verify the criminal.

The same process can even be done with the hand-drawn sketch making the application backward compatible with traditional approaches.

II. REVIEW OF LITERATURE

There are lot of studies on face sketch construction and recognition using various approaches. Dr. Charlie Frowd along side Yasmeen Bashir, Kamran Nawaz and Anna Petkovic designed a standalone application for constructing and identifying the facial composites, the initial system was found to be time consuming and confusing as the traditional method, later switching to a new approach in which the victim was given option of faces and was made to selected similar face resembling the suspect and at the end the system would combine all the selected face and try to predict automatically the criminal?s facial composite. The Results where promising and 10 out of 12 composite faces where named correctly out of which the results 21.3% when the witness was helped by the department person to construct the faces and 17.1% when the witness tried constructing faces by themselves.

Xiaoou Tang and Xiaogang Wang proposed a recognition method of photo-sketch synthesized using a Multiscale Markov Random Field Model the project could synthesis a give sketch into photo or a given photo in to sketch and then search the database for a relevant match for this the model divided the face sketch in to patches. In this they first synthesized the available photos in to sketch and then trained the model making the model to decrease the difference between photos and sketch this enhanced the overall efficiency of the recognition model. For testing this they took few samples in which the photos where synthesized in to sketch and the same faces where drawn from sketch artist and then the model was trained from 60% data and remaining 40% data for testing the model. The overall results where impressive but not up to the mark needless to say.

Another proposed method was sketch to photo matching proposed by Anil K Jain and Brendan Klare which used SIFT Descriptor, the method proposed displayed result based on the measured SIFT Descriptor distance between the face photos in the database and the sketches. The algorithm first converts the face photos using linear transformation which was based on Tang and Wang proposed model and then the sketch was used to measure the SIFT descriptor distance compared to the face photo and in some cases distance between images in the databases too where measured for better accuracy. The experimental result shows that the dataset used where very similar to the those usedby Tang in their experiment and the addition in the algorithm was the measurement of the descriptor which gave a better result and accuracy from the model proposed by Tang and Wang.

P. C. Yuen and C. H. Man too proposed a method to search human faces using sketches, this method converted sketches to mug shots and then matched those mugshots to faces using some local and global variables been declared by the face matching algorithms. However, in some cases the mugshots where hard to be matched with the human faces in the databases like FERET Database and Japanese Database. The proposed method showed an accuracy of about 70% in the experimental results, which was fair decent but still lacked the accuracy needed by the law enforcement department.

The common issue with all the proposed algorithm where that they compared the face sketches with human face which were usually front facing making it easier to be mapped both in drawn sketch and human face photograph, but when a photograph or sketch collected had their faces in different direction the algorithms were less likely to map it and match with a face from the database which is front facing.

There are even system been proposed for composite face construction but most system used facial features which where been taken from photographs and then been selected by the operator as per described by the witness and at last complied to form a single human face making it much more complicated for human as well as any algorithm to match it with a criminal face as every facial feature was been taken from the separate face photograph having various dissimilarity and when combined together made it harder to recognize.

Thus, all the previous approaches proved either inefficient or time consuming and complicated. Our application as mentioned above would not only overcome the limitations of the mentioned proposed techniques but would also fill in the gap between the traditional handdrawn face sketch technique and new modernized composite face sketch technique by letting user to upload the hand-drawn face sketches and facial features.

III. REPORT ON PRESENT INVESTIGATION

A. Requirement Analysis

1. Scope: The Project „Forensic Face Sketch Construction and Recognition? is currently designed to work on very few scenarios like on face sketches and matching those sketches with the face photos in the law enforcement records. The platform can be much enhanced in the future to work with various technologies and scenarios enabling it to explore various media and surveillances medium and get a much wider spread and outputs, The platform can be modified to match the Face sketch with the human faces from the video feeds by using the 3D mapping and imaging techniques and same can be implemented to the CCTV surveillances to perform face recognition on the Live CCTV footage using the Face Sketch. The platform can further be connected to social media has social media platforms acts has a rich source for data in today?s world, this technique of connecting this platform with the social media platform would enhance the ability of the platform to find a much more accurate match for the face sketch and making the process much more accurate and speeding up the process.
2. Feasibility Study: The overall crime rate is increasing day-by-day and to cope up with this the law enforcement departments too should find ways that would speed up the overall process and help them in bringing one to justice. One such way can be using face recognition technology for identifying and verifying the criminal. The traditional approach here is to use the hand-drawn face sketches drawn by forensic sketch artist to identify the criminal, modernizing this would mean using the hand-drawn sketch and then matching them with the law enforcement departments database to identify the criminal. Using this approach would result in the various limitations with latest technologies and even would be time consuming as there are very few forensic sketch artists available when compared to the increasing crime ratio. Thus, there is a need for creating an application which would not just provide a set of individual features like eyes, ears, mouth, etc.
3. Hardware Requirements

This application is been designed to run on the minimum possible configuration of hardware.

a. Client/ Node Machine

• Processor: Intel Dual Core CPU and above
• RAM: 1 GB and above
• Hard Disk: 250GB and above

b. Server Machine:

• Processor: Intel Core i3 CPU and above
• RAM: 4 GB and above
• Hard Disk: 1 TB and above

4. Software Requirements

This application is been designed to run as a desktop application with part of the data saved on server for security purpose.

a. Client/ Node Machine

• Operating System: Windows 7 and above
• Framework: Java JDK
• Cloud: Amazon Web Services CLI

b. Server Machine

• Operating System: Windows Desktop OS or Windows Server Edition
• Framework: Java JDK
• Cloud: Amazon Web Services CLI
• Database: SQLite

B. Security and Privacy

The major concern of the law enforcement department before adapting any system is security and privacy. Keeping this in mind the application is designed to protect the privacy and carry out the security measures in the following ways.

1. Machine Locking: The Machine locking technique would ensure that the application once installed on a system could not be tampered and could not been operated on any other system, for which the application uses two locking parameters i.e. one software and one hardware locking parameter.

      HD ID – Volume serial of hard-drive with OS

      NET ID – Hardware ID – MAC Address.

2. Centralized Usage: The system which has the application been installed would be connected to a centralized server of the law enforcement department campus containing the database and the other important feature set of the application, thus the application could not be operated once disconnected from the server.

C. Problem Statement

In this modern age, the overall crime rate is increasing day-by-day and to cope up with this the law enforcement departments too should find ways that would speed up the overall process and help them in bringing one to justice. One such way can be using face recognition technology for identifying and verifying the criminal.

The traditional approach here is to use the hand-drawn face sketches drawn by forensic sketch artist to identify the criminal, modernizing this would mean using the hand-drawn sketch and then matching them with the law enforcement departments database to identify the criminal.

Using this approach would result in the various limitations with latest technologies and even would be time consuming as there are very few forensic sketch artists available when compared to the increasing crime ratio.

Thus, there is a need for creating an application which would not just provide a set of individual features like eyes, ears, mouth, etc. to be selected to create a face sketch that would help in finding the criminal much faster and efficiently.

D. Project Design

1. System Flow: Our Application would be majorly used by the Law Enforcement Departments in order to reduce the overall time required to bring the criminal to justice and even to enhance the workforce and speed up the system by keeping accuracy in mind.

So, keeping this scenario in mind the platform is designed to be as simple as possible in order to make sure that a user can create a sketch in the application without a formal training. The above flowchart represents the overall flow of the system starting with the login page to the actual results been displayed after the sketch is been matched by the records in the database.

The privacy and security are been kept in mind from the very first stage itself starting with the login page itself, the login page consists of two parts. At the start the login page fetches the Mac Address along with IP Address and HDD ID which is then been matched with the data been collected while installing the platform in the host machine and if the data does not match the platform would lock itself and won?t allow the user to move further and use any feature of the platform. This would make sure that the platform could not be accessed when the host machine is been tampered or the hard- disk is been tampered to be used in other machine making it more secure and much more reliable than any other platform currently available. After the secure login on to the platform and moving further the platform uses something called as Backward Compatibility, this feature is been introduced in order to make a smooth transition from the current technique on to the new platform. The current technique been the use of hand drawn sketch been drawn by an expert forensic artist with years of experience and then the sketch been used by the law enforcement department to be showed on to various platforms in order to create a sense of awareness in people in order to find someone to recognize the suspect. So backward compatibility allows the law enforcement department to upload those hand drawn sketches on to the platform in order to use our face recognition module and match the suspect sketch with the large record and reducing the overall time and the efforts used in the previous age-old technique. If the law enforcement department doesn?t have a hand drawn sketch and the law enforce department would wish to use the platform for creating a face sketch using our platform, they can access the canvas where they would find a wide range of facial elements in the database. The elements can be easily selected to create a described face sketch of the suspect and use the feature like drag and drop in order to arrange the elements according to the eye witnesses description. The platform is designed in such a way that one can use the platform without a prior professional training and knowledge of sketching. The user thus can select the main face category he/she wishes to select and would then prompt with a variety of option under that particular face category and then can select one feature based on the description provided by the suspect. The platform even would allow the user to change a selected feature to be replaced by any other feature if it does not match the description even after selection.

The selected face categories would be placed one another to create a complete face sketch and can be moved on using the mouse for placing that face feature on to another spot based on the description been provided by the eye witness. This canvas can then be saved as JPG format image in order to further use the image in possible medium other than out platform like sharing on social media or for printing purpose. Once the sketch is created the platform gives access to the face prediction module, where in the database of all the criminals until now in been saved on the data centers for maintaining a level of security and for this purpose the sketch too is been uploaded to the data center first and then the prediction is been performed on the cloud for security purpose.

Our platform uses deep learning alongside with Amazon Web Services (AWS) in order to give the best and accurate result so as to bring the criminal to justice.

The prediction module divides the screen in to four parts, first the sketch to be predicted is been uploaded to the data centers for security purposes and the second part is the match found in the database followed with the third part which is the accuracy been shown in the predicted/match images and lastly the forth part is called the meta which can be customized in order to show the data about the match as per need and then can be exported and shared with other if required.

2. Face Sketch Construction Module: As mentioned earlier, security and accuracy are the key features been focused while developing our platform for the law enforcement department. So, this module of the project mainly focuses on creating a face sketch based on the description been provided by the Eye Witness to the Law enforcement department.

The above flowchart illustrates the users flow been followed by the platform to provide an construct accurate face sketch based on the description, the dashboard is designed simple in order to encourage no professional training to go through before using this platform already saving the timeframe which would have been taken a lot time and resources of the Department. Keeping it simple thus ensures that the user doesn?t have to be a professional sketch artist from the forensic department rather any one from the law enforcement department using the descriptions narrated by the eye witness or in some cases the eye witness too can take control of the platform but that would not be recommended as it can tamper the  security protocols. Moving further  the  dashboard consists of  Five  main modules, First the important module is the Canvas been shown at the middle of the dashboard which would house the face sketch components and the elements of the face sketches helping in the construction of the face sketch.

3. Face Sketch Recognition Module: As mentioned earlier, security and accuracy are the key features been focused while     developing our platform for the law enforcement department. So, this module of the project mainly focuses on creating a face sketch based on the description been provided by the Eye Witness to the Law enforcement department.

The above flowchart illustrates the users flow been followed by the platform to provide an recognize accurate face sketch based on the description, the dashboard is designed simple in order to encourage no professional training to go through before using this platform already saving the timeframe which would have been taken a lot time and resources of the Department.

Keeping it simple thus ensures that the user doesn?t have to be a professional sketch artist from the forensic department rather any one from the law enforcement department using the descriptions narrated by the eye witness or in some cases the eye witness too can take control of the platform but that would not be recommended as it can tamper the security protocols.

The above image demonstrates the first part before using the platform to recognize faces is making the existing records in with the law enforcement department suitable for our platform by training and making the platforms algorithm recognize and assign IDs to the face photo to the user in the existing records in with the law enforcement department.

For this the platforms algorithms gets connected to the records and breaks each face photo in to various smaller feature and assign an ID to the multiple features generated for a single face photo. Now, the Module which is majorly designed to be run on the Law enforcements server for security protocols, is been executed where in the user first opens either the hand drawn sketch or the face sketch constructed on our platform saved in the host machine, after which the opened face sketch is been uploaded to the Law enforcements server housing the recognition module so that the process or the data of the record are not tampered and are secure and accurate.

Once the sketch is uploaded on to the server the algorithm first traces the sketch image in order to learn the features in the sketch and map the features as shown in the below figure in order to match those with the features of the face photos in the records.

E.  Methodology

Technology Stack

Our platform was designed and developed using various technology stack in order to provide the law enforcement department with state-of-the-art security features and accuracy which in turn provide the law enforcement department with a better crime solving rate and efficiency.

1. Machine locking: The Machine locking technique would ensure that the application once installed on a system could not be tampered and could not been operated on any other system, for which the application uses two locking parameters i.e. one software and one hardware locking parameter.

       HD ID – Volume serial of hard-drive with OS           

       NET ID – Hardware ID – MAC Address.

2. JAVA

a. Java offers higher cross- functionality and portability as programs written in one platform can run across desktops, mobiles, embedded systems.

b. Java is free, simple, object-oriented, distributed, supports multithreading and offers multimedia and network support.

c. Java is a mature language, therefore more stable and predictable. The Java Class Library enables cross-platform development.

d. Being highly popular at enterprise, embedded and network level, Java has a large active user community and support available.

e. Unlike C and C++, Java programs are compiled independent of platform in bytecode language which allows the same program to run on any machine that has a JVM installed.

f. Java has powerful development tools like Eclipse SDK and NetBeans which have debugging capability and offer integrated development environment.

g. Increasing language diversity, evidenced by compatibility of Java with Scala, Groovy, JRuby, and Clojure.

h .Relatively seamless forward compatibility from one version to the next

In conclusion, almost 20 years after its inception, Java continues to deliver considerable value to the world of software development. Java 8, in fact, offers new features such as a scalable and flexible platform for the Internet of Things, less boilerplate code, new date and time library and API, refreshed graphics toolkit, integration with JavaScript, and others.

3. JAVAFX: JavaFX is a set of graphics and media packages that enables developers to design, create, test, debug, and deploy rich client applications that operate consistently across diverse platforms. Written as a Java API, JavaFX application code can reference APIs from any Java library. For example, JavaFX applications can use Java API libraries to access native system capabilities and connect to server- based middleware applications. The look and feel of JavaFX applications can be customized. Cascading Style Sheets (CSS) separate appearance and style from implementation so that developers can concentrate on coding. Graphic designers can easily customize the appearance and style of the application through the CSS. If you have a web design background, or if you would like to separate the user interface (UI) and the back-end logic, then you can develop the presentation aspects of the UI in the FXML scripting language and use Java code for the application logic. If you prefer to design UIs without writing code, then use JavaFX Scene Builder. As you design the UI, Scene Builder creates FXML markup that can be ported to an Integrated Development Environment (IDE) so that developers can add the business logic.With JavaFX, you can build many types of applications. Typically, they are network-aware applications that are deployed across multiple platforms and display information in a highperformance modern user interface that features audio, video, graphics, and animation.

4. AWS (Amazon Web Services): Amazon Web Services (AWS) is a subsidiary of Amazon that provides on-demand cloud computing platforms and APIs to individuals, companies, and governments, on a metered pay-as-you-go basis. In aggregate, these cloud computing web services provide a set of primitive abstract technical infrastructure and distributed computing building blocks and tools. One of these services is Amazon Elastic Compute Cloud (EC2), which allows users to have at their disposal a virtual cluster of computers, available all the time, through the Internet. AWS's version of virtual computers emulates most of the attributes of a real computer, including hardware central processing units (CPUs) and graphics processing units (GPUs) for processing; local/RAM memory; hard-disk/SSD storage; a choice of operating systems; networking; and pre-loaded application software such as web servers, databases, and customer relationship management (CRM).The AWS technology is implemented at server farms throughout the world, and maintained by the Amazon subsidiary. Fees are based on a combination of usage (known as a "Pay-as-you-go" model), the hardware/OS/software/networking features chosen by the subscriber, required availability, redundancy, security, and service options. Subscribers can pay for a single virtual AWS computer, a dedicated physical computer, or clusters of either. As part of the subscription agreement, Amazon provides security for subscribers' systems. AWS operates from many global geographical regions including 6 in North America.

In 2020, AWS comprised more than 212 services including computing, storage, networking, database, analytics, application services, deployment, management, mobile, developer tools, and tools for the Internet of Things. The most popular include EC2 and Amazon Simple Storage Service (Amazon S3). Most services are not exposed directly to end users, but instead offer functionality through APIs for developers to use in their applications. Amazon Web Services' offerings are accessed over HTTP, using the REST architectural style and SOAP protocol for older APIs and exclusively JSON for newer ones.

Amazon markets AWS to subscribers as a way of obtaining large scale computing capacity more quickly and cheaply than building an actual physical server farm. All services are billed based on usage, but each service measures usage in varying ways. As of 2017, AWS owns a dominant 34% of

all cloud (IaaS, PaaS) while the next three competitors Microsoft, Google, and IBM have 11%, 8%, 6% respectively according to Synergy Group.

5. Centralized Computing (Aws For Now): Centralized computing is computing done at a central location, using terminals that are attached to a central computer. The computer itself may control all the peripherals directly (if they are physically connected to the central computer), or they may be attached via a terminal server. Alternatively, if the terminals have the capability, they may be able to connect to the central computer over the network.

The terminals may be text terminals or thin clients, for example.

It offers greater security over decentralized systems because all of the processing is controlled in a central location. In addition, if one terminal breaks down, the user can simply go to another terminal and log in again, and all of their files will still be accessible.

Depending on the system, they may even be able to resume their session from the point they were at before, as if nothing had happened. This type of arrangement does have some disadvantages. The central computer performs the computing functions and controls the remote terminals. This type of system relies totally on the central computer. Should the central computer crash, the entire system will "go down" (i.e. will be unavailable). Another disadvantage is that central computing relies heavily on the quality of administration and resources provided to its users. Should the central computer be inadequately supported by any means (e.g. size of home directories, problems regarding administration), then your usage will suffer greatly. The reverse situation, however, (i.e., a system supported better than your needs) is one of the key advantages to centralized computing.

6. Deep Learning For Face Recognition: Face recognition is the problem of identifying and verifying people in a photograph by their face. It is a task that is trivially performed by humans, even under varying light and when faces are changed by age or obstructed with accessories and facial hair. Nevertheless, it is remained a challenging computer vision problem for decades until recently. Deep learning methods are able to leverage very large datasets of faces and learn rich and compact representations of faces, allowing modern models to first perform as-well and later to outperform the face recognition capabilities of humans. Generally, we refer to this as the problem of automatic “face recognition” and it may apply to both still photographs or faces in streams of video. Humans can perform this task very easily. We can find the faces in an image and comment as to who the people are, if they are known. We can do this very well, such as when the people have aged, are wearing sunglasses, have different colored hair, are looking in different directions, and so on. We can do this so well that we find faces where there aren?t any, such as in clouds. Nevertheless, this remains a hard problem to perform automatically with software, even after 60 or more years of research. Until perhaps very recently.

CNN Algorithm

All facial recognition and detection systems require the use face datasets for training and testing purposes. In particular, the accuracy of CNNs is highly dependent on large training datasets. For example, the development of very large datasets such as ImageNet, which contains over 14 million images, has allowed the development of accurate deep learning object detection systems. More specifically, face detection and recognition datasets developed alongside benchmarks such as the MegaFace Challenge, the Face Detection Dataset and Benchmark (FDDB) dataset and the Labeled Faces in the Wild (LFW) dataset provide a means to test and rank face detection, verification and recognition systems using real-life, highly challenging images in unconstrained settings. Notable and widely used datasets are listed in Table, along with information regarding their intended usage, size and the number of identities they contain. Upon analysis of the results attained by face verification and identification algorithms tested on small datasets such as the LFW dataset, one may be led to believe there remains little scope for improvement. This is far from true: when tested on millions of images, algorithms achieving impressive results on smaller testing sets produce far from ideal accuracies. The MegaFace Challenge was created in response to the saturation of small datasets and benchmarks, providing a large-scale public database and benchmark which requires all algorithms to be trained on the same data and tested on millions of images, allowing fair comparison of algorithms without the bias of private dataset usage. This addresses the problem of lack of reproducibility of results caused by the usage of private databases for training by state-of-the-art CNN methods. Although a shortage of cross-age identity sets is one limitation of the MegaFace dataset, results thus far have indicated there is ample scope for algorithm improvement, with the highest identification and verification accuracies attained by the stateof-the-art method ArcFace reaching 82.55%, and 98.33% respectively. Similarly, the MSCeleb-1M database was created to provide both training and testing data, to enable the comparison of face recognition techniques by use of a fixed benchmark. However, despite the benefits conferred by their size, both MegaFace and MS-Celeb-1M are disadvantaged by annotation issues and long tail distributions. Face detection is a fundamental step in facial recognition and verification. It also extends to a broad range of other applications including facial expression recognition, face tracking for surveillance purposes, digital tagging on social media platforms and consumer applications in digital technologies, such as auto-focusing ability in phone cameras. This survey will examine facial detection methods as applied to facial recognition and verification.  Historically, the greatest obstacle faced by face detection algorithms was the ability to achieve high accuracy in uncontrolled conditions. Consequently, their usability in real life applications was limited. However, since the development of the Viola Jones boosting based face detection method, face detection in real life settings has become commonplace. Significant progress has since been made by researchers in this area due to the development of powerful feature extraction techniques including Scale Invariant Feature Transform (SIFT), Histograms of oriented Gradients (HoGs), Local Binary Patterns (LBPs) and methods such as Integral Channel Features (ICF). For a recent and comprehensive review of these traditional face detection methodologies, readers are referred to.

This review will alternatively focus on more recently proposed deep learning methods, which were developed in response to the limitations of HoG and Haar wavelet features in capturing salient facial information under unconstrained conditions which include large variations in resolution, illumination, pose, expression, and color.

 Essentially, it is the limitations of these feature representations which have thus far limited the ability of classifiers to perform to the best of their ability. Feature extraction usually occurs immediately after face detection and can be considered as one of the most important stages in face recognition systems, as their effectiveness is dependent upon the quality of the extracted features. This is because facial landmarks and fiducial points identified by a given network determine how accurately features are represented. Traditional fiducial point locators are model-based, whilst many recent methods are cascaded regression based. Lately, key improvements have been made with the development of deep dual pathway methods, and other confidence map-based solutions, such as and Traditional model-based fiducial point methodologies include Active Shape Model (ASM), which suffers from low accuracy, partially rectified by the work of, Active Appearance Model (AAM), and Constrained Local Models (CLM). CLMs are generally outperformed by cascaded regression, models due to the latter?s inherent inability to mode the complex variation of local feature appearances. It must be noted however, that highly effective methods based on CLMs have been developed. For example, is based on CLMs but takes advantage of the neural network architecture, proposing a Convolutional Experts Network (CEN) and Convolutional Experts Constrained Local Model (CE-CLM) which uses CLM as local detector, achieving very competitive results particularly on profile images. Subsequent to feature extraction, facial recognition is performed. Recognition can be categorized as either verification or identification. Modern face recognition systems using DCNNs involve deep feature extraction, and lastly, similarity comparison. More specifically, verification involves comparison of one-to-one similarity between a probe image and a gallery of a known identity, whilst identification determines one to many similarities to determine the identity of the probe. Both these processes require robust feature representation, and a discriminative classification model or similarity measure. Traditional methods used for feature representation include LBP, HoGs, and Fisher Vector. Relevant metric learning methods include cosine metric learning, Mahalanobis metric learning, and one-shot similarity kernel. Others include large margin nearest neighbor, Joint Bayesian and attribute-based classifiers. These methods are thoroughly reviewed by. Thus, for the sake of relevance and context, we have only included a brief overview of the role these methods play in modern face recognition and have chosen to focus on the most recently developed state of the art methodologies, which largely rely on DCNNs. Since then, research has accelerated significantly, leading to the development of highly sophisticated DCCNs capable of detection, recognition and verification with accuracy approaches that of humans. Although the development of CNNs was impeded by lack of computing power, recent hardware advances have allowed rapid improvement and a significant increase in CNN depth, and consequently, accuracy. One outstanding feature is an increase in depth, and width to allow for improved feature representation by improving non-linearity. However, this leads to issues such as reduction in efficiency and overfitting. This section will explore the various methods which have aimed to address these problems in the context of facial recognition, through an examination of general improvements in DCCN architecture and loss functions. CCNs are generally more suitable to object recognition than standard feedforward neural networks of similar size due to the use of fewer connections and parameters which facilitates training and efficiency, with only slight reduction in performance. CNNs were designed specifically for classification of 2D images due to their invariance to translation, rotation and scaling. A CNN is comprised of a set of layers, including convolutional layers, which are a collection of filters with values known as weights, non- linear scalar operator layers, and down sampling layers, such as pooling. Activation values are the output of individual layers which are used as input in the next layer. The use of CNNs in facial recognition tasks is comprised of two essential steps; namely, training and inference. Training is a global optimization process which involves learning of parameters via observation of huge datasets. Inference essentially involves the deployment of a trained CNN to classify observed data. The training process involves minimization of the loss function to establish the most appropriate parameters, and determination of the number of layers required, the task performed by each layer, and networking between layers, where each layer is defined by weights, which control computation. CNN face recognition systems can be distinguished in three ways; the training data used to train the model, the network architecture and settings, and the loss function design. DCNN?s have the capacity to learn highly discriminative and invariant feature representations, if trained with very large datasets. Training is achieved using an activation function, loss function and optimization algorithm. The role of the loss function is to determine the error in the prediction. Different loss functions will output different error values for an identical prediction, and thus determine to a large extent the performance of the network. Loss function type depends on the type of problem, e.g. regression or classification.   Minimization of the error is achieved using back propagation of the error to a previous layer, whereby the weights and bias are modified. Weights are learned and modified using an optimization function, such as stochastic gradient descent, which calculates the gradient of the loss function with respect to weights, then modifies weights to reduce the gradient of the loss function

7. Backward Compatibility: The major drawback in adapting any new system is the complication been involved in completing migrating from the previous technique to the new technique, Hence resulting in the wastage of time resources. To overcome this issue, we have designed our application in such a way that even the hand-drawn sketches can be uploaded and the user can use the deep learning algorithms and cloud infrastructure to identify and recognize the criminal using the hand-drawn sketch.

8. Face Sketch Construction using Drag and Drop:  In this application, accurate composite face sketch can be constructed using the predefined facial feature sets provided as tools allowing to be resized and repositioned as per requirement/described by the eye-witness. Here, the human face is be categorized into various facial features such as head, eyes, eyebrow, lips, nose, ears, etc. and some important wearable components such as hats, specs, etc. too are been available in the application for use. Every facial feature when selected would open a wide range of options to choose from based on the requirement/description of the eye-witness. The machine learning algorithm would learn and in future try to suggest all the facial features which could suit the single selected feature and would try to help in completing the composite face sketch much sooner and much efficiently.

Such are the facial features which can be used in the application to create the composite face sketch of the suspect based on the description been provided by the eye-witness to the law enforcement and forensic department.

F. mplementation Plan

1. Semester VII

Fig 4.1.24. Police Record with Face Images Details (Face Images Details Stored in the Server)

Conclusion

The Project „Forensic Face Sketch Construction and Recognition? is been designed, developed and finally tested keeping the real-world scenarios from the very first splash screen to the final screen to fetch data from the records keeping security, privacy and accuracy as the key factor in every scenario. The platform displayed a tremendous result on Security point of view by blocking the platform use if the MAC Address and IP Address on load didn?t match the credentials associated with the user in the database and later the OTP system proved its ability to restrict the use of previously generated OTP and even generating the new OTP every time the OTP page is reloaded or the user tries to relog in the platform. The platform even showed good accuracy and speed while face sketch construction and recognition process, provided an average accuracy of more than 90% with a confidence level of 100% when tested with various test cases, test scenario and data sets, which means a very good rate according to related studies on this field. The platform even has features which are different and unique too when compared to related studies on this field, enhancing the overall security and accuracy by standing out among all the related studies and proposed systems in this field.

References

[1] Hamed Kiani Galoogahi and Terence Sim, “Face Sketch Recognition By Local Radon Binary Pattern: LRBP”, 19th IEEE International Conference on Image Processing, 2012. [2] Charlie Frowd, Anna Petkovic, Kamran Nawaz and Yasmeen Bashir, “Automating the Processes Involved in Facial Composite Production and Identification” Symposium on Bio-inspired Learning and Intelligent Systems for Security, 2009. [3] FACES 4.0, IQ Biometrics, http://www.iqbiometrix.com. [4] W. Zhang, X. Wang and X. Tang, “Coupled information theoretic encoding for face photo-sketch recognition”, in Proc. of CVPR, pp. 513-520, 2011. [5] X. Tang and X. Wang, “Face sketch synthesis and recognition”, in Proc. of ECCV, pp. 687-694, 2003. [6] X. Tang and X. Wang, “Face sketch recognition”, IEEE Trans. Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 50-57, 2004. [7] B. Klare and A. Jain, “Sketch to photo matching: a featurebased approach”, SPIE Conference on Biometric Technology for Human Identification, 2010. [8] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A nonlinear approach for face sketch synthesis and recognition,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1005–1010, June 2005. [9] P. Yuen and C. Man, “Human face image searching system using sketches,” IEEE Trans. SMC, Part A: Systems and Humans, vol. 37, pp. 493–504, July 2007. [10] H. Han, B. Klare, K. Bonnen, and A. Jain, “Matching composite sketches to face photos: A component-based approach,” IEEE Trans. on Information Forensics and Security, vol. 8, pp. 191–204, January 2013. [11] FaceVACS Software Developer Kit v. 8.2, Cognitec Systems GmbH, http://www.cognitec-systems.de. [12] Identi-Kit, Identi-Kit Solutions, www.identikit.net. [13] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 5967–5976. [14] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2242–2251. [15] Y. Song, J. Zhang, L. Bao, and Q. Yang, “Fast preprocessing for robust face sketch synthesis,” in Proc. 26th Int. Joint Conf. Artif. Intell., 2017, pp. 4530–4536. [16] Y. C. Lai, B. A. Chen, K. W. Chen, W. L. Si, C. Y. Yao, and E. Zhang, “Data-driven npr illustrations of natural flows in chinese painting,” IEEE Trans. Vis. Comput. Graph., vol. 23, no. 12, pp. 2535–2549, Dec.2017. [17] F.-L. Zhang, J. Wang, E. Shechtman, Z.-Y. Zhou, J.-X. Shi, and S. M. Hu, “PlenoPatch: Patch-based plenoptic image manipulation,” IEEE Trans. Vis. Comput. Graph., vol. 23, no. 5, pp. 1561–1573, May2017. [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, 2017. [19] M. Zhu, N. Wang, X. Gao, and J. Li, “Deep graphical feature learning for face sketch synthesis,” in Proc. 26th Int. Joint Conf. Artif. Intell., 2017, pp. 3574–3580. [20] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A nonlinear approach for face sketch synthesis and recognition,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, pp. 1005–1010. [21] N. Wang, X. Gao, L. Sun, and J. Li, “Bayesian face sketch synthesis,” IEEE Trans. Image Process., vol. 26, no. 3, pp. 1264–1274, Mar.2017. [22] Y. Song, L. Bao, S. He, Q. Yang, and M.-H. Yang, “Stylizing face images via multiple exemplars,” Comput. Vis. Image Understanding, vol. 162, pp. 135–145, 2017. [23] N. Wang, X. Gao, and J. Li, “Random sampling for fast face sketch synthesis,” Pattern Recognit., vol. 76, pp. 215–227, 2018. [24] Y. J. Huang, W. C. Lin, I. C. Yeh, and T. Y. Lee, “Geometric and textural blending for 3d model stylization,” IEEE Trans. Vis. Comput. Graph., vol. 24, no. 2, pp. 1114– 1126, Feb.2018. [25] S. S. Lin, C. C. Morace, C. H. Lin, L. F. Hsu, and T. Y. Lee, “Generation of escher arts with dual perception,” IEEE Trans. Vis. Comput. Graph., vol. 24, no. 2, pp. 1103–1113, Feb.2018. [26] N. Wang, X. Gao, and J. Li, “Random sampling for fast face sketch synthesis,” Pattern Recognit., vol. 76, pp. 215–227, 2018 [27] Bin Sheng, Ping Li, Chenhao Gao, Kwan-Liu Ma, \"Deep Neural Representation Guided Face Sketch Synthesis\", IEEE Trans. Vis. Comput. Graph., vol. 25, no. 12, pp. 3216- 3230, Dec.2019

Copyright

Copyright © 2022 Srujan Mahajan, Vipul Humbe, Advait Raorane, Asmita Deshmukh. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.