Interpreting Doctor Notes using Handwriting Recognition

Abstract

Handwriting recognition of medical prescriptions has been a challenging problem over the recent years with constant research in providing possible accurate solutions. Indecipherable handwritten prescription and inefficiency of Pharmacist to understand the medical prescription can lead to serious and harmful effect to the patients. Even in the recognition of handwriting, mainly doctors notes, they are very difficult for everyone to understand and it takes time for a person to analyse it. So, this idea mainly focused on interpreting doctor’s notes using handwritten recognition and deep learning techniques. The handwritten or printed document pictures are transformed into their electronic counterparts using an optical character recognition (OCR) system. Due to individuals\' inconsistent writing styles, dealing with handwritten texts is significantly more difficult than dealing with printed ones. Handwritten text recognition could be done by Image processing, Machine Learning or Deep Learning Techniques. Out of these Deep Learning remains to be the most popular and prominent. Some of the Deep Learning techniques includes Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). This gives a review of the various recognition methodologies used for interpreting handwritten texts. It includes the most important algorithms that could be used for detecting the handwritten word/text/character by using various approaches for the recognition process. In the end we are thus comparing the accuracies provided by these systems.

Introduction

I. INTRODUCTION

In an era characterized by rapid advancements in healthcare technologies, the digitization of medical records stands as a transformative milestone. The transition from paper-based records to electronic health records (EHRs) has streamlined patient information management. However, the legacy of handwritten medical notes continues to pose a significant challenge in the quest for comprehensive digital healthcare ecosystems. The persistence of handwritten medical notes in healthcare settings is a testament to the trust placed in the insights they provide. Yet, these valuable records, due to their analog nature, remain largely siloed from the digital health ecosystem, limiting their full potential. As healthcare continues to evolve towards data-driven decision-making, the imperative to bridge this gap has never been more critical. Doctors' handwritten notes have been an integral part of healthcare for generations, providing a concise record of clinical observations, diagnoses, treatment plans, and patient histories. These notes often contain critical information that informs medical decisions and patient care. Nevertheless, the interpretation of such handwritten records remains a labor intensive and error-prone endevor. The art of handwriting allows each person to convey their ideas on paper in their own unique way. Depending on the individual, it might vary greatly. Specifically, when talking about a doctor's busy schedule, more consultations are scheduled in a short amount of time, and the diagnosis is given more importance than the prescription's handwriting. As a result, they frequently have poor handwriting, making it sometimes difficult to read the prescription and recognise the drugs and their possible dosages. It is quite challenging for patients and young pharmacists to distinguish the doctor's handwriting. The spectrum of outcomes from drug errors ranges from no apparent symptoms to death. It can sometimes result in a new ailment that is either transient or permanent, such itchiness, rashes, or skin deformity. Despite being rare, drug mistakes can seriously harm individuals. The primary purpose of this study is to create an application that can actively recognise medical prescription images or scan them for subsequent conversion to digital text. This is accomplished by deploying deep learning techniques such as CNN and RNN for image recognition. In order to process the data OCR, word segmentation is used in image processing. Furthermore, a tailored output is provided to offer an optimal summary that most consumers may grasp even if they have no prior knowledge. The construction procedure is far more convenient for average people to use in order to take their daily dosages as prescribed by their doctors. This also makes it easier for new pharmacists and also consumers to conduct their tasks more efficiently and properly. Handwriting recognition is a technology that can be used to interpret and extract information from handwritten documents, such as doctors' notes. This can be a challenging task, as handwriting can vary widely in terms of style, legibility, and formatting.

However, advances in machine learning and OCR technology have made it possible to develop handwriting recognition systems that can accurately interpret a wide range of handwriting styles. There are several approaches that can be used to develop a handwriting recognition system. One approach is to use machine learning algorithms to train a model on a large dataset of handwritten documents. The model can then be used to recognize and interpret handwriting in new documents. Another approach is to use OCR technology to convert handwritten text into digital text, which can then be processed using NLP techniques to extract meaning from the text.

There are several types of classification models used for handwritten character recognition, including:

A. Image Processing: Converting an image using digital technologies and functions to output a magnifying image this way it leads to Image Processing . By this technique we can produce and retrieve necessary information for the future uses. There are input and output data where we can input images, videos or frame and the excepted output can be an image or its characteristics which belong to that image.

B. Machine Learning: Artificial intelligence is a vast sector or technological field thus it subcategory is Machine Learning. Computers need teaching and this can be done from data input and the process with improvise the experience of computers. Technique is used to replace the programming which is explicitly being used. Machine learning includes algorithms that are used to find patterns and interdependencies in huge sets of data this leads to the best outputs and predictions based on previously assumed analysis.

C. Deep Learning: Convolution Neural Network is the full form of CNN. Convolutions allude to complexity and enlargement. Human brains are similar to neural networks. These are planned by taking inspiration from human brain. The cause of CNN is being used for Image classification. Depending on the requirements of technology CNN constitutes of many layers. RNN is an abbreviated form of Recurrent Neural Network that utilises feedback connections. It bases its output on the input received from the previous computation, which is stored in its internal memory. Over the course of several computations, RNNs are able to retain information inside its internal memory and use those in subsequent processes. Hence, the information flows in sequence. A type of handwriting where letters of each word may be connected to each other is used here, it is a necessity to use RNN to get more accurate results of the words.

II. LITERATURE REVIEW

The proposed system uses a combination of image processing techniques and machine learning algorithms to recognize handwritten medicine names from images of prescription notes. The system first applies preprocessing techniques such as image subtraction and noise reduction to improve the quality of the images. Then, it uses a convolutional neural network (CNN) to classify and extract features from the images. Finally, it applies optical character recognition (OCR) to identify the names of the medicines with low accuracy, and compares the results to a dataset of all known medicines to improve accuracy. The system has reportedly achieved an accuracy of 70% when tested on real cases. [1]

The proposed system aims to address the problem of doctors' handwriting being difficult to read, which can lead to misunderstandings and errors when filling prescriptions. The system will use machine learning techniques, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory (LSTM) networks, to recognize and translate handwritten prescription notes written in various languages. The system will be designed as a mobile application that allows users to upload images of prescription notes, which will then be pre-processed and analyzed using deep learning techniques. The system will use Unicode to match words from different languages and will also utilize fuzzy search and market basket analysis to optimize the results from a pharmaceutical database and present them to the user in a structured format. [2]

The proposed Medicine Box system is a mobile application that uses a convolutional neural network (CNN) to recognize and translate handwritten medicine names from prescription notes. The system aims to address the problem of misinterpreting medicine names due to doctors' handwriting being difficult to read, which can lead to errors and harm to patients. The system uses the TensorFlow machine learning library and a custom repository to match partial strings with drug names. By providing a clear and readable digital text of the prescribed medicines, the Medicine Box system aims to decrease instances of misinterpretation and help both patients and pharmacists understand the prescriptions more easily. [3]

According to this system, Handwritten character recognition, especially for medical prescriptions, can be a challenging task due to the poor handwriting of doctors and the lack of domain knowledge among those trying to read the prescriptions. This can lead to misreading of the content, which can have serious consequences for the patient, including health issues and even threats to their lives. In order to address these problems, researchers have been working on developing optical character recognition systems, including using neural networks and knowledge bases, to more accurately read the content of medical prescriptions.

These efforts have shown success in improving the accuracy of prescription identification and can serve as a foundation for further improvements in the field. [4]

The use of artificial intelligence (AI) and machine learning (ML) has the potential to improve efficiency and reduce waiting times in the outpatient pharmacy process. One way that AI and ML can be used in this context is to develop a system that can accurately recognize and interpret handwritten prescriptions, which can be a challenge due to the poor handwriting of doctors and the variety of handwriting styles. By using machine learning to identify different characters for specific medications, it may be possible to improve the accuracy of medication identification and reduce the risk of errors. A mobile application that can read handwritten medicine names and provide a readable textual version of the medication and dose can be a useful tool for both pharmacists and patients. By taking multiple prediction methods into account and analyzing the results of various data sets, it may be possible to develop a machine learning-based framework for healthcare that is effective in improving patient satisfaction and reducing waiting times. [5]

The proposed approach is intended to help recognize handwritten medical prescriptions written by doctors in developing countries, where many doctors are too busy to write digital prescriptions and the majority of prescriptions are handwritten and lack legibility. The proposed approach involves the development of a dataset of handwritten medical terms and the use of data augmentation techniques to improve the recognition efficiency. A bidirectional long shortterm memory (LSTM) network is used to recognize the handwriting, and the results show an average accuracy of 93.0% using the proposed method, which is 19.6% higher than the recognition result without data expansion. The proposed technology could be installed in a smartpen for doctors to recognize and digitize their handwriting in realtime, potentially reducing medical errors, saving medical costs, and promoting healthy living in developing countries.[6]

III. PROPOSED WORK

To address the issue of interpreting handwritten medical prescriptions, we would train a model. The main goal of this model is to provide a solution for both patients and pharmacists to accurately interpret these prescriptions and avoid any potential misunderstandings or errors. In order to achieve this, we implement a machine learning model using python and train it on various data sources, including Kaggle and UCI. We adopt the transformer model, a state-of-the-art approach for text recognition, to accurately identify the medication listed on the prescription. In order to improve the performance of the model, we have also conducted pre-processing on the data using various python libraries.

A. Data Pre-Processing and Training Model

Data Preprocessing: Following data collection, the gathered data will be unsorted so processed before training the model. We all know that if we feed the model garbage data, we will get garbage results. As a result, we must tread cautiously when training the model.

1. Grayscale conversion: To prepare the pictures for model training and drawing a conclusion from them, image preprocessing is performed.Gray scaling is the first step in digital image pre-processing that must be done.
2. Here each pixel's value solely encodes the light's intensity information.
3. Normalization: It is done to resize the picture pixels to a preset range where they will be consistent with the data given. And by doing so, the model will perform common learning, avoiding irregularities in training the model. 
4. Data Augmentation: It is used to make slight changes to a picture in order to deliver a diverse range of data in a single identifiable format. And the procedure frequently includes rotation, cropping, shearing, horizontal and vertical flipping, and so on. Furthermore, by doing so, we can keep the neural network from learning from irrelevant data. And followed by this image stabilization will be performed. 
5. Image standardization: To convert the height and weight of a picture to a common scale, which will convert all of the given data into an appropriate size. We increase the consistency of the given data as well as the quality, which is a priority while completing these activities. Once these processes are in place, there will be an optimal dataset that can be utilised to train the model. As a result, the neural network model will not be trained on irregular data.
6. Training the model: After processing the handwritten text input, we must train the model using structured data. So, in order to get the optimal recognition model, we need to avoid overtraining the model, which will result in the model being suited only to that particular data.

B. Model Building and Prediction

Neural networks are used in every area of the system. CNN and RNN neural networks is used to train the model. The Python libraries TensorFlow and Keras are used for coding. As previously stated, the model has two CNN layers and two RNN layers. 32 and 64 filters are used for the two levels of CNN, respectively. RNN has two layers, each with 64 and 128 filters. The model is trained using these approaches. The SoftMax activation function is used in this technique. Connectionist Temporal Classification (CTC) Loss is used to compute the loss function.

 Furthermore, to avoid errors, check the medical database if the model recognises the few phrases of it or compare the medicine. Following the recognition process, we will additionally utilize market basket analysis and fuzzy search to offer further assistance to the model.

IV. RESULT

The processed text will be given in an organized manner in the application interface, along with a brief explanation of the identified medicines. And this application will be autonomous and simple to use, allowing any user to effortlessly upload an image and receive understandable text in return.

Conclusion

The common areas of failure for OCR systems are hardware limitations that hinders the system from efficiently delivering the results. The users of such systems might have to scan the image by a camera phone with high camera resolution, photos should be captured from an appropriate perspective and zoomed in angle, to get a clear image with its content evident. As for the future work, existing dataset will be augmented with more handwritten medicine names and will be used and trained to obtain higher accuracy. Pre-processing techniques from all the mentioned papers will be implemented. Here,we will analyze and design three models in this research and test using the Pre – trained model dataset, identify handwritten characters. The models were based on algorithms for deep and learning techniques, including support vector machines. The models were compared based on their characteristics, including accuracy and execution time, to determine which was the most accurate. It was found that the CNN model had the highest accuracy for handwritten digit recognition, and that extending the total number of time periods without changing the algorithm\'s parameters did not improve performance due to the limitations of the model. The model also tended to overfit the dataset a certain number of periods later, leading to biased predictions. As a result, it was concluded that the CNN model was the most suitable for predicting image data and other types of prediction problems.

References

[1] Hassan, Esraa, Habiba Tarek, Mai Hazem, Shaza Bahnacy, Lobna Shaheen, and Walaa H. Elashmwai. \"Medical prescription recognition using machine learning.\" In 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0973-0979. IEEE, 2021. [2] Padmanabhan, Sharan, and Nuvvuru Divya. \"Doctors Handwritten Prescription Recognition System In Multi LanguageUsingDeep Learning.\" arXiv preprint arXiv:2210.11666 (2022). [3] Kamalanaban, E., M. Gopinath, and S. Premkumar. \"Medicine box: Doctor’s prescription recognition using deep machine learning.\" International Journal of Engineering and Technology (UAE) 7 (2018): 114-117. [4] Wijewardena, W. R. A. D. \"Medical Prescription Identification Solution.\" PhD diss., 2021. [5] Sandhya, P., and K. P. Rama Prabha. \"Comparison Of Various Machine Learning Algorithms For Recognizing Text On The Medical Prescriptions.\" Journal of Pharmaceutical Negative Results (2022): 2083-2091. [6] Tabassum, Shaira, Nuren Abedin, Md Mahmudur Rahman, Md Moshiur Rahman, Mostafa Taufiq Ahmed, Rafiqul Islam, and Ashir Ahmed. \"An online cursive handwritten medical words recognition system for busy doctors in developing countries for ensuring efficient healthcare service delivery.\" Scientific reports 12, no. 1 (2022): 1-13. [7] Zobeir Raisi, Mohamed A. Naiel, Steven Wardell, Paul Fieguth and John Zelek \" Handwritten Recognition using Transformer for Medical Prescription Application.\" IEEE transactions on transformers, no. 1 (2019): 265-284. [8] Rani, Seerat, Abd Ur Rehman, Beenish Yousaf, Hafiz Tayyab Rauf, Emad Abouel Nasr, and Seifedine Kadry. \"Recognition of Handwritten Medical Prescription Using Signature Verification Techniques.\" [9] T. B. Sheridan and J. M. Thompson, “People versus computers in medicine,” in Human Error in Medicine, pp. 141–158, CRC Press, 2018.

Copyright

Copyright © 2024 Prof. A.V. Sable, Avantika Patil, Mayur Rathi, Ayush Shriwas. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.