Analysis of Physiotherapy Practices using Deep Learning

Abstract

The proposed physiotherapy assessment system, utilizing deep learning, aims to enhance the accuracy and efficiency of assessments. Traditional manual methods used by physiotherapists are often time-consuming and prone to errors, potentially leading to incorrect diagnoses and treatment plans. This system tackles these challenges by employing advanced deep learning algorithms to detect angles and provide personalized audio feedback to patients based on their posture. The system commences by capturing the patient\'s video with a webcam and extracting frames using OpenCV. These frames are then analyzed through a media pipe library, which identifies key body points. These points are utilized to connect relevant body parts for specific exercises, calculating angles between them. Subsequently, the system evaluates posture correctness and delivers tailored audio feedback, counting reps if correct or providing guidance if incorrect. Each exercise receives unique audio feedback, offering precise guidance to improve posture. Moreover, the system tracks patient progress and displays visual representations of improvement over time. This feature aids patients in monitoring their progress and fosters motivation to adhere to therapy. By leveraging deep learning algorithms and the media pipe library, this system presents a precise, efficient, and economical approach to physiotherapy assessments.

Introduction

I. INTRODUCTION

A. Overview

In today's context, physiotherapists frequently rely on subjective assessments, such as patient self-evaluations and manual examinations, which are vulnerable to inaccuracies and biases. Additionally, manual evaluations often consume considerable time and may lack the necessary detail for precise diagnosis and treatment planning. The integration of deep learning into the realm of physiotherapy assessment emerges as a promising avenue. By utilizing deep learning techniques, physiotherapists have the potential to attain a more objective and nuanced understanding of a patient's condition, potentially leading to improved treatment effectiveness and better outcomes. Moreover, deep learning methodologies hold the promise of automating specific tasks like data compilation and analysis, thus optimizing processes and enhancing efficiency. In essence, incorporating deep learning into physiotherapeutic approaches represents a rich area for exploration, offering prospects for significant advancements in patient assessment and therapeutic strategies. We based our work on an idea presented by a team of dentists and  physiotherapists describing a wearable posture trainer to help dentists achieve ergonomic postures at work

B. Motivation

The impetus to integrate deep learning into physiotherapy assessment technology stems from the urgent need for refined precision, objectivity, and effectiveness in assessment methodologies. A fundamental challenge in physiotherapy lies in the dependency on subjective measurements, such as patient self-assessment and manual evaluations, which are vulnerable to inaccuracies and biases. These approaches can be influenced by variables such as the patient's pain levels or anxiety, potentially leading to an imprecise representation of the patient's condition. Moreover, manual assessments often consume considerable time, limiting the number of patients attended to in a day and may lack the requisite depth for meticulous diagnosis and treatment planning. Deep learning offers a remedy to these obstacles by providing a more objective and thorough comprehension of a patient's condition. Through neural networks, deep learning can scrutinize extensive datasets, including movement patterns, and generate predictions or decisions based on this information. Consequently, this approach can facilitate more precise diagnosis and treatment planning, ultimately resulting in enhanced patient outcomes. Essentially, the integration of deep learning into physiotherapy assessment technology is motivated by the imperative for heightened accuracy, objectivity, and efficiency in assessment methodologies. It holds the potential to empower physiotherapists to make well-grounded treatment decisions, thereby augmenting overall patient outcomes.

C. Objective

This application aims to create a deep learning-based solution for assessing physiotherapy exercises by leveraging the capabilities of the MediaPipe library. It will analyze the angles of the patient's posture during exercises to evaluate their accuracy. Depending on the posture's correctness, the application will deliver personalized feedback, including rep counting or audio cues for each exercise. Additionally, it will track the user's progress over time and share this information with both the patient and therapist, facilitating progress monitoring and adjustments to treatment plans as needed. By combining computer vision techniques and deep learning algorithms, the application will scrutinize the patient's posture during exercises. The MediaPipe library will detect the key points and angles of the patient's posture, feeding into a deep learning model trained to recognize the correct posture for each exercise. Using the model's output, the application will provide tailored feedback, such as rep counting or audio cues, specific to the exercise being performed. User progress will be continuously monitored and displayed in a user-friendly format, streamlining progress tracking and enabling adjustments to treatment plans.

D. Scope

The project outlined above, focusing on utilizing a deep learning-driven approach for evaluating physiotherapy exercises, has a broad scope and holds substantial potential to revolutionize the field of physiotherapy in various ways. Here are practical scenarios where the project can be applied:

1. Integration into Rehabilitation Centers: Integration of this application into rehabilitation centers can aid physiotherapists in assessing and treating patients with a range of conditions such as musculoskeletal injuries, neurological disorders, or chronic pain. The ability to detect angles and provide personalized feedback would augment the precision and efficiency of patient evaluation and treatment.
2. Home-based Therapy Support: Furthermore, the application can facilitate home-based therapy, enabling patients to conduct exercises at home while receiving immediate feedback and tracking their progress. This would be particularly beneficial for patients encountering difficulties in accessing rehabilitation centers or residing in rural areas.
3. Implementation in Telerehabilitation: Additionally, the application could be deployed in telerehabilitation setups, allowing patients to receive physiotherapy remotely via video conferencing. Real-time angle detection and feedback provision would enable physiotherapists to remotely evaluate and treat patients, potentially enhancing physiotherapy accessibility for individuals in remote regions or with mobility limitations.

The importance of this application is underscored by its ability to tackle contemporary challenges encountered by physiotherapists. A significant obstacle is the reliance on subjective measurements like patient self-reports and manual assessments, which are prone to errors and biases. By furnishing a more objective and detailed understanding of a patient's condition, this application holds promise in enhancing treatment effectiveness and outcomes.

E. Current Assessment Methodology

The prevailing method for evaluating physiotherapy exercises involves manual assessments administered by a physiotherapist. Throughout these evaluations, the physiotherapist observes the patient's performance of the exercise, assessing the accuracy of posture and the range of motion. This assessment process typically relies on the physiotherapist's expertise, supplemented by established guidelines and exercise protocols. Additionally, tools such as goniometers or inclinometers may be employed to measure joint angles, although these measurements are still subject to human error and variability. Ultimately, the physiotherapist predominantly depends on visual observations to evaluate the execution of the exercise.

F. Proposed Approach

The proposed system introduces a deep learning-driven solution for evaluating physiotherapy exercises, leveraging the MediaPipe library. Employing computer vision techniques and deep learning algorithms, the system analyzes the patient's posture during exercises. Initially, the patient's video is captured using a webcam, and the OpenCV library extracts frames from the video. These frames are then processed by the MediaPipe library, employing computer vision techniques to identify key points on the patient's body. These key points are interconnected to create a representation of the patient's pose specific to the exercise being performed. Subsequently, the system calculates the angles between the key points using trigonometry. These angle measurements serve as input for a deep learning model trained to identify the correct posture for each exercise. Based on these angle measurements, the system assesses whether the patient's posture is accurate.

Correct postures prompt the system to provide audio feedback with rep counting, while incorrect postures prompt audio feedback detailing the errors. The feedback is tailored to the specific exercise, aimed at assisting the patient in improving posture and range of motion.

Furthermore, the system tracks the user's progress over time and presents this data in a user-friendly format, facilitating easy monitoring of progress and adjustments to treatment plans. Mediapipe's Blazepose algorithm offers an advantage over other widely used human pose recognition frameworks such as Openpose. Notably, it boasts lower system requirements, faster recognition speed, and higher accuracy, rendering it an optimal solution for the increasing demand for smart fitness applications on various devices.

II. LITERATURE REVIEW

In the field of physiotherapy, there has been significant exploration into leveraging assessment technology and machine learning methods to improve the precision and effectiveness of physiotherapy exercises. One promising avenue involves the adoption of "earable" inertial sensing devices like smartwatches or fitness trackers, enabling real-time tracking of head posture [1]. Furthermore, studies have utilized Kalman filter-based noise reduction frameworks to process depth sensor data for posture estimation [2]. Similarly, machine learning and computer vision techniques have been employed for analyzing indoor workouts [3].

Exploring the assessment of squat angles has also been a focal point, with investigations focusing on tracking body movements [4], and utilizing MediaPipe for fitness action counting [5]. Artificial intelligence and machine learning applications in musculoskeletal physiotherapy have garnered attention across various studies [6]. For example, research has delved into the potential of computer vision-based marker-less human motion analysis for rehabilitation [7], while other efforts have resulted in the development of real-time hand motion detection systems for unsupervised home training [8].

Additionally, there has been considerable research on whole-body pose estimation using deep learning techniques. The Openpose method, for instance, has been introduced [9], and studies have employed compact optical flow-based motion representations for real-time action recognition in surveillance settings [10]. Furthermore, deep learning frameworks have been proposed for assessing physical rehabilitation exercises [11], alongside real-time 3D human pose estimation systems utilizing a single RGB camera [12].

Recent studies have proposed technological solutions for real-time monitoring with feedback during low back pain physical therapy [13]. Moreover, the applications of artificial intelligence and machine learning in musculoskeletal physiotherapy have been discussed in detail [14], including the recognition of shoulder physiotherapy exercises using machine learning techniques applied to inertial signals from smartwatches [15].

In conclusion, the literature highlights the significant potential of assessment technology and machine learning methodologies in improving the accuracy and efficiency of physiotherapy exercises. Wearables and computer vision-based techniques have seen substantial exploration, along with deep learning approaches for whole-body pose estimation. However, further research is essential to fully exploit the capabilities of these methodologies in the field of physiotherapy.

IV. IMPLEMENTATION

A. Overview of Technological Integration

The proposed project amalgamates a variety of state-of-the-art technologies to develop a comprehensive solution for physiotherapy assessment. At its core, deep learning algorithms are employed to analyze and interpret recorded video data. This innovative approach utilizes neural networks to extract valuable insights from video frames, facilitating accurate detection of key points on the user's body. To bolster this analytical process, the project utilizes the MediaPipe library, well-known for its robust and efficient body key point detection and tracking capabilities.

In addition to deep learning and MediaPipe, several other technologies play pivotal roles in the project. OpenCV, a powerful computer vision library, is utilized to access and manipulate video frames from the webcam, ensuring thorough visibility of the entire body for precise assessment. Furthermore, Flask, a Python-based web framework, is implemented to design and develop the web application segment of the project. This enables seamless integration of the physiotherapy assessment system into a user-friendly and accessible web interface.

By harnessing this diverse array of advanced technologies, the project aims to transform the landscape of physiotherapy assessment, providing an automated and accurate solution capable of assisting both patients and healthcare professionals in monitoring and improving their physical performance.

B. Implementation details of modules

1. Video Frame Capture and Key Point Identification:

• Acquiring Video Frames and Detecting Key Points with OpenCV.
• Tracking Body Key Points Using MediaPipe for Exercise Analysis.
• Extracting Relevant Body Key Points from Video Frames.

2. Angle Calculation and Feedback Provision:

• Determining Angles and Providing Feedback Assessment
• Computing Angles Between Key Points to Evaluate Posture
• Establishing Thresholds for Posture Assessment and Feedback Delivery

3. Progress Tracking and Visualization:

• Implementing User Data Storage for Progress Monitoring
• Monitoring Exercise Repetitions and Timestamps
• Comparing Exercise Duration with Targets for Progress Evaluation
• Visualizing Progress Data Using Charting Libraries

C. Overcoming Challenges and Strategic Approaches

1. MediaPipe Library Integration: Navigating the integration and setup of the MediaPipe library for body key point detection demands a thorough grasp of its documentation and APIs. Conquering this obstacle involves in-depth research, hands-on experimentation, and tapping into community resources or forums for guidance and solutions.
2. Precision in Angle Calculation: Ensuring the accuracy of angle calculations amidst video frame noise and varied body poses presents a challenge. Enhancing precision involves deploying robust algorithms, incorporating filtering methods like smoothing algorithms, and rigorous testing with diverse datasets to refine angle calculation accuracy.
3. Bridging with Flutter Interface: Establishing connectivity between the Python backend and the Flutter frontend poses potential hurdles. If direct integration proves intricate, an alternative approach entails crafting a web application using Flask. Leveraging the Flask framework allows embedding Python functionalities within a web app accessible from Flutter via web APIs or web views.
4. Deployment and Scalability: Addressing deployment challenges to accommodate simultaneous user access is paramount. Harnessing cloud services such as AWS or Google Cloud Platform, adopting containerization with Docker, and implementing load balancing techniques aid in achieving robust deployment and scalability solutions.
5. User Interface Design: Crafting an intuitive interface and offering clear exercise instructions and posture corrections necessitate thoughtful design. Conducting thorough user testing and gathering feedback enable iterative refinement of the user experience, addressing usability hurdles effectively.

V. TESTING AND EXPERIMENTAL ANALYSIS

A. Module Evaluation

Unit testing entails assessing the functionality of individual software modules to ensure their proper performance and absence of unexpected outcomes. PyUnit, a widely used testing framework for Python programs, was employed to rigorously test each module within our project. All test cases were successfully validated.

B. System Integration Verification

Integration testing involves evaluating the coherent functioning of interconnected modules when tested collectively as a group. The output consistency of both the multi-ingredient classifier and recommendation system was verified during integration testing.

C. Experimental Dataset Utilization

Following the testing phase selection, the software and hardware components are integrated to form a complete system, which is then subjected to system testing. This process effectively assesses the fulfillment of end-to-end system specifications.

VI. FUTURE SCOPE

There is potential for further enhancement through advanced deep learning models, expanded computer vision techniques, or additional feedback mechanisms like visual or haptic feedback.

VII. ACKNOWLEDGEMENT

We actively sought the expertise of Dr. Ravi Shirahatti, Professor and Head, Public health dentistry, SDM College of dental sciences and hospital, Shri Dharmasthala Manjunathwara University, Dharwad to provide valuable insights and guidance pertaining to various facets of our project. His gracious acknowledgment of our outreach and willingness to offer support were warmly received and appreciated.

Conclusion

The proposed system represents a significant leap forward in physiotherapy assessment by integrating computer vision and deep learning techniques. Utilizing the MediaPipe library, it offers a comprehensive solution for accurately estimating a patient\'s pose during physiotherapy exercises, which is crucial for precise assessment. Real-time feedback and progress monitoring enhance therapy outcomes significantly. This system addresses traditional manual assessment limitations such as subjectivity and human error, providing an objective evaluation of posture and range of motion. Its efficient data collection and analysis empower physiotherapists to make well-informed treatment decisions and adjustments. The experiment conducted provided successful outcomes across five distinct categories of exercises. Upon analysis, it was determined that out of a total of ten trials, eight yielded accurate results. This observed success can be attributed to delays in server responsiveness.

References

[1] Nagarkoti, R. Teotia, A. K. Mahale and P. K. Das, \"Realtime Indoor Workout Analysis Using Machine Learning & Computer Vision,\" 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2019, pp. 1440-1443, doi: 10.1109/EMBC.2019.8856547. [2] M. A. Zulkifley, N. A. Mohamed and N. H. Zulkifley, \"Squat Angle Assessment Through Tracking Body Movements,\" in IEEE Access, vol. 7, pp. 48635-48644, 2019, doi: 10.1109/ACCESS.2019.2910297. [3] X. Li, M. Zhang, J. Gu and Z. Zhang, \"Fitness Action Counting Based on MediaPipe,\" 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP- BMEI), Beijing, China, 2022, pp. 1-7, doi: 10.1109/CISP-BMEI56279.2022.9980337. [4] Godse, Sachin P., et al. \"Musculoskeletal physiotherapy using artificial intelligence and machine learning.\" International Journal of Innovative Science and Research Technology volume 4 issue 11 (2019): 592-598. [5] Hellsten T, Karlsson J, Shamsuzzaman M, Pulkkis G. “The Potential of Computer Vision-Based Marker-Less Human Motion Analysis for Rehabilitation”. Rehabilitation Process and Outcome. 2021;10. doi:10.1177/11795727211022330. [6] Poster titled \"Ergo sure posture trainer\" presented by Dr Ravi Shirahatti and team on National science day, 28th February 2023 at SDM University, Dharwad.

Copyright

Copyright © 2024 Dr. Shrinivasrao B Kulkarni, Abhishek Kuppelur, Akash Shetty, Shashank Bidarakatti, Taranath Sangresakoppa. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.