Smart Cane: A Machine Learning-Powered Assistive Device

Abstract

A visually disabled person finds it difficult to find the presence of any obstacles in his approach and it is terrible to look for the exact position of the stick if it is not correctly placed. Therefore, the multifunctional blind stick comes as a planned answer to help visually impaired people in their daily lives, without the help of others. This paper explains the different methods used to detect objects in front of the user.

Introduction

I. INTRODUCTION

According to the World Health Organization, there are about 285 million people with some form of visual impairment, of which 86 percent have low vision and 14 per centare blind. Vision is one of the most important vital senses for humans to survive. Vision helps us to connect with the environment. People without vision depend on other dependencies such as a simple walking cane or other people. In well-known places like the interior of a house, they remember the directions of the place, and obstacles on their way and navigate according to them. However, it is not always safe for the blind to rely on memory to move from one place to another. Especially when they are outside. It is not always possible for blind people to be helped by others, thus a device, such as a stick, is required that can help visually impaired people in all forms of life. The blind is someone who is limited by their vision. They often experience obstacles in their daily life.

Visual limitations certainly cause problems in accessing information and new encounters that take place in daily life. There are various frameworks of transportation systems for travelers externally weak to explore rapidly and securely against obstacles and different dangers. Each development has its own preferences and disadvantages. At present, blind people use stick for rolling and walking. Due to the amount of stress that our eyes are exposed to, there is an increase in the number of blind people.

The aim of the blind stick is to make the blind more independent. In fact, it will give a positive dimension to their lives. Traditional obstacle-detection methods are outdated and require considerable modifications.

In this paper, we focus on different types of ways in which a multifunctional blind stick can be made. The first domain is the IOT where multiple sensors such as ultrasonic sensors, water sensors, infrared sensors etc can be used. As well as different microcontrollers can also be used like Arduino uno, Arduino mega, STM32, MSP 430, PIC microcontroller, etc for controlling all the devices and acting as the brain of the blind stick. The second domain is Machine Learning where small computers like Raspberry Pi can be used with camera functionality to detect the object. It doesn’t stop here      at this only one can also identify the type of object      such as whether it is a chair, table, car, bike, glass, person, etc. This can be achieved with the help of different types of object detection algorithms which are R-CNN (Region Based Convolution Neural Network), Masked R-CNN, Fast R-CNN, YOLO (You Only Look Once), SSD (Single Shot Detector), Mobile net, etc. Out of these algorithms, YOLO is the fastest to detect an object because it is one stage detector while R-CNN is a two-stage detector but its accuracy is more than YOLO. For communications GPS and GSM are included which will send the location of the user, RF transmitter to know the location of the stick if it is lost.

II. LITERATURE SURVEY

Various object detection methods are proposed in [1] a Walking Stick with an in-built ultrasonic sensor with a Raspberry Pi The authors have used an ultrasonic sensor is used to detect obstacles. On sensing obstacles, the sensor passes the data to the microcontroller. The microcontroller then processes the data and calculates if an obstacle is close enough. If an obstacle is close then the microcontroller sends an Alert signal to the blind person. In addition, they have also planned to embed the e-SOS system. Whenever a blind person feels any discomfort while navigating then he presses an e- SOS distress call button on the stick to give a video call to his family member using an RPI camera. The video is streamed in an Android mobile via Android application. The Android application also shows the location of the blind person to his family member. In Arnesh et al [2]proposed to design an artificial navigating system with adjustable sensitivity with the help of ultrasonic proximity sensor and a GPS module to assist these blind persons to walk fearlessly and independently in both indoor and outdoor environment. Their system can detect any type of upcoming obstacles and potholes using the reflection properties of ultrasound. In [3] a blind stick with MSP430 Microcontroller is proposed in which ultrasonic sensor, water sensor         will help to identify obstacles. The microcontroller will retrieve data and pass it on as vibrations which will notify the user about hurdles on the way. The authors have also used RF module to locate the stick      if misplaced. In [4] the authors proposed a blind stick using PIC microcontroller 16F877A. An ultrasonic sensor module, HC-SR04 is used for obstacle detection in the path of the blind person and a buzzer is used to make the person alert. It can detect obstacle within 5 to 35 cm range of distance. In G.SRINIVAS et al. [5] a blind stick using Arduino Uno with the help of an ultrasonic sensor which detects the distance between object and the person is proposed. If any obstacle comes in front of blind person, he/she can know about the obstacle by hearing the sound generated by the BUZZER. In [6] there is a blind stick using Arduino Uno with the help of an ultrasonic sensor and infrared sensor which detects the distance between object and the person. In [7]a blind stick which is integrated with ultrasonic sensor, infrared sensor, push button, soil moisture detector along with Arduino UNO. The ultrasonic sensors are used to detect obstacles and alert the user through buzzer. The GSM, GPS is used to send location of the user to specific person. In [8] a blind stick using Raspberry PI, ultrasonic sensor, camera for object detection. An interfacing GPS is also used to identify the exact location in case of any trouble. In [9]  a blind stick using Arduino Mega, ultrasonic sensor. They have used four ultrasonic sensors for four directions i.e. left, right, front and down. If ultrasonic sensor senses an object, it will give its response through the buzzer. In [10] a blind stick which is embedded with Arduino uno, vibrator, switch and sensor. If sensor senses the obstacle, the vibrator which is placed over the handle vibrates. Ultrasonic sensor detects obstacle ahead using ultrasonic waves. An application developed in [11] detects objects for visually impaired people. It can detect the objects in the user’s surroundings. The application is developed using Android operating system and Android-based smartphones. The limitations to this system are smart phone should be carried by the user all the time and it should be ensured with battery backup. The limitation could be overcome by developing wearable device which is more convenient. In [12] an object detection system using CNN, YOLO and python. Computer webcam has been used to capture an image and identify it with the help of YOLO framework. In [13] also a mobile application is proposed using OpenCV for image processing and TensorFlow for machine learning process.

III. COMPONENTS USED

1) Microcontroller: For communicating with different sensors, switches, modules, an Arduino Uno (see Fig.1) microcontroller is used. The Arduino Uno is categorized as a microcontroller that uses the Atmega328 as as controller in it. The board contains 14 digital input/output pins in which 6 are analog input pin, one power jack, USB connector, one reset button, ICSP header, and other components.  It works as a decision-making controller by obtaining various signals from the sensors and appropriately triggering output sensors.

The block diagram of Arduino Uno is shown in (Fig. 10). It shows all the important components embedded in the smart stick. The water sensor will detect the presence of water and will send the information to the microcontroller in digital form, the Arduino Uno will analyze this data/information and will send the command to the user notification setup i.e., buzzer according to our program to alert the blind person with two beeps.

The ultrasonic sensor will detect nearby obstacles up to the length of 5-35 cm that can cause an accident for blind people and the blind person will get informed about the same in a similar fashion as in the case of the water sensor but the buzzer would give only one long beep. One of the main features of our stick is to track the real-time location of blind people this is done using the SIM900A GPS-GSM module. We have also provided a panic button to help the blind person in case of an emergency which when pressed would send a SMS with the exact location of the user to their near and dear ones.

B. Phase 2: Detecting the type of object using Raspberry Pi

The block diagram of Raspberry Pi 3B+ is shown in (Fig. 11). It shows that the RPI camera is the input source for detecting objects with the help of machine learning models embedded in Raspberry Pi 3B+. The model has been trained to detect 90+ objects.

The various components used to determine the type of object are as follows:

1. TensorFlow Object Detection Framework: TensorFlow Object Detection is a framework built on top of TensorFlow that enables the training and deployment of object detection models. It provides a set of pre-built models and tools to train and evaluate custom object detection models using different architectures, including Faster R-CNN, Mask R-CNN, and SSD.
2. MobileNet SSD: This is a specific object detection model architecture that is optimized for deployment on mobile and embedded devices with limited computational resources. It is a combination of MobileNet, a lightweight convolutional neural network architecture, and SSD, a single-shot detection framework that predicts the bounding boxes and class probabilities for all objects in a given image in a single forward pass.
3. OpenCV: OpenCV (Open-Source Computer Vision) is a popular open-source computer vision and machine learning software library that provides a wide range of tools and functions for real-time image and video processing. In this project, it is used to capture the image of an object.
4. GTTS: GTTS (Google Text-to-Speech) is a Python library and command-line tool that allows you to convert text to speech using the Google Text-to-Speech API. The object name which we get on the bounding box is converted to speech and played with the help of an earphone/speaker giving the message “Object Detected.”

V. RESULTS AND IMPLEMENTATION

A. Hardware Implementation

The following (Fig. 11) image shows the various components mounted on the stick.

VI. FUTURE SCOPE

The paper has covered the basic requirement of a blind person. But certain additional upgrades can be added to it.

1. Powerful Raspberry Pi such as RPI 4 or an RPI with a minimum 16GB ram can be used for faster processing.
2. Higher-quality cameras can be used for clear images as well as to capture an object if it is at a long distance.
3. YoloV8 object detection model can be used to train to increase the accuracy.
4. Sensors can be mounted at different angles for better object detection.
5. Instead of a stick, smart spectacles can be made to detect objects as well as to read texts.

VII. ACKNOWLEDGEMENT

We sincerely thank Atharva College of Engineering and Prof. Mahalaxmi Palinje for their cooperation and support throughout the project at every step.

Conclusion

This technical paper has proposed and explained the implementation of a multifunctional blind stick to assist visually impaired people. It has concluded all the objectives which are the safety of blind people by detecting objects around them and also sending their location to friends/ family. The blind stick proposed in this paper can aid the visually impaired user by helping him/her navigate by detecting types of objects using machine learning and getting processed output in the form of audio output. Lastly, we used the and were able to produce around 65% accuracy during detection. The advantage of the system lies in the fact that it can be proven to be a very low-cost solution for millions of blind persons worldwide.

References

[1] S. Mohapatra, S. Rout, V. Tripathi, T. Saxena and Y. Karuna, \"Smart Walking Stick for Blind Integrated with SOS Navigation System,\" 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI), 2018, pp. 441-447, doi: 10.1109/ICOEI.2018.8553935. [2] A. Sen, K. Sen and J. Das, \"Ultrasonic Blind Stick for Completely Blind People to Avoid Any Kind of Obstacles,\" 2018 IEEE SENSORS, 2018, pp. 1-4, doi: 10.1109/ICSENS.2018.8589680. [3] M. P. Agrawal and A. R. Gupta, \"Smart Stick for the Blind and Visually Impaired People,\" 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), 2018, pp. 542-545, doi:10.1109/ICICCT.2018.8473344. [4] Dey, Naiwrita & Paul, Ankita & Ghosh, Pritha & Mukherjee, Chandrama & De, Rahul & dey, Sohini. (2018). “Ultrasonic Sensor Based Smart Blind Stick”. 1-4. 10.1109/ICCTCT.2018.8551067. [5] Srinivas, G., Raju, G. M., Ramesh, D., & Sivaram, S. (2019). “Smart Blind stick connected system using Arduino”. IJRAR-International Journal of Research and Analytical Reviews, 6(2), 934-939. [6] N. Loganathan, K. Lakshmi, N. Chandrasekaran, S. R. Cibisakaravarthi, R. H. Priyanga and K. H. Varthini, \"Smart Stick for Blind People,\" 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), 2020, pp. 65-67, doi: 10.1109/ICACCS48705.2020.9074374. [7] V. Kunta, C. Tuniki and U. Sairam, \"Multi-Functional Blind Stick for Visually Impaired People,\" 2020 5th International Conference on Communication and Electronics Systems (ICCES), 2020, pp. 895-899, doi: 10.1109/ICCES48766.2020.9137870. [8] T. S. Aravinth, \"WiFi and Bluetooth based Smart Stick for Guiding Blind People,\" 2020 3rd International Conference on Intelligent Sustainable Systems (ICISS), 2020, pp. 317-320, doi: 10.1109/ICISS49785.2020.9316084. [9] Tirupal, Talari & Murali, B & Sandeep, M & Kumar, K & Kumar, C & Head,. (2021). “Smart Blind Stick Using Ultrasonic Sensor”. 7. 34-42. [10] Patwardhan, S., Karivadekar, M. D., Phadtare, M. P., More, M. K., & Rabade, M. S. (2022). “Smart Blind Stick Using Arduino UNO”. Power, 600, 5V. [11] Aralikatti, A., Appalla, J., Kushal, S., Naveen, G. S., Lokesh, S., & Jayasri, B. S. (2020, December). Real-time object detection and face recognition system to assist the visually impaired. In Journal of Physics: Conference Series (Vol. 1706, No. 1, p. 012149). IOP Publishing. [12] Annapoorani, A., Nerosha Senthil Kumar, and V. Vidhya. \"Blind-Sight: Object Detection with Voice Feedback.\" International Journal of Scientific Research & Engineering Trends 7.2 (2021): 644-648. [13] S. TOSUN and E. KARAARSLAN, \"Real-Time Object Detection Application for Visually Impaired People: Third Eye,\" 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), 2018, pp. 1-6, doi: 10.1109/IDAP.2018.8620773,IEEE.

Copyright

Copyright © 2023 Suraj Sant, Mihir Salunkhe, Kunj Shah, Gaurav Kumawat, Prof. Mahalaxmi Palinje. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.