Application of Mixed Reality for Assembly Line in Manufacturing Plants

Abstract

Augmented reality and mixed reality-based applications are used widely in different areas for last decade. Using AR/MR for AI simulation and training is not a new thing for many trainings simulations AR/MR is used in areas like military and medical. Although same cannot be said with the manufacturing industry there is need in manufacturing plants for training new employees/visitors about the process and machines along with instructions to use them. This can be done using mixed reality and marker less detection. Assembly line in Manufacturing plants is a brief process which needs to be followed by workers for better efficiency and quality. Using mixed reality for training workers by walking though assembly line will help immensely as mixed reality-based applications are very intuitive and they help in learning. This study focusses on finding challenges and proof of concept android mobile application while benchmarking them with various metrics which will help in further developments in area.

Introduction

I. INTRODUCTION

Augmented Reality and Mixed reality are growing markets and transforming many industries according to recent survey AR/MR market generated 35 billion USD $ revenue in 2023 [1]. There is opportunity in this market and many industries are aligning with Mixed Reality based applications. Manufacturing industry is huge industry with over 500 Billion USD revenue (source: Statista) [2]. There are many processes though which a product goes while manufacturing there are special tools/software’s used to manage these processes known as product life cycle management [3]. One of the core processes in every product’s life cycle is production phase where product is made in the manufacturing plant having assembly line to maximize the efficiency and ensure quality.

Every assembly line for product has different set of tools and machinery to operate with each tool and machine comes with operating instruction and maintenance information i.e., when was the last maintenance performed on the machine. Many visual indicators and stickers are used which are updated each time when there is update. Also, there is visual limitation as this information is written in small area this can lead to serious issues and health hazards caused by lack of maintenance and awareness in working environment.

Mixed reality-based application can be used to make this information available using there smartphones. As MR applications are intuitive and user friendly it is easy to get snapshot of the view and perceive information in short duration without going through catalog and menus of the machine. As applications are connected to internet it is easy to show real-time information like last maintenance and latest instruction on top of machine/tools view.

II. CHALLENGES

A. Need Of Marker Less Detection

Assembly line and manufacturing plant is a working environment which should be distraction free and easy to maintain introducing markers i.e., QR codes or Bar codes for object recognition creates its own problems to maintain markers and update them time to time this can be avoided by using marker less object detection where tools/machines will be used as a marker along with GPS location. In environments like assembly line machines are stationery and location is changed rarely. It is safe to assume marker-less object detection along with geo-tagging can give us unique results.

Due to recent advancements in technology marker less object detection can be achieved offline (without internet/remote services) using on device machine learning and image processing.

B. Multiple Markers

Assembly line due to its nature can have multiple machines and tools at one place and each one having different information can create conflicts MR application used in such should support multiple marker detection and show information for all of them in one view.

???????C. Data Synchronization

Information related to maintenance of the system should be synchronized and updated frequently for this new catalog and machine learning models should be deployed when updated as this is critical information should be handled with care. System should be robust and maintain transactions.

Each organization maintains information related to assembly line in central database which can be accessed through product life cycle management software.MR application should be synced with this information in order to get latest information and removing dependency of maintaining information at two places.

III. IMPLEMENTATION

A. YOLO object detection.

YOLO (You Only Look Once) is object detection algorithm used for detecting multiple markers in single snapshot of the view along with the pixel co-ordinates of the detected objects providing bounding boxes.

“YOLO9000, a state-of-the-art, real-time object detection system that can detect over 9000 object categories” [3]  YOLOv2 as paper describes is the fastest algorithm to achieve real time multiple object detection with such robust algorithm trained with well-known and widely used dataset “ImageNet” [4] will create robust algorithm using transfer learning YOLO can be trained with images of machines and tools for assembly line marker less object detection.

YOLO is CNN (Conventional Neural Network) based algorithm and takes time to train models and can be used with already trained dataset for transfer learning to reduce training time and ensure dataset quality. 

B. Cloud Computing for synchronization

Various public cloud providers like amazon, google and azure have dedicated services for model deployment which ensure that all the devices connected (mobile devices running MR application) have latest model and information synced with cloud.

This saves efforts to create separate infrastructure for managing and updating models for deployment. Using cloud services is better option as this is critical application and data integrity is important factor.

IV. RESULTS

We tested YOLO v2 with proof-of-concept application developed in Android using TensorFlow Lite model format specially developed for mobile application which ensures small model size. Testing conditions for this POC were sufficient visible light and general android smartphone with 4 GB Ram and latest generation MediaTek Processor

Figure bellow shows frame processed per second, This is important metric for MR application as application with low framerate can miss information and cause distress. Recommended framerate for  such application is 40 FPS which is ideal but hard to achieve due to diverse nature of android eco system and multiple vendors with different specification

Same application tested with K-Fold accuracy metrics to ensure correctness and accuracy of the machine learning model generated.

Table 1: Accuracy.

Conclusion

To conclude from results and metrics recorded for different type of datasets Mixed reality-based application can be used for assembly line in manufacturing plant using on device machine learning and YOLO along with cloud computing for model distribution. Application performs faster (30 FPS) and accurately (93% ~).

References

[1] AR & VR - worldwide: Statista market forecast (no date) Statista. Available at: https://www.statista.com/outlook/amo/ar-vr/worldwide#revenue (Accessed: January 8, 2023) [2] Deng, J. et al. (2009) “ImageNet: A large-scale hierarchical image database,” 2009 IEEE Conference on Computer Vision and Pattern Recognition [Preprint]. Available at: https://doi.org/10.1109/cvpr.2009.5206848 [3] Published by Statista Research Department and 30, S. (2021) Forecast: Industry revenue of \"manufacturing\" in the United Kingdom 2012-2025, Statista. Available at: https://www.statista.com/forecasts/858948/manufacturing-revenue-in-the-united-kingdom (Accessed: May 8, 2023). [4] Redmon, J. and Farhadi, A. (2017) “Yolo9000: Better, faster, stronger,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) [Preprint]. Available at: https://doi.org/10.1109/cvpr.2017.690.Conclusions

Copyright

Copyright © 2023 Rutuja Balasaheb Chavan, Dr. Sulochana Sonkamble, Dr. Nisha Auti. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.