Augmented Reality Based Learning App for Children with Hearing Disorder

Abstract

The pandemic hits the world today has made teachers and parents to develop attractive learning media for young learners/students to keep them excited while learning from home. Android smartphone devices are currently widely used for interaction media between children’s and teachers during learning and teaching from home. From this Method, software developers developed learning media using Augmented Reality that can be easily accessed from Android smartphone for young students. This application focuses on creating learning media and structures based on Android using Augmented Reality (AR) technology. The AR application was made using Unity with Vuforia SDK which is A software development tool. It is used to track down an target image so that the AR objects can be displayed on the screen in augmented form. Vuforia SDK detects target image by comparing captured images with target image which are stored in the database. This application resulted three-dimensional AR objects, two-dimensional AR objects which are ready to be used as learning media which are feasibly to evaluate student’s skills after learning through AR objects.

Introduction

I. INTRODUCTION

AR has been used in various types of fields from past to present, including the training of pilots, medical education, and engineering, among others. In the broadest terms, AR can be defined as a computer-generated image that overlays virtual objects (augmented components) into the real world by using a display device in a real-world environment. Such technology works interactively and in real-time. Another technology in which real and virtual environments are used that is virtual reality (VR), although VR adopts completely different approaches. The actual difference is that the AR uses a virtual knowledge to develop the real world while viruses the elements of the real world to create virtual environments. Despite this difference, both will aim to enhance the personal sensorial environment. Some researchers have also described mixed/merged reality (MR), which differs somewhat from AR and VR. For example, Intel has developed a virtual reality headset, which is described as a “merged reality” device, which allows nearby objects from the real world to be integrated into a brand-new computer-generated view. However, both AR and MR uses the same image source, environment, display devices, perspective, and presence; they vary in terms of awareness. In AR, virtual objects will be identified by their nature and behaviour, while in MR, it is impossible to differentiate the smoothly rendered virtual objects from real objects [1].

II. DEVELOPMENT TOOL INTRODUCTION

A. Unity 3D Introduction

Unity3D is a game engine and AR-VR application development toolkit which is developed by Danish company Unity. It is a multi-platform game and application development tool, a fully integrated professional game engine, and can provide many functions such as rendering engine, physical engine, script engine, lighting mapping and scene management. Unity3D is mainly used to create games and interactive 3D and 2D experiences such as training simulation, medical and structural visualization. A complete Unity3D project is composed of several scenes, each of which contains many models (3D Game Obiect) and their behaviour which is controlled by scripts (including C#). What we see in the scene are presented and controlled by camera and UI elements [2].

B. Vuforia SDK Introduction

Vuforia is a Qualcomm's software development kit for mobile AR applications, and it is an AR-VR SDK for developers. It uses computer vision technology to identify and capture flat images or simple three-dimensional objects in real time, and allows developers to place virtual objects through camera viewfinders and adjust the position of objects in the physical background in front of the camera lens [2].

III. PROJECT IDEA

Augmented Reality has attracted students towards itself in past few years. Students found gaming more more attractive and comfortable because of 3D models. As we have seen education is in traditional on paper diagram and images format from decades. To attract students towards education and found it more interesting we can use AR technology. AR technology is very instructive for the presentation of abstract and informative content. Because of its characteristics, AR-VR technology has great potential and lots of opportunity for development in education field.  The characteristics of AR technical simulation and interaction can display the abstract and obscure knowledge in a more vivid, intuitive and comprehensive way, and can enhance students' sense of immersion. The mobile AR system can strengthen the ubiquitous cooperative and scene learning with the help of virtual objects in the real environment. Everything in real life can become a prop for AR learning, achieving convenience, interactivity, situation, connectivity and personalization.

IV. LITERATURE SURVEY

The research for augmented reality technology has brought up development of various applications in the field of education. In this literature review, it shows how the use of augmented reality in various fields using unity 3D.

Several research studies are used as guidelines and references in making this AR application, among others. Research conducted by Latius Hermawan and Mochamad (2015) titled as "The Utilization of Augmented Reality as Campus Information Media Using Brochures". This study aims to produce an android application that can display a 3D brochure model in an environment Augmented Reality so that it can help readers to know which campus to choose well (Hermawan, Hariadi, 2015) [3].

In the journal Prihantoro Herlambang, (2015) entitled "Augmented Reality Edu game, the Life Cycle of Living Beings Based on Android". The purpose of this research is to make a learning game in order to attract students 'interest in learning and help teaching and learning activities for the 2013 curriculum. The benefit of this research is to help teachers become easier in teaching because with the addition of games and augmented reality to attract students' attention to be more focused on material and students can get to know better what games are educational, and can distinguish between games that are useful for education and those that are only harmful (Prihantoro, 2015) [3].

RisYesn Arief SetYeswan, (2016) with title "Analysis of the Use of the Marker Tracking Augmented Reality of Traditional Central Java Musical Instruments". The purpose of this research is to develop an AR application to display traditional musical instruments of Central Java Province in the form of simple 3- dimensional animation, and its benefits are to providing the introduction of traditional Central Javanese musical instruments using AR technology (SetYeswan, 2016) [5].

Yuni Sartika, (2016) entitled "Solar System Learning Applications for Grade 6 Elementary Schools Using Augmented Reality Android-Based". The purpose of this research is to build a Solar System Learning Application for Class 6 Elementary School Using Augmented Reality Based on Android, providing learning media application that contain descriptions of the circulation of planets in the solar system with animated images, describing the characteristics of the planets in the solar system. The benefit provided is that students can understand the material (Sartika, et al, 2016) [6].

Agmita Clara Rosa, et al (2019) in a study entitled “Engineering Augmented Reality Planets in the Solar System as a Learning Media for Students of SMP Negeri 57 Palembang”.  The purpose of this research is to build a simulation system for the planet’s solar system of the Milky Way as an interactive and fun to learning medium for students using technology augmented reality android- based which is effective in providing information about planet descriptions, planetary layers and can be used for learning, there are quiz questions in the application (Rosa et al, 2019). Comparison with previous research can be seen in Table [7].

V. SYSTEM ARCHITECTURE

VI. METHODOLOGY & IMPLEMENTATION

A. Configuring Image Targets

To build AR applications, it is necessary to activate the AR camera using Game Object Vuforia Engine AR Camera. Image targets trigger the appearance of virtual objects in an AR app. The corresponding target should be specified so that the application recognizes it. Vuforia Developer Portal and Target Manager page are implemented to set up the image targets. Add Database button is used to proceed to the Create Database page where target image database can be set up, while Add target button, to find and select the target images.  Vuforia supports four types of targets – Single Image, Cuboid, Cylinder, and 3D Object (we choose Single Image target). Vuforia adds the star rating that shows how easy it will be for the camera to recognize image. Aim for 5 stars. Database can be used in Unity3D after downloading and selecting Unity Editor as the development platform [8].

B. Adding Image Targets and 3D Objects

Image Targets represent Marker that Vuforia Engine can detect and track-down. Unlike traditional fiducial markers, data matrix codes, and QR codes, Image Targets do not need special black and white regions or codes to be recognized [8]. The Engine detects and tracks the key-point features that are naturally found in the image itself by comparing these natural features against a known target resource stored database.  Once the Target image is detected, Vuforia Engine will track-down the image as long as it is at least partially in the camera’s field of view.  Image targets can be created with the Vuforia Target Marker Manager using JPG or PNG or jpeg images in RGB or grayscale. The size of the input images which we want to store in database must 2 MB or less. Features extracted from these images are stored in a database, which can then be downloaded and packaged together with the application. The database can then be used by Vuforia Engine for comparison in runtime of application. To add an Image Target to the Scene, it should be selected with Vuforia? Image from the menu Game Object. The image target appears on your Hierarchy screen and can be used to connect to the 3D object as a target marker. ?hen the type of object that would like to appear in your AR application needs to be selected [8].

Digital content to augment target will be added by simply adding assets as children of the target in the scene hierarchy, Parenting content with a target object automatically sets up the necessary rendering and physics behaviours. It is used ?sset à Import New Asset for this purpose. Unity will work with 3D models of any shape that can be created with modeling software. There are also number of primitive object types that can be created directly within Unity. Unity can read .fbx, .dae (Collada),  .3ds, .dxf, and .obj files. Exported files from 3D modeling software in generic formats can be imported and edited by a wide variety of different software. Model files which will contain a variation of data, such as character and terrain Meshes, Animation Rigs and Clips, as well as Materials and Textures. In this case 3D model is created using SolidWorks software and imported as .fbx file. Vuforia Engine provides a simulator in the Game view that can be activated by pressing the Play button. This feature can be used to evaluate and rapidly prototype scene(s) without having to deploy at a device [8].

C. Building the APPLICATION

After developing the application in Unity, it is  necessary to choose File à Build Settings, switch platform  to Android, and then click Build to export the application as an APK file. This APK file can then be installed on an  Android phone [8].

VII. RESULT & DISCUSSION

The Augmented Reality application can be run directly at the notebook and smartphone when the camera pointed towards the target marker, the AR run without problem, graphic, animation all worked well on screen. When camera moved away from the target marker, the AR animation also stopped, this is intended as we want that AR only runs when camera pointed towards the target marker because AR system usually triggered to a particular Marker [9]. The final result of this project is the apk file (Android application package). The apk file is an installer file that developed for devices that run on Android operating system. The .apk files required to be installed first on the Android smartphone. After installation, Run the app and point the camera towards the "Target marker." If the app detects the Target Image Marker, either it's on the paper, on monitor, laptop or tablet screen, the set 3D model object will pop up on the smartphone's screen in Augmented form. The graphics, animation, and sound also worked well; the AR animation also stopped when we put away the camera from a target marker.[9]

VIII. ACKNOWLEDGEMENT

We take this opportunity to thank our principal Dr. Prashant Deshmukh., Head of the Department Dr. Sanjay Sharma and project guide Ms. Sheetal Jadhav for their valuable guidance and for giving all the necessary facilities which were indispensable for the completion of this project.

We are also thankful to all the staff members of the Department of Computer engineering of New Horizon Institute of technology and Management for their valuable time, support, comments, suggestions and persuasion.

Conclusion

The aim of this Project was to demonstrate the basic usage of Vuforia and Unity 3D to make an application. There is no limitation in development of the quality of the application with these platforms but because of the short time frame, focus was switched to the background and basic usage of the soft- ware. The application has been also developed from the scratch but with help of open-source objects. Virtual Reality and Augmented Reality, which is a new technological advancement in the digital world, and it will slowly changing everything. Virtual realities has been already being used in many parts of the world for numerous purposes. Education systems can be entirely replaced by switching to Virtual Reality, by providing education to the children in refugee camps and in remote locations where there are no schools. The use of Virtual Reality can entirely flip marketing, tourism, training, the medical world, and many more aspects of life; it can enhance user experience and reduce risk in practice. Virtual Reality is the future; in the coming years, the world can completely shift to Virtual Reality, and our life will be more luxurious and comfortable. With more technological advancements, virtual Reality can provide any experience in the world with real-lifelike scenarios. It will change how the industries operate, healthcare and education systems, commerce, and entertainment experiences will be completely enhanced. Immersive environments and surreal experiences will enhance the human lifestyle, and technology will once again make our lives easier.

References

[1] Yilmaz, Ozkan. “Augmented Reality in Science Education: An Application in Higher Education.” Shanlax International Journal of Education, vol. 9, no.3,2021, pp.136-148.DOI: https://doi.org/10.34293/education.v9i3.3907 [2] ChangYuan Li BaiHui Tang, Research on The Application of AR Technology Based on Unity3D in Education Department of Communication and Information Systems, Collaborative Innovation Center of Communication University of China, IOP Conf. Series: Journal of Physics: Conf. Series 1168(2019)032045,doi:10.1088/1742-6596/1168/3/032045 [3] Hermawan and Mochamad. 2015. \"Using Augmented Reality as Information Media.\" National Seminar on Information and Communication Technology 2015 [4] Herlambang, P. 2015. \"Edugame Augmented Reality Cycle of Living Things Based on Android\". Published. Faculty of Communication and Informatics. MuhammadiYesh University. Surakarta [5] SetYeswan, R. A., and Dzikri, A. 2016. \"Analysis of the Use of the Marker Tracking Method in Augmented Reality of Traditional Central Javanese Musical Instruments\". Batam State Polytechnic. Batam. [6] Sartika. Y, Telnoni, P, A, and Tambunan, T. D. 2016. \"The Application of SurYes Learning for Science in Grade 6 Elementary Schools Using Augemented Reality Based on Android\". E-Proceeding Of Applied Science, 895. [7] Rosa, A. C. 2019. \"RekaYessa Augmented Reality Planet in SurYes System as a Learning Media for Students of SMP Negeri 57 Palembang\". Scientific Journal of Informatics. [8] A.Bocevska , Z. Kotevski , N. Rendevski and B. Ristevski Marker-Based Mobile Augmented Reality for Application of Visualization in Product Packaging International conference on Applied Internet and Information Technologies October 3-4, 2019, Zrenjanin, Republic of Serbia [9] M Sarosa1, A Chalim1, S Suhari2, Z Sari3 and H B Hakim3 Developing augmented realitybased application for character education using unity with Vuforia SDK Phys.: Conf. Ser. 1375 012035 doi:10.1088/1742-6596/1375/1/012035

Copyright

Copyright © 2022 Omkar Kachare, Nikhil Jadhav, Sachin Kanojiya, Sheetal Jadhav. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.