Augmented Reality

Abstract

Imagine a world with a technology that creates the 3 dimensional images of a virtual object around you with which you can interact, see, hear, smell, and even touch it. Technologies such as computer graphics, virtual reality, and augmented reality together can be used to implement this in real world. Augmented reality actually superimposes virtual objects into the real environment with the real objects for enriching the viewer’s experience Augmented reality with virtual reality in virtual space, also enhances the audience perception by displaying additional information. In this survey we present the different technologies that are involved in the implementation of augmented reality. These technologies are displays which are used for displaying or combining the virtual object by the real environment, tracking or gesture recognition helps in real time interaction part while the modelling is used to register the objects into 3D for enhancing the quality and perception of the viewer.

Introduction

I. INTRODUCTION

This is the world with lots of digital technologies, which are widespread throughout this world, the technology such as computer graphics, virtual reality and augmented reality together are used to create a world of virtual objects superimposed into the real world with the real objects. These technologies enhance the visual experiences of the audience. Augmented reality or virtual reality has been a field of research from quit a longer time. Virtual reality and augmented reality are the computer generated or simulated reality. It is basically a mixed reality in which the computer generated virtual objects or images are mixed or placed together with the real object in real world. There are contributions of various technologies together in the creation Augmented Reality. These technologies create the  environment in such a way that a person can see it, hear it, smell it, touch it or in short we can say that he or she can interact with it . This paper surveys the augmented reality and various technologies that are used in it. We present the different technique that we have studied form our survey. The technologies are Displays, trackers or gesture recognition systems and 3D modelling systems.

Display technology has been used for quite a long time for creating and displaying virtual image of an object. Whatever we see or experience in augmented reality is the work of the display. There are many types of display used in AR. The trackers and gesture recognition systems are used for real time interaction in the augmented reality. We can say that it acts as the eyes and ears of the AR system. There are many techniques used for the tracking and recognizing in real interaction of AR. For enhancing the viewer’s perception or experience the virtual object need to be registered in 3 Dimension, so for doing 3D modelling technique is used. In this 3 dimensional virtual objects are created in 3D by using different modelling software.

(AR) is used to describe a combination of technologies that enable real-time mixing of computer-generated content with live video display. AR is based on techniques developed in VR and interacts not only with a virtual world but has a degree of interdependence with the real world. When we started to focus on the human being and on his perception of the world then we realized Reality cannot be increased but its perceptions can be. We will however keep the term of Augmented Reality even if we understand it as an “increased perception of reality”. Milgram et al (1994) provides a helpful visualization to represent how reality and vitality are connected (see Figure 1). It shows a continuum that encompasses all real and virtual objects and environments.

Mixed reality is an area in the middle, where the two extremes meet, and is considered a blend of both the virtual and the real.

Unity 3D is a fully integrated development engine that provides out-of-the-box functionality for the creation of interactive 3D content. Using Unity, you can publish to multiple platforms such as PC, Web, iOS, Android and Xbox. Complete toolset, intuitive workspace and on-the-fly play testing and editing feature of Unity makes developers to save the time and effort. The Vuforia AR Extension for Unity enables vision detection and tracking Functionality within the Unity and allows developers to create AR applications.

A. Scene Processing

After calculating the spot of a specific Non-marker image in real space the system looks for the corresponding Virtual model to each no-marker in the 3D.

B. Visualization Scene

At the end, the system produces the image of the projected 3D object and present digital information.

II. AUGMENTED REALITY

Augmented reality is the reality in which the virtual objects or images are mounted with the real world objects. Augmented reality is alive view of areal-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. It is a technology that supplements the computer generated virtual objects with the real world objects, it appears as if they co-exist in the same space as real world. The field Augmented reality exist from a long period of time and it is also an evolving technology, so it is a   greater challenge to find the improved technology than the currently existing technology.

AR Technology basically works on the three aspects or characteristics:

1. Combination of real and virtual environment.
2. Real time interaction.
3. Register in 3 Dimension.

These three aspects are very important for AR technology. Combining these three will give a technology of AR with which we can actually see, talk, touch, smell, and taste. There are technologies that already exist to implement the augmented reality. There are many displays or techniques with AR can be viewed in real time, so it is not limited to one particular technology like head mounted display (HMD). As this technology includes the real time interaction, so it will require all of our senses such as sense              of sight, hearing, touch and smell. The 3D registration means the virtual images or the objects are to be created or modelled into the 3 dimensional virtual object to give a real perception of the virtual object in the real world.

For implementing AR, the main components required are Display, trackers and graphics computers and software. The display technology is used to combine the virtual objects with the real object  to achieve the augmented reality. The trackers or sensors are used to capture the gesture or the movements of the person, basically it is used to track the position and orientation for registration of the virtual with the real objects. The real time 3-dmensional interactions are done by the interface techniques.

A. Scene Identification Techniques

Scene identification classifies the scenarios. There are two basic types of scene identification techniques which are discussed as follows:

1. Marker -based: The marker-based approach uses the markers which are in the form of visual tags contained within the real scene which is perceived by the AR system.
2. Non-marker-based: The Non-marker approach uses the real object image for scene identification in device.

There are two types in non-marker based AR

• Object base.
• Location Base.           

B. Types of Augmented Reality

1. Marker Based: Markers are images that can be detected by a camera and used with software as the location for virtual assets placed in a scene. Markers based image are square and using black and white colour. Simple augmented reality markers can consist of one or more basic shapes made up of black squares against a white background

A camera is used with AR software to detect augmented reality markers as the location for virtual objects. The result is that an image can be viewed, even live, on a screen and digital assets are placed into the scene at the location of the markers.

2. Object Based: Object is nothing but any Real Image we used as scene, tracks its position, and display by playing a video or any digital information in the image’s place.

Example :

C. Problem Statement

To make learning engaging, interactive and edutaining for the smarter gen x! To make education enjoyable

Classroom-based learning within subjects like chemistry, mathematics, biology, physics, astronomy, and other K-12 education or higher education study of Science is all a complex process that includes identifying a problem, investigating the problem, making hypotheses, planning the data collection method, testing the hypotheses, collecting the data and making the conclusion and results . Participating in these processes helps the student to think critically in each step in order to gather the best results.

With the advancement in technology today, children find that learning in traditional ways are dull and boring since there are so many entertainment alternatives out there which are more interesting then  learning. Not only that, preschool children have difficulties attending classes. They feel that preschools are boring and there is less entertainment in preschool compared to their homes. Preschools have also come out with some ideas to get these children to participate in activities, storytelling, and dramas Teachers too feel that children learn faster if they can

Interact with the object or even play a role in a story or situation presented to them. They are also trying to come up with other innovative ideas which will attract the children’s attention or interest in order

To make their learning sessions more interesting, fruitful and full of fun in order to encourage the learning process. Children have difficulty learning to read and write. This is because children who do not read at a young age tend to have difficulty to develop their vital language skills. Therefore, it is very important to create an engaging environment to encourage them to learn. Children are naturally curious and playful. They learn when they explore and play, manipulate a wide range of real objects and get to see the results of their actions immediately. The way children learn is by internalizing the activities, habits, vocabulary and ideas of the members of the community in which they grow up. In other words, children learn best through primarily a social activity and participant in the social life of the school. Researchers also believe that the earlier a child starts receiving education, the faster the child can absorb more knowledge.

D. Proposed System Architecture

Unity3D development software is a very powerful development programs, and augmented reality is a very powerful technology, how to put the Unity3D game development software with augmented Reality closely unifies in together, this is what I want to say next.

1. Camera: A real-world live video is feed as an input in unity module. This live video stream is given as an input to  the Image Capturing Module.
2. Image Capturing Modules: The input to Image Capturing Module is the live video feed from the same ray of a mobile device. This module analyses the camera feed, by analysing each frame in the  video.
3. Image Processing Modules: Inputs to Image Processing Module are the images from Image Capturing Module. These images are processed using an image processing technique to detect the AR camera. Detection of AR camere is essential to determine the position, where to place the virtual object. Once the AR object is detected, its location is provided as an input to the Tracking Module.
4. Object Tracking Modules: The tracking module is “the heart” of the augmented reality system; it calculates the relative position of the camera in real time.
5. Rendering Module: There are 2 inputs to Rendering Module. First is the  calculate position from the Tracking Module and other is the Virtual Object to be augmented. The Rendering Module combines the original image and the virtual components using the calculated position and renders the augmented information on the display screen of the mobile device.

E. Display

Display technology has always been a limiting factor in the field of augmented reality. There are still many technologies that are yet to be discovered or developed, so in this section the focus is on the different existing display technologies used in augmented reality. The different types of visual displays that can be used are aural (sound) display, Haptic (touch) display are visual display. The other displays olfactory and gustatory display are yet to be developed or under implementation.

1. Aural Display: Aural display application in AR is mostly limited to self- explanatory mono (0D), stereo (1D), or surround (2D), headphones and loudspeakers. Aural displays increase information display bandwidth, promote recognition of subtle or transient information that might be overlooked in a visual display, promote attention redirection to more dynamic or interesting regions of the data, and help to preserve viewer spatial orientation when traversing large and abstract datasets . Aural display is truly found in the immersive simulations of virtual environment. Aural display is still in development stage. This type of technology is implemented to increase the sense of real is mand to enhance the user experience and interface.

2. See Through Display

See through display is an electronic display which uses the display or screen of transparent material like glass for augmented reality while even after that we can see through it. The problem with it is that the various company are still working on it. See through displays are yet to be developed which have enough brightness, field of view and contrast. There are mainly two types of see through displays. The most common of them is the video see through. In this video feed of reality replaces the virtual environment and augmented reality is overlaid on digital images. Second type is the optical see through and leaves the real-world perception alone and displays only the augmented reality overlay by using the mirrors and lenses. There are various companies that are working on this, and have achieved it to some extent. Companies such as Sony and Olympus are working on such system for watching videos and playing video games. This system has relatively less resolution, less field of view and less quality. Using this display enhances the quality of perception and experience of the user. See through displays are mostly used to make head mounted displays (HMD).

a. Video See Through: The combination of real world with the computer generated images or objects is the Augmented Reality Which can be done electronically by combining or mixing videos from camera with the computer generated images or virtual objects. This method of creating augmented reality is known as video see through. But the major challenge of video see- through display is to eliminate the parallax error which affect the performance of close-range task by ensuring that the camera and user’s eye share the same optical path.

b. Optical See Through: Optical See-through is another method of See-through display that is used to create Augmented reality. In this method the combination of real world and computer generated images or virtual objects is done by projecting this computer generated images by partially reflective mirror and directly viewing the real world. In this the virtual images or objects are not exactly the same as real ones, actually they appear as the ghost image of the real objects or we can say that the opacity of that object is low as compared with the real objects which allow us to see through it .

3. Display Positioning

Augmented reality displays can also vary depending on the position between the viewer and the real environment, based on this it can be classified into 3 types

a. Head Worn Display: Head worn display. Head worn devices are display devices, worn on the head or as a part of helmet. The display attached to it can be video see- through or optical see-trough HMD, VRD and head worn projective display. This display gives one of the best experiences of the augmented reality to the user. It is one of the best display technology among the currently existing display used for augmented reality but it has limitation that it has to be connected to the graphics computers like laptops every time that restricts its portability due to limited battery. The figure shows the example of the Head-Worn display.

b. Hand-Held Display: Hand held device is any computer or electronic device that can be carried and held in your palm. This device is portable and little bit bulkier than head worn devices. The display used in this Hand Held Display.

c. Spatial Display: Spatial display is the display that are placed apart with in the environment. In this category the display is the screen base optical see-through, video see-through and projective display. These display scan be very large and it not a monitor type of screen. It gives much better experience of augmented reality due to its big screen.it can display any number of surfaces of an indoor setting at once. The displays used in the military cockpit are the example of the spatial display

F. Interaction Based Technologies

Real Time Interaction with the AR system is done by Gesture Recognition or other tracking techniques. Gestures are the body motions that are used as the information. These motions are captured using some camera and saved in the database. The gesture recognition technique is used to recognize the motion and act upon it. Mathematical interpretation of the human motion using some computation device is called Gesture Recognition.

There are no limits on the type of gestures or it does not have any specific meaning so it is a big challenge for the gesture recognition system to understand it, so it can be classified as discrete or continuous.

1. Continuous (online) Gesture: In this type gestures are being captured while are being performed. For example, a zooming metaphor with the fingers movements.
2. Discrete (off line) Gesture: In this type the gestured are being capture after being performed. For example, gesture performed for screen or mobile.

1. Pinch Gesture: The pinch gesture can be defined as the movements of our fingers, basically expansion and contraction of fingers spread. It is one of the most common technique used in gesture recognition for devices includes video see-through and optical see-through displays and hand-held projectors. Video see-through AR act as magnifying glass just as smart phones that is used to show 3D objects for example the navigation information. Similarly, the optical see-through can be used in flash light to display medical ultrasonic imaging directly over scanned organ. Interacting with the digit interface. The zoom in and zoom out technique used in our smart phone is the best example of pinch gesture in which we expand or spread our fingers for zoom in and contract our fingers for zoom out. While interacting with the real objects, Pinch recognition is basically used to grab fabrics like small objects which resembles the grabbing and picking actions and gives signal to select and move the object in the interactive environment. In augmented reality to implement interaction the pinch is the pivotal gesture.
2. Gaze Tracking: In this tracking system the pupil of user’s eyes is tracked or observed to determine the direction of gaze. Tiny cameras are used to perform this job, they record or observe the pupil for the system. In this the challenge is that it needs to be incorporated with the eye wear, calibrated to the user to filter involuntary eye movement, and positioned at fixed distance.

G. 3D Registration

Before displaying any virtual objects of AR system in the real environment, the virtual object need to be registered or modelled in 3 dimension for better experience and view of the user. Creating the 3D virtual object is a greater challenge. There are various techniques used for modelling the virtual objects into 3D. For registration of any virtual object the system requires six Degree of Freedom (6DOF): three variables for position (x, y and z) and three angles (yaw, pitch, and roll). There three ways by which the 3D model or virtual objects can be created, first is automatic, second is semi-automatic and last is the manual. Various algorithms are involved in the creation of 3D model of virtual objects. Algorithm such as Z-Buffer, Ray tracing, and many more. Ray tracing is a rendering technique that calculates an image of a 3D real world scene by simulating the way rays of  light travel. In this algorithm the source emits the light and the camera trace the light ray. Every time an object is hit by a light ray, the colour of the surface at that point is calculated, by this the whole 3D model of    the real objects can be captured. The Z-buffer algorithm determines which part of the object is hidden and which is visible. Actually the value of Z-axis of each pixel is stored in a special memory called Z-buffer. The ray tracing algorithm is better as compared to the Z-buffer algorithm for the larger data sets. There are many other technologies such as rendering, morphing, twining, etc. this technique is used in the animation of the 3 Dimensional objects that are created using the 3D registration. There are many software used in 3D registration. The prototypes are generated by 3D registration in many fields such medical, mechanical, etc.

V. FUTURE SCOPE

Though augmented reality exists from a long period of time   but it is an evolving technology. There are many areas in this field that   still are undiscovered or requires large amount of work to be done.

Many AR demonstration generates environment which have very accurate pixel registration, but this is limited to a certain area which is carefully prepared. So the challenge in this is the advancement in the tracking system in such a way that it supports accurate registration in any unprepared area, indoors or outdoors. There is a need of new visualization algorithm to handle issues such as density, general situational awareness and  occlusion. This will give a much better and realistic AR experience.

In future the real time interaction with the virtual world need to be evolved. There should be advancements in the input and output devices used in the real time interaction with the augmented realty. Now the interaction system only uses the eyes and our hand movements in the interaction process by gesture recognition but in future person can not only see but he or she can touch or smell it.

The currently existing augmented reality system requires the expert or skilled user to calibrate and operate it. To make it common for all type of user expert and non-expert, the system need to have more robustness that will avoid calibration and setup requirements. It requires research on new auto calibration or calibration free algorithms.

To replace the currently existing techniques researchers must continue the research on developing new and better technologies in augmented reality. Display technology is the limiting technology in AR, so it needs to be developed the most. Holographic displays can be advanced in AR for using it as the display.

Conclusion

AR technology is an effective tool that integrates real learning world with virtual world created by computer\\Software and it is Easy to use and interactive. Student can acquire much more knowledge and experience in learning by using this technology in education as compared to the traditional method of learning. This technology has also increased their motivational level and the results indicated that the AR technology provided a fun and engaging environment. All the 3D models can be used again and again (reusable) without getting damaged. Therefore, it’s an effective foundation to use AR technology as an educational tool. AR features are able to engage students in learning processes and help improve their visualization skills. The features can also help teachers to explain well and make the students easily understand what they are taught

References

[1] D.W.F. van Reveled and R. Poelman “A Survey of Augmented Reality Technologies, Applications and Limitations” The International Journalof Virtual Reality, 2010, 9(2):1-20M. Young, The Technical Writer’s Handbook. Mill Valley, CA: University Science, 1989. [2] Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon julier, Blair MacIntyre “Recent Advances in Augmented Reality”, Computers & Graphics, November 2001K. Elissa, “Title of paper if known,” unpublished [3] Nivedha.S1, Hemalatha.S2 “A Survey on Augmented Reality”, International Research Journal of Engineering and Technology, 2015Milgram, Paul, and Fumio Kishino. A Taxonomy of Mixed Reality Virtual Displays. IEICE Transactions on Information and Systems E77-D, 9 (September 1994), 1321-1329.

Copyright

Copyright © 2022 Omkar Mujumdar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.