Study of Various Computer Vision Algorithms for Registration in Augmented Reality: A Survey

Abstract

Augmented reality is defined as the fusion of virtual object in real world. Registration is basic step for creating an immersive AR experience to the user. The registration algorithms helps in placing the virtually created object in the most accurate position in the real world. This paper shows an in-depth comparative analysis of various registration algorithm under computer vision that are used for AR applications. The comparative analysis helps in identifying the strengths, weaknesses and the applicability of each algorithm in various AR fields by studying different case studies and real-world applications which helps in showing the practical implication of these algorithms. Also, the challenges associated with AR are discussed and future scope of this aims at advancement of the registration algorithms so as to make the AR experience more functionable and smooth. This survey paper can help researchers, developers who have interest in knowing or exploring the field of computer vision algorithms for solving the registration problem in Augmented Reality.

Introduction

I. INTRODUCTION

Augmented Reality (AR) has become a trendy IT topic in last few years. Augmented Reality is a concept of interaction of human with a mix of real and virtual world. Augmented Reality does help in enhancing the real world by overlaying the virtual or digital information on it[1][2].It can be said that Augmented Reality is real world enhanced with virtual computer generated information in real-time[3].Augmented Reality helps enriching the environment in real time by introducing immersive sensations[4]. The phrase “real-time” differentiates Augmented reality from Virtual Reality. Virtual Reality (VR) can be defined as the virtual simulated 3D environment that feels like real to the user and gives user the feel of being there at the moment[5]. Talking about Augmented Reality, in day to day life it is being used in education sector, for learning purposes, in E-commerce, in architecture, in retail and many other industries[4]. Augmented Reality is used for viewing virtual anatomy while performing a surgery[3]. It is also used for training deep neural networks[1].

Augmented Reality helps in enhancing the awareness of the situation in hazardous outdoor conditions, it provides help with navigation in urban settings. As previously talked, about the use of AR in architecture, it can be used by virtually prototyping the architectural layout and interior design of the structure that is to be constructed[2]. The research in Augmented Reality is mainly focused on tracking and registration[6]. Registration algorithms helps in overlaying the virtual objects onto the real world and helps in making the overall experience smooth and more real to the user[7].

This paper comprises of the following: Section 1.1 talks about the key role of registration in Augmented Reality. Section 2.0 comprises of comparative analysis of different indoor registration algorithms on factors like accuracy, robustness, versatility and real time applications of these algorithms. Section 3.0 gives brief about the application of these algorithms in Augmented Reality based devices. Section 4.0 discusses about the challenges and future scope in this field. Section 5.0 shows the paper conclusion.

A. Pivotal Role of Registration in Augmented Reality

The implementation of overlaying a virtual object on to the real world is done via registration, so we can say that registration play a pivot role in placement and alignment of virtually crafted object on to the real world. Registration helps in aligning the image or virtual object with the real world[8]. It is crucial maintaining and keep track of position of virtual object with respect to the real world[8]. Registration is an important step in accuracy-chain for an AR system. The registration process allows for detection of collision, comparison of variance and distance measurement in an AR based architecture planning[9].

For Augmented Reality tracking the registration methods used are broadly divided into two types[10]:

1. Indoor Registration

• Simultaneous Localization and Mapping (SLAM): It is a featureless vision registration algorithm. The goal of the SLAM procedure is to map an unknown area while also using the sensor's data to locate a sensor system within it[11]. In can be divided into different types like Visual SLAM and RGB-D Slam, LIDAR SLAM.
• Vision Based: This involves the alignment of virtual information with real world in indoor settings. It can be implemented using markers or marker less (using features)[12]. It comprises of algorithm such as SIFT, SURF, AKAZE, PTAM. These algorithms are feature based algorithms used for registration purposes.

2. Outdoor Registration

• Vision Based: This involves aligning virtual information with real world in outdoors. This can be implemented via LIDAR integration, Feature-based methods, Global positioning system integration[13].
• Inertia Based: This is done by using information from the inertial sensors like gyroscope, accelerometer, so as to align virtual object with the real world. Some approaches of Inertia based registration are Inertial Measurement Unit (IMU), Visual-Inertial Odometry (VIO), sensor fusion techniques[14].
• GPS Based: This type of registration in done using leveraging Global Positioning System (GPS) data for the alignment of the virtual object with the real-world. Some approaches for GPS- based registration are GPS positioning, GPS-aided Vision-Based Registration, Differential GPS[15].

These are the types of computer vision algorithms that are used in registration of virtual object in Augmented Reality. The overall precision of AR application while registration basically depends on the tracking method it is using.

In the concept of computer vision, registration is referred as the alignment of various data sets, that can be images or 3-dimensional models.

II. COMPUTER VISION ALGORITHMS FOR AUGMENTED REALITY

The computer vision algorithms that are to be discussed in this paper are given below:

1. SIFT (Scale- Invariant Feature Transform): This algorithm is feature based registration algorithm that come under the category of
2. SURF (Speeded-up Robust Features)
3. ORB (Oriented FAST and Rotated BRIEF)
4. AKAZE (Accelerated KAZE)
5. BRISK (Binary Robust Independent Elementary Features)
6. NFT (Natural Feature Tracking)
7. PTAM (Parallel Tracking and Mapping)

III. COMPARATIVE ANALYSIS

The objective of this paper is to compare and analyze different computer vision algorithms used for registration in Augmented Reality. Various algorithms such as SIFT (Scale-Invariant Feature Transform)[16], SURF (Speeded-Up Robust Features)[8], ORB (Oriented FAST and Rotated BRIEF)[17], AKAZE (Accelerated-KAZE)[18], BRISK (Binary Robust Independent Elementary Features)[19], NFT (Natural Feature Tracking)[20], PTAM (Parallel Tracking and Mapping)[21] are some of the robust vision-based registration algorithms of computer vision used in Augmented Reality[22]. In this paper the above-mentioned registration algorithms are compared on the basis of different parameters like accuracy, robustness, applicability to real-time systems, handling occlusions.

Table 1: Comparative analysis of various registration algorithms

The above-mentioned table shows some of the vision-based algorithms that are used for registration. The factors on which they are compared are defined as followed:

1. Accuracy: It is the ability of the algorithm to give the result as close to the desired output.
2. Robustness: Ability of the algorithm to handle the changes in the input data.
3. Use Cases: Where the specified algorithm can be used for performing suitable task.
4. Handling Occlusions: Describes the ability of the algorithm to manage the situations where features are either partially of completely obscured.  

The selection of algorithm varies according to the need of user, for more accurate construction of 3D models PTAM algorithm can be chosen, while if the priority is real-time performance, then ORB can be selected.

IV. APPLICATIONS

In Augmented Reality, computer vision algorithms play crucial role for precise registration of the virtually created object onto the real world. Some of the real-world applications of this are mentioned below:

1. Virtual prototyping in interior designing and architecture[2].
2. The use of digitally elevated maps and predicting visual clues for registration[2][23].
3. Used in process of aligning medical images[24].
4. The use of computer vision algorithms for imager recognition and real-time tracking using different device components in games such as Pokémon Go. This help in creating and immersive gaming experience for the user[4].
5. Used in retail shopping for example in IKEA, a place where a buyer can visualize the furniture before buying in their home. Computer vision algorithms help in accurate registration of virtually created furniture.
6. Development of hardware neural network for distributed robot control[25].
7. To deliver accuracy of camera pose in AR applications[26].
8. Used in profession race car events for showing race information[21].
9. Used in games like Pokémon Go, this app uses device’s camera, gyroscope, clock and GPS for performing outdoor registration[27].

These are some of the applications of augmented reality that are done using different registration algorithms.

1. INTRODUCTION

Augmented Reality (AR) has become a trendy IT topic in last few years. Augmented Reality is a concept of interaction of human with a mix of real and virtual world. Augmented Reality does help in enhancing the real world by overlaying the virtual or digital information on it[1][2].It can be said that Augmented Reality is real world enhanced with virtual computer generated information in real-time[3].Augmented Reality helps enriching the environment in real time by introducing immersive sensations[4]. The phrase “real-time” differentiates Augmented reality from Virtual Reality. Virtual Reality (VR) can be defined as the virtual simulated 3D environment that feels like real to the user and gives user the feel of being there at the moment[5]. Talking about Augmented Reality, in day to day life it is being used in education sector, for learning purposes, in E-commerce, in architecture, in retail and many other industries[4]. Augmented Reality is used for viewing virtual anatomy while performing a surgery[3]. It is also used for training deep neural networks[1].

Augmented Reality helps in enhancing the awareness of the situation in hazardous outdoor conditions, it provides help with navigation in urban settings. As previously talked, about the use of AR in architecture, it can be used by virtually prototyping the architectural layout and interior design of the structure that is to be constructed[2]. The research in Augmented Reality is mainly focused on tracking and registration[6]. Registration algorithms helps in overlaying the virtual objects onto the real world and helps in making the overall experience smooth and more real to the user[7].

This paper comprises of the following: Section 1.1 talks about the key role of registration in Augmented Reality. Section 2.0 comprises of comparative analysis of different indoor registration algorithms on factors like accuracy, robustness, versatility and real time applications of these algorithms. Section 3.0 gives brief about the application of these algorithms in Augmented Reality based devices. Section 4.0 discusses about the challenges and future scope in this field. Section 5.0 shows the paper conclusion.

1. Pivotal Role of Registration in Augmented Reality

The implementation of overlaying a virtual object on to the real world is done via registration, so we can say that registration play a pivot role in placement and alignment of virtually crafted object on to the real world. Registration helps in aligning the image or virtual object with the real world[8]. It is crucial maintaining and keep track of position of virtual object with respect to the real world[8]. Registration is an important step in accuracy-chain for an AR system. The registration process allows for detection of collision, comparison of variance and distance measurement in an AR based architecture planning[9].

For Augmented Reality tracking the registration methods used are broadly divided into two types[10]:

1. Indoor Registration

• Simultaneous Localization and Mapping (SLAM): It is a featureless vision registration algorithm. The goal of the SLAM procedure is to map an unknown area while also using the sensor's data to locate a sensor system within it[11]. In can be divided into different types like Visual SLAM and RGB-D Slam, LIDAR SLAM.
• Vision Based: This involves the alignment of virtual information with real world in indoor settings. It can be implemented using markers or marker less (using features)[12]. It comprises of algorithm such as SIFT, SURF, AKAZE, PTAM. These algorithms are feature based algorithms used for registration purposes.

1. Outdoor Registration

• Vision Based: This involves aligning virtual information with real world in outdoors. This can be implemented via LIDAR integration, Feature-based methods, Global positioning system integration[13].
• Inertia Based: This is done by using information from the inertial sensors like gyroscope, accelerometer, so as to align virtual object with the real world. Some approaches of Inertia based registration are Inertial Measurement Unit (IMU), Visual-Inertial Odometry (VIO), sensor fusion techniques[14].
• GPS Based: This type of registration in done using leveraging Global Positioning System (GPS) data for the alignment of the virtual object with the real-world. Some approaches for GPS- based registration are GPS positioning, GPS-aided Vision-Based Registration, Differential GPS[15].

These are the types of computer vision algorithms that are used in registration of virtual object in Augmented Reality. The overall precision of AR application while registration basically depends on the tracking method it is using.

In the concept of computer vision, registration is referred as the alignment of various data sets, that can be images or 3-dimensional models.

1. COMPUTER VISION ALGORITHMS FOR AUGMENTED REALITY

The computer vision algorithms that are to be discussed in this paper are given below:

1. SIFT (Scale- Invariant Feature Transform): This algorithm is feature based registration algorithm that come under the category of
2. SURF (Speeded-up Robust Features)
3. ORB (Oriented FAST and Rotated BRIEF)
4. AKAZE (Accelerated KAZE)
5. BRISK (Binary Robust Independent Elementary Features)
6. NFT (Natural Feature Tracking)
7. PTAM (Parallel Tracking and Mapping)

1. COMPARATIVE ANALYSIS

The objective of this paper is to compare and analyze different computer vision algorithms used for registration in Augmented Reality. Various algorithms such as SIFT (Scale-Invariant Feature Transform)[16], SURF (Speeded-Up Robust Features)[8], ORB (Oriented FAST and Rotated BRIEF)[17], AKAZE (Accelerated-KAZE)[18], BRISK (Binary Robust Independent Elementary Features)[19], NFT (Natural Feature Tracking)[20], PTAM (Parallel Tracking and Mapping)[21] are some of the robust vision-based registration algorithms of computer vision used in Augmented Reality[22]. In this paper the above-mentioned registration algorithms are compared on the basis of different parameters like accuracy, robustness, applicability to real-time systems, handling occlusions.

Table 1: Comparative analysis of various registration algorithms

The above-mentioned table shows some of the vision-based algorithms that are used for registration. The factors on which they are compared are defined as followed:

1. Accuracy: It is the ability of the algorithm to give the result as close to the desired output.
2. Robustness: Ability of the algorithm to handle the changes in the input data.
3. Use Cases: Where the specified algorithm can be used for performing suitable task.
4. Handling Occlusions: Describes the ability of the algorithm to manage the situations where features are either partially of completely obscured.  

The selection of algorithm varies according to the need of user, for more accurate construction of 3D models PTAM algorithm can be chosen, while if the priority is real-time performance, then ORB can be selected.

V. APPLICATIONS

In Augmented Reality, computer vision algorithms play crucial role for precise registration of the virtually created object onto the real world. Some of the real-world applications of this are mentioned below:

1. Virtual prototyping in interior designing and architecture[2].
2. The use of digitally elevated maps and predicting visual clues for registration[2][23].
3. Used in process of aligning medical images[24].
4. The use of computer vision algorithms for imager recognition and real-time tracking using different device components in games such as Pokémon Go. This help in creating and immersive gaming experience for the user[4].
5. Used in retail shopping for example in IKEA, a place where a buyer can visualize the furniture before buying in their home. Computer vision algorithms help in accurate registration of virtually created furniture.
6. Development of hardware neural network for distributed robot control[25].
7. To deliver accuracy of camera pose in AR applications[26].
8. Used in profession race car events for showing race information[21].
9. Used in games like Pokémon Go, this app uses device’s camera, gyroscope, clock and GPS for performing outdoor registration[27].

These are some of the applications of augmented reality that are done using different registration algorithms.

Conclusion

The comparison analysis of various vision-based computer vision algorithm for registration was done. It was concluded that algorithms vary in their accuracy and robustness, depending on the input provided to them. How an algorithm responds to different type of occlusion was also compared, as it is important to know about the working of algorithm in such situations. The use cases of these algorithms were also discussed. An overall understanding of the algorithm and their characteristics, along with their thorough testing can help in implementation of various application in real world. Therefore, the selection of algorithm for registration depends on the requirements of the user. If on algorithm is considered good for its one application, it might not be best for fulfilling some different condition.

References

[1] H. Abu Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger, and C. Rother, “Augmented Reality Meets Computer Vision: Efficient Data Generation for Urban Driving Scenes,” Int. J. Comput. Vis., vol. 126, no. 9, pp. 961–972, 2018, doi: 10.1007/s11263-018-1070-x. [2] R. Behringer, “Registration for outdoor augmented reality applications using computer vision techniques and hybrid sensors,” Proc. - Virtual Real. Annu. Int. Symp., pp. 244–251, 1999, doi: 10.1109/vr.1999.756958. [3] J. Carmigniani and B. Furht, Handbook of Augmented Reality. 2011. doi: 10.1007/978-1-4614-0064-6. [4] S. Dargan, S. Bansal, M. Kumar, A. Mittal, and K. Kumar, “Augmented Reality: A Comprehensive Review,” Arch. Comput. Methods Eng., vol. 30, no. 2, pp. 1057–1080, 2023, doi: 10.1007/s11831-022-09831-7. [5] Wohlgenannt, A. Simons, and S. Stieglitz, “Virtual Reality,” Bus. Inf. Syst. Eng., vol. 62, no. 5, pp. 455–461, 2020, doi: 10.1007/s12599-020-00658-9. [6] L. Feng, N. G. Weng, and L. Ma, “3D Point Cloud Data Registration Algorithm Based on Augmented Reality Technology,” Wirel. Commun. Mob. Comput., vol. 2023, 2023, doi: 10.1155/2023/5169781. [7] Y. Xia and W. Zhou, “and KLT for Augmented Reality System”. [8] S. M. Hossein, “Utilizing SURF Features and KLT Tracking Algorithm in Augmented Reality ( AR ), Using Kinect V . 2 with the Aim of Autism Therapy,” Seyed Muhammad Hossein Mousavi, no. February, 2020. [9] K. Pentenrieder and P. Meier, “Registration approaches for augmented reality - A crucial aspect for successful industrial application,” GRAPP 2008 - Proc. 3rd Int. Conf. Comput. Graph. Theory Appl., pp. 426–431, 2008, doi: 10.5220/0001099704260431. [10] A. Dad, B. Arora, N. Parker, and T. Rachh, “Augmented Reality: Tracking Methods,” Int. J. Eng. Res. Technol., vol. 5, 2018, [Online]. Available: https://api.semanticscholar.org/CorpusID:88485757 [11] A. Macario Barros, M. Michel, Y. Moline, G. Corre, and F. Carrel, “A Comprehensive Survey of Visual SLAM Algorithms,” Robotics, vol. 11, no. 1, 2022, doi: 10.3390/robotics11010024. [12] S. ?ukovi?, M. Gattullo, F. Pankratz, G. Devedži?, E. Carrabba, and K. Baizid, “Marker based vs. natural feature tracking augmented reality visualization of the 3D foot phantom,” pp. 24–31, 2015. [13] A. Thangarajah, J. Wu, B. Madon, and A. K. Chowdhury, “Vision-based registration for augmented reality-a short survey,” in 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), 2015, pp. 463–468. doi: 10.1109/ICSIPA.2015.7412236. [14] Y. Hao, M. He, Y. Liu, J. Liu, and Z. Meng, “Range–Visual–Inertial Odometry with Coarse-to-Fine Image Registration Fusion for UAV Localization,” Drones, vol. 7, no. 8, pp. 1–18, 2023, doi: 10.3390/drones7080540. [15] C. Santos, T. Araújo, J. Morais, and B. Meiguins, “Hybrid Approach Using Sensors, GPS and Vision Based Tracking to Improve the Registration in Mobile Augmented Reality Applications,” Int. J. Multimed. Ubiquitous Eng., vol. 12, no. 4, pp. 117–130, 2017, doi: 10.14257/ijmue.2017.12.4.10. [16] J. Wu, Z. Cui, V. S. Sheng, P. Zhao, D. Su, and S. Gong, “A comparative study of SIFT and its variants,” Meas. Sci. Rev., vol. 13, no. 3, pp. 122–131, 2013, doi: 10.2478/msr-2013-0021. [17] C. Luo, W. Yang, P. Huang, and J. Zhou, “Overview of Image Matching Based on ORB Algorithm,” J. Phys. Conf. Ser., vol. 1237, no. 3, 2019, doi: 10.1088/1742-6596/1237/3/032020. [18] L. Kalms, K. Mohamed, and D. Göhringer, “Accelerated Embedded AKAZE Feature Detection Algorithm on FPGA,” ACM Int. Conf. Proceeding Ser., pp. 3–8, 2017, doi: 10.1145/3120895.3120898. [19] S. Leutenegger, M. Chli, and R. Y. Siegwart, “BRISK: Binary Robust invariant scalable keypoints,” Proc. IEEE Int. Conf. Comput. Vis., pp. 2548–2555, 2011, doi: 10.1109/ICCV.2011.6126542. [20] U. Neumann and S. You, “Natural feature tracking for augmented reality,” IEEE Trans. Multimed., vol. 1, no. 1, pp. 53–64, 1999, doi: 10.1109/6046.748171. [21] A. H. Basori, F. N. Afif, A. S. Almazyad, H. A. S. AbuJabal, A. Rehman, and M. H. Alkawaz, “Fast Markerless Tracking for Augmented Reality in Planar Environment,” 3D Res., vol. 6, no. 4, pp. 1–11, 2015, doi: 10.1007/s13319-015-0072-5. [22] S. A. K. Tareen and Z. Saleem, “A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK,” in 2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), 2018, pp. 1–10. doi: 10.1109/ICOMET.2018.8346440. [23] M. A. Audette, F. P. Ferrie, and T. M. Peters, “An algorithmic overview of surface registration techniques for medical imaging,” Med. Image Anal., vol. 4, no. 3, pp. 201–217, 2000, doi: 10.1016/S1361-8415(00)00014-1. [24] H. Li and R. Hartley, “The 3D-3D registration problem revisited,” Proc. IEEE Int. Conf. Comput. Vis., 2007, doi: 10.1109/ICCV.2007.4409077. [25] I. J?kobsone, “ The 2 nd International Conference ‘Nutrition and Health’ ,” Proc. Latv. Acad. Sci. Sect. B. Nat. Exact, Appl. Sci., vol. 72, no. 2, pp. 128–130, 2018, doi: 10.2478/prolas-2018-0021. [26] G. Hua and X. S. Hua, Mobile cloud visual media computing: From interaction to service. 2015. doi: 10.1007/978-3-319-24702-1. [27] S. Minaee, X. Liang, and S. Yan, “Modern Augmented Reality: Applications, Trends, and Future Directions,” 2022, [Online]. Available: http://arxiv.org/abs/2202.09450 [28] A. W. Fitzgibbon, “Robust registration of 2D and 3D point sets,” Image Vis. Comput., vol. 21, no. 13–14, pp. 1145–1153, 2003, doi: 10.1016/j.imavis.2003.09.004.

Copyright

Copyright © 2024 Sachin Sharma, Deepak Thakur, DR. A J Singh. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.