Applications of Augmented Reality, Virtual Reality and Mixed Reality in Urological Interventions: A Systematic Review

Abstract

Purpose of review: To highlight and review the Applications of Augmented Reality, Virtual Reality and Mixed Reality in Urological Interventions. The systematic review was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) checklist. Recent Findings: This review discusses the newer advancements in Augmented Reality, Virtual Reality and Mixed Reality in Urological Interventions management strategies that holds great promise to provide an essential step for personalized patient care and improved decision making. Summary: The constantly evolving literature and future research should focus more on Augmented Reality, Virtual Reality and Mixed Reality in Urological Interventions that can be used universally, to guide urologists.

Introduction

I. INTRODUCTION

Surgical training has traditionally been one of apprenticeship, where the surgical trainee learns to perform the surgery under the supervision of a trained surgeon. This is costly, time consuming, and is of variable effectiveness. Training using virtual reality simulator (a computer simulator) is an option to supplement standard training. A Cochrane analysis in 2009 shows that practice in a VR, MR and AR simulated environment is at least as good as practice in a more traditional and accepted ‘‘box-trainer’’ environment and concludes that individuals with such training have shorter operation time and make fewer errors. [1]. Virtual, augmented, and mixed reality advances have ushered in a slew of new consumer products. The idea of using AR, VR, MR in health care applications has been proposed for many years.[2] Recent advances in VR technology also support the creation, application, evaluation, and delivery of interactive VR applications at a lower cost.[3] one of the characteristics that can define VR is immersion, since it can transport the user to a completely virtual world, being able to move away from physical reality. Therefore, this immersion is done through sight and hearing.[4] Increasingly complex surgical procedures, intraoperative visualizations are becoming an important part of surgical environments. In this regard, augmented reality (AR) is a concept under wide investigation. AR research and development has made rapid progress in the last few decades, moving from research laboratories to widespread availability on consumer devices.[5] The difference for the user is the level of immersion. With AR, a large majority of what the user sees is still the real world whereas with VR, the user is fully immersed into a virtual environment. [6] These new technologies are expected to solve a major information deficit evident in current surgical procedures by providing see-through vision of hidden objects under organ surfaces. [7] In the field of urology in, the traditional way of education and apprenticeship is evolving with times and technology. The conventional method of teaching surgery by “see one, do one, teach one” has radically changed with new technologies. Simulations have become a constant element of education and training - as well as augmented (AR) and virtual reality (VR) [8]. The transition between mixed reality, augmented reality (AR), augmented virtuality (AV), and virtual reality (VR) is continuous. These terms were first described by Paul Milgram in 1994 who consequently coined the notion of mixed reality (MR), to subsume all possible applications in the reality-virtuality (RV) continuum. In this study, we describe the current capabilities and future challenges of mixed reality, virtual reality, and augmented reality. The analysis aims at raising awareness concerning the potential and, more importantly, the demands of using MR, AR, VR in a clinical setting.[9]

II. METHODS

A total of 225 articles was initially identified. Various articles were found using keywords like “Virtual Reality”, “Augmented Reality”, “Mixed Reality”, “Urological Intervention”, “Telehealth”, “Tele-Mentoring”, “Telesurgery”. Further relevant articles were shortlisted by filtering out articles which were related to the field of Urology and Medicine.

III. SEARCH STRATEGY AND ARTICLE SELECTION

A review of all English language literature published in the last 2 decades (2001-2021) was conducted in using MEDLINE, Scopus, CINAHL, Clinicaltrials.gov, EMBASE, Cochrane library, Google Scholar, and Web of Science. A dedicated search string was then created based on a combination if the following keywords: “Augmented reality”,” Mixed Reality”, “Virtual Reality”, “Urological Intervention”, “Telehealth”, “Tele-Mentoring”, “Telesurgery”. The systematic review was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) checklist. Only original articles in the English language were included.   

A. Inclusion Criteria

1. Articles on Augmented Reality, Virtual Reality, Mixed Reality, Urological Intervention, Telehealth, Tele-Mentoring, Telesurgery
2. Full-text original articles on all aspect of diagnosis, treatment, and outcomes of Urological interventions

B. Exclusion Criteria

1. Animal, laboratory, or cadaveric studies
2. Reviews, Editorial, Animal and Laboratory studies

The literature review was performed according to the inclusion and exclusion criteria. The titles and abstracts were evaluated and after the screening, analysis of the full article text was conducted for selected articles that met the inclusion criteria. The references list of the selected articles was individually and manually reviewed to screen for additional articles of interest. Disagreements about eligibility were resolved by discussion for a consensus decision.

IV. RESULTS

A. Evidence Synthesis

The initial search identified a total of 225 unique articles. From this list, 170 articles remained following the initial screening the articles removed didn’t meet with the inclusion criteria, with 75 remaining after a further screening of the abstracts. The removed articles were found not suitable to the requirements of this paper.After additional review of the full-text articles, a total of 15 articles were identified that met our inclusion criteria and were subsequently included in the final review as per PRISMA

V. DIFFERENCE BETWEEN AUGMENTED REALITY, MIXED REALITY, AND VIRTUAL REALITY

1. Augmented Reality: An augmented reality system provides the surgeon with computer-processed imaging data in real-time via dedicated hardware and software. The projection of AR is made possible by using displays, projectors, cameras, trackers, or other specialized equipment.[10]
2. Mixed Reality: Both augmented reality and virtual reality devices show promise, however there are significant limitations that have slowed their widespread adoption. These include the inability to interact with three-dimensional data packets in augmented reality, and exclusion of the real-world environment in virtual reality. Mixed reality merges many of the benefits of virtual reality and augmented reality. [11]
3. Virtual Reality: VR is broadly defined as an environment where users can accept and respond to artificial stimuli in a natural way. [12]

The fundamental difference is that VR users have an entirely virtual experience, while virtual elements are added to AR users’ real-world experience. MR users can interact with these added virtual elements during their real-world experiences.

VI. APPLICATIONS OF AUGMENTED REALITY, VIRTUAL REALITY, MIXED REALITY

Many typical urological treatments are conducted employing MIS techniques, utilising endourological approaches, making urology a good fit for simulative training methods (e.g., cystoscopy, TURP, PCNL, URS, laparoscopy, or robotic-assisted procedures). These operative techniques can be replicated with considerable authenticity with current MIS simulation technologies. Virtual reality-based simulation for flexible cystoscopy (VR URO-Mentor) has been shown to improve time to procedure completion and decreases rates of trauma. Schout et al. demonstrated that trainees who performed simulation-based cysto-urethroscopy (via the URO Mentor VR) performed significantly better than those who did not undergo simulation-

based training when subsequently operating on patients.[13]

The ideal manifestation of AR technology would be centred around mobile technology or in a head mounted display, allowing the surgeon to visualize the anatomical target within the field of view rather than using an alternative screen [14]

VII. AUGMENTED REALITY

A. Augmented Reality in Telesurgery

There are many cloud-based AR platform that allows real-time collaboration between a local (operating) surgeon and a remote (assisting) surgeon using a birds eye view of the operative field. Both surgeons communicate by means of a 2-way audio stream, using several integrated AR features to further clarify the advice provided. For example, the remote surgeon can proctor their own hands into the virtual surgical field using a webcam or highlight important structures using a range of annotation and drawing tools.[15]

One of the drawbacks is the requirement for dedicated communication networks to control the time latency in the loop in which a surgeon performs an action until the result becomes visible. This latency has been the subject of research and should be below 105 msec to avoid deterioration of the performance and user experience [16]

This can presently only be achieved using private networks. Research using the public Internet and securing high reliability as well as low latency was conducted by Obenshain and Tantillo [17] using an LTN overlay network and a da Vinci robot as the tele surgical device. Progress in image compression technologies and network capabilities would be expected to open this field to wider applications [18].

B. Augmented Reality in Tele-mentoring

Tele-mentoring can improve treatment of combat trauma injuries by connecting remote experienced surgeons with local less-experienced surgeons in an austere environment. Current surgical tele-mentoring systems force the local surgeon to regularly shift focus away from the operating field to receive expert guidance, which can lead to surgery delays or even errors. The System for Tele-mentoring with Augmented Reality (STAR) integrates expert created annotations directly into the local surgeon’s field of view. The local surgeon views the operating field by looking at a tablet display suspended between the patient and the surgeon that captures video of the surgical field. The remote surgeon remotely adds graphical annotations to the video. The annotations are sent back and displayed to the local surgeon while being automatically anchored to the operating field elements they describe. A technical evaluation demonstrates that STAR robustly anchors annotations despite tablet repositioning and occlusions. In a user study, participants used either STAR or a conventional tele-mentoring system to precisely mark locations on a surgical simulator under a remote surgeon’s guidance. Participants who used STAR completed the task with fewer focus shifts and with greater accuracy. The STAR reduces the local surgeon’s need to shift attention during surgery, allowing him or her to continuously work while looking “through” the tablet screen.[15]

In 2002, Rassweiler et al. [20] published a study of research techniques relating to urological laparoscopy. Many of the described problems involving handling problems and hardware could be resolved by new technologies such as the higher resolution provided by 4k cameras [22] and AR displays with stereoscopic capabilities, thereby overcoming the use of shutter glasses and heavy video helmets

C. Navigation using Augmented Reality

With increasingly complex surgical procedures, intraoperative visualizations are becoming an important part of surgical environments. In this regard, augmented reality (AR) is a concept under wide investigation. It is expected to solve a major information deficit evident in current surgical procedures by providing see-through vision of hidden objects under organ surfaces.[22]

The evolution of AR technology is accelerating, and the rapidly growing market requires an increasing number of skilled professionals.[23]

In AR-based surgical navigation systems, patient-specific 3-D models commonly generated from preoperative images (e.g., CT, MRI) are superimposed on the real views of the surgical field to provide surgeons with improved visualizations of the anatomical structures and/or to assist them throughout the procedure.[19]

Teber et al. evaluated in 2009 initially in ten porcine kidneys a novel soft tissue navigation system developed to enhance the surgeon’s perception and to provide decision-making guidance before the resection phase during laparoscopic PN. The setup consisted of custom-designed navigation aids, a mobile C-arm allowing for cone-beam imaging, and a computer. After the pre-clinical experience, ten patients underwent AR-guided retroperitoneoscopic PN. The error between the real-time images and the virtual superimposed ones was equal to 0.5 mm. Operative time averaged 165 min, with a mean time for tumour localization=20 min.

Mean warm ischemia time was 25 min (range 12–32 min), with 240 ml mean blood losses (range 150–330 ml). No complications were recorded. All patients had negative surgical margins. [24]

 In a study by Hughes-Hallett et al of over 60 cases, it was shown that surgery is undergoing an important change with a movement towards minimal invasive surgery (MIS) becoming a new standard of care . The proposed 3D image-guided surgery relies on 2 phases: planning and execution. Although the first phase requires a large amount of anatomical data, the second only requires a subset of this information, but of much higher accuracy. The 3D models needed during the planning phase are visualized to the surgeon both on a tablet and a daVinci console. [25]

1. Navigation During Urological Interventions

Navigation efforts for urinary disorders are mainly focused on nephrolithotomy (urolithiasis) and nephrectomy (renal cell carcinoma). For these applications, (cone-beam) computed tomography (CT) – the modality proposed for these indications in the European Association of Urology guidelines – is the main modality used for the creation of the roadmap that provides the basis for the navigation process. [26]

Nakamura et al. used an image-processing application to create 3D volume-rendered images of the renal vessels, kidneys, and surrounding organs. Using image-capture software, these photos were fused into live video at the same time.[27] Two laparoscopic radical nephrectomies and three laparoscopic radical nephrectomies. This technology was used to do laparoscopic PNs. In less than 2 minutes, the images were merged. The average operative time is 210 minutes (range 158–263 minutes) and 40 ml of blood were lost. (About 10–105 mL)

2. Navigation During Percutaneous Nephrolithotomy

Rassweiler et al. has clinically applied navigation technologies during nephrolithotomy using a futuristic iPad tablet-assisted guidance approach. Using CT-derived roadmaps, this approach provides a movable two-dimensional augmented reality ‘window into the body,’ thereby facilitating the optimal access site for needle puncture. Roadmap-to-patient registration was realized using a vision-based tracking setup, encompassing the two-dimensional camera of the iPad and radiopaque coloured fiducial markers, placed on the patient’s skin. During needle puncture, fluoroscopy allowed adjustments to be made from the plotted needle course. A comparison to traditional ultrasound fluoroscopy combinations, yielded a similar outcome, but also revealed that significantly higher X-ray doses and longer puncture times occurred with the iPad approach. Incorporation of electromagnetic needle tracking was proposed as a future refinement. In a porcine study, Rodrigues et al. demonstrated that the incorporation of an electromagnetic sensor at both the needle and catheter-tip allowed needle placement to be performed by relying on the electromagnetic tracking and vision of the ureteral fiberscope only. Integration of electromagnetic needle tracking and the iPad approach could thus help limit the X-ray dose while allowing visualization of the needle trajectory with respect to the organ models. In a similar percutaneous setup, interventional radiology (that is ablations of lesions in the kidney) indicated that electromagnetic based navigation of ultrasound devices in preintervention ally acquired CT or MRI image planes improves the accuracy of needle placement. [27]

3. Navigation During Nephrectomy

During (robot-assisted) laparoscopic partial nephrectomy [(RA)LPN], various forms of CAS have been applied. Furukawa et al. [27] used a CT-based three-dimensional virtual reality roadmap of the kidney, tumor, and vasculature that facilitates selective arterial clamping presented in the surgeon’s console parallel to the laparoscopic feed using the TilePro function. A comparison of this approach with traditional clamping procedures not only indicated virtual reality assistance was feasible, it was also shown to reduce the decrease in estimated glomerular filtration rates early after surgery. Here, the use of an iPad can help facilitate the interaction with the virtual reality model, thereby improving the intraoperative appreciation of the hilar vascular anatomy. Alternatively, Wang et al.  describe the use of a manually positioned two-dimensional augmented reality overlay, presenting a CT-model in the laparoscopic feed (i.e. static screenshots). Comparisons between patients receiving augmented reality guidance and those that went without, suggested the technology could help reduce the operating time and reduce blood loss values. For open liver surgery, similar manual two-dimensional augmented reality overlays have been helpful in localizing multiple tumor lesions within the organ. Ukimura and Gill used a near-infrared (NIR) optical tracking system (OTS) to support the automatic alignment of such augmented reality CT models in the two-dimensional laparoscopic feed, during (non-robotic) LPN. [27] Simpfendörfer et al. introduced a novel idea for navigated laparoscopic interventions in 2016 that combined intraoperative cone-beam CT scans for increased depth perception with marker-based AR guiding to create a novel concept for navigated laparoscopic treatments. [28]

D. Prostate Surgery using Augmented Reality

Rassweiler's team announced the first human in vivo implementation of an AR navigation system during RP in a 54-year-old man with biopsy-proven prostate Gleason 3+4 cancer in 2011.[29] The operation took a total of 205 minutes. There were no difficulties. With the navigation assistance, RP was done with negative surgical margins. Changes in the spatial layout of the navigation aids were identified in the case of tissue deformations, protecting the system from erroneous navigation depiction. The authors verified that the virtual 3D model was correctly superimposed on top of the real navigation aids for 98 percent of the surgery time by superimposing the virtual navigation aids on top of the real navigation aids.

E. Exploratory pre?clinical AR Studies

Müller et al. examined a novel mobile AR system utilised during percutaneous access for PCNL in a feasibility study in 2013.[30] A tablet was placed above the patient, with the camera aimed at the intervention area. Fiducial markers were used to register the images from the tablet camera with the CT image. Structures of interest were semi-transparently placed on the video images. On kidney phantoms, an urologist trainee and two professionals performed 53 punctures. In terms of puncturing time, the trainee excelled with the AR system, while the specialists excelled with fluoroscopy. The authors measured significantly less radiation exposure using AR guidance.

VIII. VIRTUAL REALITY

The field of VR training in healthcare is growing exponentially. The Virtual Reality Modelling Language was first introduced in 1994 and was intended for the development of ‘virtual worlds’ without dependency on headsets. All modern VR displays are based on technology developed for smartphones, including gyroscopes and motion sensors for tracking head, hand, and body positions; small HD screens for stereoscopic displays; and small, lightweight, and fast processors.[23]

Visualization technology in the OR has continued to change the surgical workflow and surgical training process. It has provided an opportunity for multimodal integration that allows surgeons to efficiently gather all the relevant information needed to create a complete mental picture.[31]

The field of VR and simulation covers many different types. The most important and commonly used technologies are presented in this section with their features and limitations: ubiquitous, immersive, interactive, haptic and therapy [8]

A. Ubiquitous Systems

One of the simplest and vastly used virtual training systems is E-learning. They represent a ubiquitous source of learning materials and courses which can be accessed from anywhere at any time and can therefore be considered virtual classrooms. Knowledge and lectures can be accessed without instructors and assessments are easily implemented.

They can be part of a curriculum as an online extension of a university or urology education institution, or they can be a service offered by the different associations in the urology sector with the aim of promoting new technologies and practices available to their members and urologists all over the world. Although great advantages such as the ease of access and the broad field of subjects that can be covered, the E-learning systems have obvious limitations: Even with conventional videos and text, the implication of the user and learner is restricted to theory. The technology simulates learning, but not the actual procedure and action necessary in urology [8]

B. Immersive Systems

The theoretical aspect and the lack of immersion of the previous technology can be overcome by the usage of VR360 degree videos. These panoramic movies represent immersive VR experiences. They are recorded with omni-directional cameras either in stereoscopic 3-dimensional (3D) or 2-dimensional.

The videos are usually watched using VR headsets or HMDs. These exist in 2 variations: the first is a fully dedicated device. The second variation is an often more affordable solution for presenting VR360 videos: It consists of a smart phone in combination with a cardboard type of HMD. The sensors in the smart phone are used to calculate the appropriate viewing angle for the user. Recent research has shown that both variations of this technology provide similar learning results. Two applications of this immersive technology are worth mentioning: One in the field of educational promotion, the other in the domain of immersive urological training videos. [8]

C. Interactive Systems

Virtual labs are dedicated game like computer programs that simulate handling and physics in a certain environment. They are often used on laptops and classroom screens when a real lab is too costly or not available. Virtual labs take advantage of the performance of modern computers in combination with powerful GPUs. Virtual labs are generally used in undergraduate education and in institutions which teach many students and do not have the equipment available e.g., to perform large scale chemical, biotechnology, or physics experiments. Students can perform tutorials and exercises like a game on their web browsers. Assessment of the courses is done inside the lesson sessions. Studies have shown that such gamified simulations in virtual labs can improve education.[8]

D. Haptic Systems

Accurate and precise simulations are usually provided by devices referred to as box simulators or bench top simulators. They consist of dedicated devices that aim at accurately and precisely reproducing anatomic and haptic environments of the procedure to simulate. Urologist surgeons need training with simulators to get familiar for example with the laparoscopic and endoscopic skills. The use of simulators helps urologist surgeons to improve their skills and to acquire more experience outside of the often dangerous and costly environment of the operating room. A recent example can be also observed in the programmable pneumatic prostate palpation simulator. This new device allows the urologist apprentice to practice the detection of various diseases with realistic haptic feedback. The simulator provides high level haptic realism and can generate multiple stages of anomalies for training. [8]

IX. MIXED REALITY

A. Mixed Reality in Tele-mentoring

ARTEMIS (Augmented Reality Technology to Enable remote Integrated Surgery), an immersive AR-VR tele-mentoring infrastructure that allows experienced surgeons to remotely aid less experienced medical professionals in the field. ARTEMIS provides Mixed Reality immersive visual aids by tracking a patient in real-time and showing a reconstructed 3D point cloud in a VR environment; expert surgeons can interact with the 3D point cloud representation of the patient, instruct the remote novice through real-time 3D annotations projected in AR on the patient’s body, using hand-manoeuvres shown in AR through an avatar of the expert surgeon, and by projecting small video clips of specific procedures in the AR space for the novice to follow.[23]

1. MR Technologies: The main requirements for any AR/VR system are a tracking system that estimates the device location within the application environment and display systems that use this location to register realistic virtual content on the real environment in case of AR or to navigate a virtual world in case of VR. [9]
2. Display/Visualization: MR display technologies can be technically split into two main sub-types according to the hardware used to fuse virtual and real content: optical see-through (OS) and video see-through (VS). For OS, transparent screens allow the user to perceive reality directly, while VS reality is captured by the camera(s) and the resulting video stream is augmented and displayed. Examples of OS displays are the HoloLens, MagicLeap, DAQRI smart glasses, amongst others. Examples for VS displays are HTC VIVE Pro, Oculus Rift S/Quest, VRgineers XTAL, amongst many others.
3. Technological Challenges for AR/MR: Current AR/MR systems have been shown to perform well in static scenarios and specifically prepared environments. An OR can be an example of such a controlled environment in some cases. A large number of applications, however, need robust tracking functionality in highly dynamic scenes requiring SLAM systems. Scene understanding is a key challenge for tracking systems. Most existing SLAM systems create a sparse map of their environment. Such a geometric model representation is useful for localization. However, for AR/MR, a dense scene mapping is preferred as it allows for the full interaction of virtual and real objects. Further capabilities will be unlocked when a semantic understanding of scenes is achieved in addition to the pure geometric one. The fusion of SLAM system maps with the output of deep neural networks providing segmentation and labelling is an active research topic in computer vision. Apart from the tracking technologies, scalability is another major issue for AR. Systems that function in any environment, without requiring many preparatory steps (e.g., calibration) and manual setup are required. [9]
4. Surgery: MR can provide significant support at various levels when performing surgery. For example, VR can be effectively used in surgical planning. Urologists were presented with a 3D model of the anatomy in scope along with pathological annotations and additional images like CT, MRI, or PET. The purpose of using VR technology in this setting was to present three-dimensional data using a dedicated system, instead of projecting it onto a two-dimensional display. Since a patient specific planning can be performed exploiting only on pre-acquired data, a registration between surgeon, patient and tools is not necessary. The main aspects to accept such technology are ease of use, quality of presentation, and interactive capabilities to support the planning process. Antonelli et al. used holographic reconstructions and report about the usefulness of three-dimensional preoperative planning before partial nephrectomy. In another study, it was shown that three-dimensional holograms in MR can be used for the preoperative planning before nephron sparing surgery . In a recent study, interactive VR renal models were used in preoperative planning before laparoscopic donor nephrectomy. The authors observed that the operative time was reduced, the donor's results improved, and the patients' preoperative anxiety decreased.

Moving on from pure VR, AR can be used to replace traditional display technologies. Al Janabi et al. describe a system to replace the monitors for an endoscopic surgery by a head mounted display (HoloLens). They report about significant benefits when applied in a synthetic training scenario with novice, intermediate and expert surgeons.[25]. Borgman et al. demonstrated that the use of smart glasses was safe and feasible in 31 AR-assisted urological surgeries performed using smart glasses.[30]

Surgical robots can take MR a huge step further. Taking the da Vinci Surgical System as an example, the aforementioned VS technology is used. The surgeon uses a special operating console featuring a fixed stereoscopic view as well as in-hand manipulators to operate surgical tools. All surgical tools will be rigidly attached to the robotic arms on the patient cart. MR approaches tremendously benefit from such a setup. Since the surgeon's view and the means of interaction are fixed, and the position of cameras and tools are known in advance, precise and stable calibration/registration is possible. This allows for accurate visualization of stereoscopic information, even for multiple surgeons at the same time. Due to the strictly controlled scenario, surgical robots can provide MR solutions of highest quality. Schiavina et al. reported that the AR-3D guided surgery can be used to improve intraoperative “real time navigation” to identify the index lesion in robot-assisted radical prostatectomy. Moreover, nerve-sparing surgery approach during robot-assisted radical prostatectomy can be regulated through AR-3D guidance. Additionally, surgical robots have the potential to provide haptic feedback to the surgeon.

Unfortunately, robot-assisted surgery has not proven beneficial under all circumstances. In most cases traditional surgery (including minimal invasive surgery) by an expert is still preferred. In traditional surgery the application of AR/MR is very involved, since the surgeon, the patient and consequently the surgical field, the surgical tools, and the displays (HMD, monitors, projectors, etc.) need to be calibrated and tracked. Especially challenging in urology is the fact that most anatomy is comprised from soft tissue which easily deforms due to incisions, pressure, or just under the influence of gravity.

Kong et al. reported in on the use of fluorescent gold fiducials for ex vivo and in vivo experiments in a pig. The fiducials have a helix-shape preventing migration once inserted into the kidney. Their fluorescent coating can be detected in the near-infrared spectrum during surgery and the markers can be reliably identified in CT images. Due to their specific shape and optical properties, the fiducials can be recognized in pre- and perioperative images maximizing computer-based support during kidney surgery. The reported accuracy under deformation was below 1 mm. Regarding the location and borders of small solid renal tumours, the procedure can be beneficial to verify real-time visual information [31]. Further research needs to clarify whether adverse effects are to be expected using the markers. Inserting the markers into the kidney clearly has the technical advantage that internal deformations can be captured.

In some cases, custom markers and therewith also AR can be used in elegant ways as reported by Yu et al. Their main goal is to protect the urethra from damage during MIS. The challenge is to provide reliable information about the urethra position during MIS through augmentation of endoscopic images. [32] Their solution is to use a surface-lighting plastic optical fibre and to illuminate it using coded light. The fibre is inserted into the urethra and thus encodes its position over its complete length. An automatic process can thus extract and mark the urethra position in the endoscopic images.[9]

X. STRENGTH, LIMITATIONS, AND AREAS OF FUTURE RESEARCH

Augmented Reality, Virtual Reality, Mixed Reality has been used in all areas of Urology including diagnosis, for predicting treatment suitability and success, basic science. However, it is still a research-based tool and is not used universally in clinical practice. This could be due to a lack of data infrastructure needed to train the algorithms, wider applicability in all groups of patients, complexity of its use, and cost involved with it. Future AR, VR and MR studies should also focus more on cost of Urological treatment and come up with common algorithms that can be used universally.

XI. COMPLIANCE WITH ETHICAL STANDARDS

1. Conflict of Interests: The authors declare that they have no conflict of interest.
2. Human and Animal Rights Informed Consents: This article does not contain any studies with human or animal subjects performed by any of the authors.

Conclusion

MR, which includes VR and AR, is a fascinating technology with a wide range of applications. Its advantage is the capacity to transfer information without altering reality by blending real and virtual content in a seamless manner. This could have far-reaching implications in the medical field, ranging from surgical planning to operation execution, treatment, and rehabilitation. Both, technology providers as well as medical experts are putting significant effort into the development of suited hardware, software, and application procedures. In scenarios with moderate technological demands like patient education, MR can be use defectively. For surgical support, the achievable accuracy is often not sufficient. Especially in urology the non-rigid nature of the organs poses a major challenge. Pre-operatively acquired data needs to be deformed to fit current organ shape. Deformable models, often taking even internal organ deformation into account, are under development. Overall, one can state that the underlying workflow and hardware to use AR/MR in urology are established, although many performance requirements are not yet met. The main challenges lie in intraoperative data acquisition, online and multimodal registration and calibration of devices and data, appropriate display hardware, as well as cooperative devices and tools in the operation theatres. On the other hand, the progress made in recent years is tremendous in all respects and the gap is constantly shrinking.AR and VR technologies will become more widely used in surgery because of their enormous potential. The benefits of these new technologies for relevant endpoints, on the other hand, are yet unknown. Clinical trials should be conducted to investigate this. Physicians should be at the forefront of this technology transformation to fully realise the benefits of AR and VR for their patients. A. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

References

[1] Gurusamy, K. S., Aggarwal, R., Palanivelu, L., & Davidson, B. R. (2009). Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane database of systematic reviews, (1). [2] Lateef, F. (2010). Simulation-based learning: Just like the real thing. Journal of Emergencies, Trauma and Shock, 3(4), 348. [3] Xie, B., Liu, H., Alghofaili, R., Zhang, Y., Jiang, Y., Lobo, F. D., ... & Yu, L. F. (2021). A review on virtual reality skill training applications. Frontiers in Virtual Reality, 2, 645153. [4] González Izard S, Vivo Vicent C, Juanes Méndez JA, Palau R: 2020 Oct 21 (pp. 414-421). Virtual Reality In Higher Education: An Experience With Medical Students: Research into how Virtual Reality can be used as a powerful training tool for medicine students. InEighth International Conference on Technological Ecosystems for Enhancing Multiculturality [5] Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. E. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37. [6] Hettig, J., Engelhardt, S., Hansen, C., & Mistelbauer, G. (2018). AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment. International journal of computer assisted radiology and surgery, 13, 1717-1725. [7] Hettig, J., Engelhardt, S., Hansen, C., & Mistelbauer, G. (2018). AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment. International journal of computer assisted radiology and surgery, 13, 1717-1725. [8] Hamacher, A., Whangbo, T. K., Kim, S. J., & Chung, K. J. (2018). Virtual reality and simulation for progressive treatments in urology. International neurourology journal, 22(3), 151. [9] Reis G, Yilmaz M, Rambach J, Pagani A, Suarez-Ibarrola R, Miernik A, Lesur P, Minaskan N: 2021 May 13:102394: Mixed reality applications in urology: Requirements and future potential. Annals of Medicine and Surgery. [10] Vávra, P., Roman, J., Zon?a, P., Ihnát, P., N?mec, M., Kumar, J., ... & El-Gendi, A. (2017). Recent development of augmented reality in surgery: a review. Journal of healthcare engineering, 2017. [11] Tepper, O. M., Rudy, H. L., Lefkowitz, A., Weimer, K. A., Marks, S. M., Stern, C. S., & Garfein, E. S. (2017). Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plastic and reconstructive surgery, 140(5), 1066-1070. [12] Concannon, B. J., Esmail, S., & Roduta Roberts, M. (2019, August). Head-mounted display virtual reality in post-secondary education and skill training. In Frontiers in Education (Vol. 4, p. 80). Frontiers Media SA. [13] Robinson, A. J., Miller, G., & Rukin, N. (2017). Simulation in urological training: Where are we in 2017?. Journal of Clinical Urology, 10(6), 548-556. [14] Hettig, J., Engelhardt, S., Hansen, C., & Mistelbauer, G. (2018). AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment. International journal of computer assisted radiology and surgery, 13, 1717-1725. [15] Greenfield, M. J., Luck, J., Billingsley, M. L., Heyes, R., Smith, O. J., Mosahebi, A., ... & Hachach-Haram, N. (2018). Demonstration of the effectiveness of augmented reality telesurgery in complex hand reconstruction in Gaza. Plastic and Reconstructive Surgery Global Open, 6(3). [16] Kumcu, A., Vermeulen, L., Elprama, S. A., Duysburgh, P., Platiša, L., Van Nieuwenhove, Y., ... & Philips, W. (2017). Effect of video lag on laparoscopic surgery: correlation between performance and usability at low latencies. The International Journal of Medical Robotics and Computer Assisted Surgery, 13(2), e1758. [17] Hamacher, A., Kim, S. J., Cho, S. T., Pardeshi, S., Lee, S. H., Eun, S. J., & Whangbo, T. K. (2016). Application of virtual, augmented, and mixed reality to urology. International neurourology journal, 20(3), 172. [18] Cheng, L., & El Zarki, M. (2016). U.S. Patent No. 9,237,347. Washington, DC: U.S. Patent and Trademark Office. [19] Fida, B., Cutolo, F., di Franco, G., Ferrari, M., & Ferrari, V. (2018). Augmented reality in open surgery. Updates in surgery, 70(3), 389-400. [20] Rassweiler, J., & Frede, T. (2002). Robotics, telesurgery and telementoring--their position in modern urological laparoscopy. Archivos espanoles de urologia, 55(6), 610-628. [21] Nakasu, E. (2012). Super hi-vision on the horizon: A future TV system that conveys an enhanced sense of reality and presence. IEEE Consumer Electronics Magazine, 1(2), 36-42. [22] Fominykh, M., Wild, F., Klamma, R., Billinghurst, M., Costiner, L. S., Karsakov, A., ... & Smolic, A. (2020, June). Developing a model augmented reality curriculum. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (pp. 508-509). [23] Weibel, N., Gasques, D., Johnson, J., Sharkey, T., Xu, Z. R., Zhang, X., ... & Davis, K. (2020, April). Artemis: Mixed-reality environment for immersive surgical telementoring. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-4). [24] Teber, D., Guven, S., Simpfendörfer, T., Baumhauer, M., Güven, E. O., Yencilek, F., ... & Rassweiler, J. (2009). Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. European urology, 56(2), 332-338. [25] Hamacher, A., Kim, S. J., Cho, S. T., Pardeshi, S., Lee, S. H., Eun, S. J., & Whangbo, T. K. (2016). Application of virtual, augmented, and mixed reality to urology. International neurourology journal, 20(3), 172. [26] van Oosterom, M. N., van der Poel, H. G., Navab, N., van de Velde, C. J., & van Leeuwen, F. W. (2018). Computer-assisted surgery: virtual-and augmented-reality displays for navigation during urological interventions. Current opinion in urology, 28(2), 205-213. [27] Nakamura, K., Naya, Y., Zenbutsu, S., Araki, K., Cho, S., Ohta, S., ... & Igarashi, T. (2010). Surgical navigation using three-dimensional computed tomography images fused intraoperatively with live video. Journal of endourology, 24(4), 521-524. [28] Simpfendörfer, T., Gasch, C., Hatiboglu, G., Müller, M., Maier-Hein, L., Hohenfellner, M., & Teber, D. (2016). Intraoperative computed tomography imaging for navigated laparoscopic renal surgery: first clinical experience. Journal of endourology, 30(10), 1105-1111. [29] Simpfendörfer, T., Baumhauer, M., Müller, M., Gutt, C. N., Meinzer, H. P., Rassweiler, J. J., ... & Teber, D. (2011). Augmented reality visualization during laparoscopic radical prostatectomy. Journal of endourology, 25(12), 1841-1845. [30] Borgmann, H., Rodríguez Socarrás, M., Salem, J., Tsaur, I., Gomez Rivas, J., Barret, E., & Tortolero, L. (2017). Feasibility and safety of augmented reality-assisted urological surgery using smartglass. World journal of urology, 35(6), 967-972. [31] Kong, S. H., Haouchine, N., Soares, R., Klymchenko, A., Andreiuk, B., Marques, B., ... & Marescaux, J. (2017). Robust augmented reality registration method for localization of solid organs’ tumors using CT-derived virtual biomechanical model and fluorescent fiducials. Surgical endoscopy, 31, 2863-2871. [32] Müller, M., Rassweiler, M. C., Klein, J., Seitel, A., Gondan, M., Baumhauer, M., ... & Maier-Hein, L. (2013). Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. International journal of computer assisted radiology and surgery, 8, 663-675.

Copyright

Copyright © 2023 Shriansh Chaabra, Hritik Baweja . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.