A Review of the Existing Literature on 3D Reconstruction Methods

Abstract

Using deep neural networks, supervised 3D reconstruction has made significant progress. However, large-scale annotations of 2D/3D data are necessary for this performance boost [4]. How to effectively represent 3D data to feed deep networks remains a challenge in 3D deep learning. Volumetric or point cloud representations have been used in recent works, but these methods have a number of drawbacks, including computational complexity, unorganized data, and a lack of finer geometry [7]. The ultrasound (US) examination is one method used to diagnose carotid artery disease. The flow conditions in the artery also play a role in the onset and progression of vascular diseases [5]. Stenting the carotid artery is a treatment option for carotid atherosclerosis. It is impossible to know for sure where an injection with a needle will begin. The place of the conduits is in the body, thusly, deciding the beginning stage of needle infusion is finished by assessment just and can\'t not entirely set in stone. The first thing that must be done is to locate the carotid artery in order to identify it. To determine it, we propose a modified template matching based on the ellipse feature for a 3D reconstruction of the carotid artery [1]. Data acquisition, pre-processing, segmentation, outlier selection for ellipse parameter fitting, and visualization are all used to process it. In comparison to the template matching method and the Hough Circle method, the proposed procedure with pre-processing produces the highest accuracy [1]. The objective of this research was to create three- dimensional (3D) ultrasound imaging of the carotid arteries to lessen the variability of volume measurements between and within examiners during follow-up scans of atherosclerotic plaques.

Introduction

I. INTRODUCTION

One of the cardiovascular illnesses that affects people is carotid artery stenosis (CAS). It is critical to get this disease caught early because if it is not treated properly, it may lead to worsening effects like a crippling stroke. The classic computer vision problem of inferring the 3D geometry of an object or scene from its 2D projection on the image plane has numerous applications, such as object recognition, scene understanding, medical diagnosis, animation, and more [3]. A large number of images has traditionally been used for dense 3D reconstruction from images.

Dense matching, direct minimization, or reprojection errors extract geometry. Implicit volumetric reconstruction and explicit mesh-based approaches are two examples of common approaches [6]. This becomes even more appealing and motivating when millions of 3D CAD models from various categories are made publicly available online [7]. Currently, the modalities of Magnetic Resonance (MR), Compute Tomography (CT), and Ultrasound (US) are typically used to detect and quantify carotid stenosis. In interventions, they aid in the decision- making process. The outcomes of this intervention can be used to help insert a catheter or use a thrombus aspiration machine to remove the plaque.

MR or CT scan angiography has been used to locate carotid arteries in previous studies. Mechanical scanning, an optical tracker, and a magnetic tracker all play a role in reconstruction. The distance between the ultrasound probe's marker and the world coordinate must be determined using the tracker. The goal of this study is to use two-dimensional B-mode images to reconstruct the CA lumen area's 3D object. The reconstruction was carried out using data collecting, pre-processing, segmentation, choosing outlier segmentation, ellipse parameter fitting, and visual analytics. [1].

The proposed paper contains different sections like image analysis, data acquisition, pre-processing, related works, segmentation and outlier selection, results and discussion, conclusion.

II. 3D RECONSTRUCTION OF THE CAROTID ARTERY

The patient's carotid artery is 3D reconstructed using the patient's unique clinical imaging data. One of the key problems is the small number of 2D transversal cuts in the patient data set that was employed. The generalized model served as the foundation for resolving the issue of missing cuts, which was then enhanced using the available data [5].

The carotid artery (CCA) transversal cut (shown in Fig. by the B line)1) as well as the external carotid artery (ECA; Fig. C line denotes lower segment).1) is utilized to define the cross-sectional shapes of these segments' carotid arteries. The internal carotid artery (ICA) longitudinal cut (shown in Fig. by the A segment)1) is used to extract the ICA's center line and diameters from this segment, while the ICA's transversal cuts (the C line in the upper segment and the D line in the cross-section in Fig.)1) are also utilized to provide a more precise definition of these specific cross-sections. Based on the data gathered from the cuts, the lengths of the branches are determined.

III. IMAGE ANALYSIS

In order to improve image registration between the two scans, the bifurcation of the index side was the focal point of the 2D and 3D scan geometries, which is the side with the increased stenosis in the ultrasonography. The index side artery was the focus of each and every subsequent analysis. A previously published five-point scale was used to evaluate the image quality: exams with image quality below two were not included in the review process. To match picture locations between scans, the common carotid artery's bifurcation into the internal and exterior carotid arteries was independently located on both 2D and 3D image sets. To assess the role of the partial volume effect and improvements with a thinner slice thickness, the original 3D pictures were additionally reformatted to a 2D-like dataset (referred to as 3Dr below). [28]. Four adjacent 3D slices with a 0.5-mm slice thickness that anatomically matched a corresponding 2D image were summed per pixel during this procedure. Figure depicts the scheme for matching images and reconstructing 3D data, as well as examples of native and reformatted images. As a result, three sets of images: review and quantitative measurements could be done with 2D, 3D, and 3Dr.

IV. PROPOSED PROCEDURE

A. Data Acquisition

The method for 3D reconstruction of the carotid artery utilising US pictures as input was developed and validated using a collection of original and annotated US images from the Toshiba test. The annotated photos were utilised for processing and training, while the original images were used for validation. For each patient, transversal and longitudinal projections show the common carotid artery, its branches, and the carotid bifurcation. B- mode was used for the examination. In keeping with the safety and security of the data, all imaging data were anonymized [32]. There are various frames in each data set depending on the probe's speed. The position and constant direction of the probe influence the ability to retrieve data. The image contains spatial data, and it is expected that position data is sequential from the data acquisition's beginning to its conclusion.

B. Pre-processing

The goal of the pre-processing step is to better prepare the data for the segmentation step. An elliptical morphology feature is the one used. In order to make the ellipse's characteristics more apparent, data on the carotid artery's surface area are exposed. So the pre-processing step is to make images that show the carotid artery's location. A stage of pre-processing that uses median filters, histogram equalization, and a series of Gaussian filters. The image is processed using gaussian filters to produce more uniform data in certain areas with finer results. The image is processed using gaussian filters to produce more uniform data in certain areas with finer results. The value of the standard deviation parameter (σ) is 0.5. Histogram Equalization follows the Gaussian filter process's output. The data have salt and pepper noise, which can make it hard to segment the data. To circumvent this problem, the median filter is employed. In some sections of the image data, the divided piece with holes in the surrounding area can offer a high gradient value. It is overcome by the application of dilation morphology. [1].

C. Segmentation

The information about texture, shape, and contours, among other things, is used in the traditional image segmentation techniques perform well when there is less noise in the images. However, it has been discovered that the objective function is typically much more complex in the context of medical images. As a result, the number of classical techniques that can be used is limited. Using anisotropic diffusion or geodesic active contours, the segmentation method that is used to measure the IMT was suggested in some papers. A novel approach to the processing of B-mode ultrasound images that makes use of all of the data in the image to reduce anisotropic noise was developed by others. It is currently being made available for RF ultrasound data as well as 3D carotid image data[2].

V. RELATED WORK

The study of reconstructing 3D shapes from 2D images has increased thanks to recent advancements in 3D representation learning, reconstruction, and completion in 3D computer vision field. Based on the quantity of input images, this research can be divided into the following categories: single-view and multi- view three-dimensional shape reconstruction. However, the associated work can also be characterised as voxel-based 3D shape reconstruction, point cloud-based 3D shape reconstruction, or mesh-based 3D shape reconstruction according to various representation forms of 3D shapes. In particular, the single-view 3D shape reconstruction that is based on point clouds includes the 3DAttriFlow that is proposed in this paper. For convenience, the discussion of related work will be arranged in accordance with 3D shape output forms.

A. Outlier Selection

Selection of outlier is the process of calculating a data's segmentation error using all of the available data. The carotid artery may not be correctly segmented throughout the segmentation process. The segmentation method saves ten of the best elliptical data, which can be utilised to fix the segmentation error. Calculating the standard deviation () is necessary to spot anomalous carotid artery segmentation results [1].

B. Visualisation

Using a 3D point cloud, the polynomial fitting functions for the parameters x, y, a, and b are visualised. To draw the ellipse, a polynomial fitting function is applied to each parameter. Each ellipse is separated by 0.01 cm, while the beginning and last ellipses are separated by 5 cm. Leveraging the CloudCompare application for visualisation, the smooth visualisation is achieved by using the close spacing between the ellipses.

VI. RESULTS AND DISCUSSION

This work created a 3D reconstruction of an artery that allowed to visualise surfaces based on ellipses that represent arterial walls. Data collection, pre- processing, segmentation, outlier selection, curve fitting with polynomial fitting, also point cloud visualisation are the first steps in this study. Ten datasets are produced by data collecting from 10 individuals. The number of pictures in each dataset varies depending on how quickly the probe moves during the data collecting process. The same frequency and depth parameters are used during data collecting. Whether a good or awful image is created for each person determines the gain setting, which ranges from 85% to 95% [1].

Patients who have cardiac arrhythmia are another category of patients for whom 3-D ultrasound imaging poses challenges because freehand 3-D ultrasound is not a real-time imaging technique that relies on ECG-gated acquisition to capture serial images of the vessel at the same point in the cardiac cycle. Unless combined ECG- and respiratory-gated acquisition is used, patients who have trouble breathing should also be thought of for exclusion from 3-D ultrasound assessment to prevent significant motion artefacts from being introduced into the reconstruction. Respiratory motion was not found to introduce significant artefacts for the majority of patients scanned in this study, however. It is still unknown how useful measures like plaque volume and volumetric stenosis are in the clinical setting. Some of these measurements may help in the identification and management of high-risk patients by offering a measure that more accurately reflects the true severity of the disease in relation to the dimensions of the vessel wall or by tracking plaque progression shown by changes in serial measurements [19].

Conclusion

The surface artery was visualized in three dimensions as a result of this study. The stages of this study are as follows: segmentation, selection of outliers, ellipse fitting parameters, and visualization are all part of the data acquisition process. In the proposed method, segmentation without pre- processing yields results with lower accuracy than those with pre-processing. The proposed approach with pre-processing, compared to the template matching and hough circle methods, obtains the maximum accuracy with a 99.41% accuracy rate and the smallest standard deviation of 0.05. The polynomial equation, which is the best match for all data, has the minimum mean error value in the 22nd order, 0.26303. Therefore, the carotid artery position on the B-mode ultrasound image can be seen in three dimensions using the modified template matching with ellipse feature.

References

[1] The link for Toshiba dataset: https://drive.google.com/drive/folders/1RcrLy2V3 wzPAnsXgBtvKe4sTN05EQlEC [2] 3D Reconstruction Dataset, http://vision.ia.ac.cn/data [3] I- Made Gede Sunarya, Eko Mulyanto Yuniarno, Tri Arief Sardjono, Ismoyo Sunu, P. M. A. (Peter) van Ooijen & I. Ketut Eddy Purnama (2020) 3D reconstruction of carotid artery in B-mode ultrasound image using modified template matching based on ellipse feature, Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 8:3, 301, 312, DOI: 10.1080/21681163.2019.169223 [4] R. Hemalatha, N. Santhiyakumari, M. Madheswaran, S. Suresh, Segmentation of 2D and 3D Images of Carotid Artery on Unified Technology Learning Platform, Procedia Technology, Volume 25, 2016, Pages 12-19, ISSN 2212-0173, https://doi.org/10.1016/j.protcy.2016.08.075. [5] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S. (2016). 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. https://doi.org/10.1007/978-3-319-46484-8_38 [6] J. Gwak, C. Choy, M. Chandraker, A. Garg and S. Savarese, \"Weakly Supervised 3D Reconstruction with Adversarial Constraint,\" in 2017 International Conference on 3D Vision (3DV), Qingdao, China, 2017 pp. 263-272. doi: 10.1109/3DV.2017.00038 [7] Tijana Djukic, Branko Arsic, Igor Koncar, & Nenad Filipovic. (2020, October 30). 3D reconstruction of patient-specific carotid artery geometry using clinical ultrasound imaging. https://doi.org/10.5281/zenodo.4563917 [8] C. Hane, S. Tulsiani and J. Malik, \"Hierarchical Surface Prediction for 3D Object Reconstruction,\" in 2017 International Conference on 3D Vision (3DV), Qingdao, China, 2017 pp. 412-420. doi: 10.1109/3DV.2017.00054 [9] Pontes, J.K., Kong, C., Sridharan, S., Lucey, S., Eriksson, A., Fookes, C. (2019). Image2Mesh: A Learning Framework for Single Image 3D Reconstruction. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11361. Springer, Cham. https://doi.org/10.1007/978-3-030-20887-5_23 [10] mith, Edward & Fujimoto, Scott & Meger, David. (2018). Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation. [11] Xu, Qiangeng & Wang, Weiyue & Ceylan, Duygu & Mech, Radomir & Neumann, Ulrich. (2019). DISN: Deep Implicit Surface Network for High- quality Single-view 3D Reconstruction. [12] Michalkiewicz, M., Parisot, S., Tsogkas, S., Baktashmotlagh, M., Eriksson, A., Belilovsky, E. (2020). Few-Shot Single-View 3-D Object Reconstruction with Compositional Priors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12370. Springer, Cham. https://doi.org/10.1007/978-3- 030-58595-2_37 [13] Jin, J., Patil, A.G., Xiong, Z., Zhang, H. (2020). DR-KFS: A Differentiable Visual Similarity Metric for 3D Shape Reconstruction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12366. Springer, Cham. https://doi.org/10.1007/978-3-030-58589-1_18 [14] Li, X. et al. (2020). Self-supervised Single-View 3D Reconstruction via Semantic Consistency. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12359. Springer, Cham. https://doi.org/10.1007/978-3- 030-58568-6_40 [15] Xu, Y., Fan, T., Yuan, Y., Singh, G. (2020). Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12346. Springer, Cham. [16] Liu, Weiyang & Liu, Zhen & Paull, Liam & Weller, Adrian & Schölkopf, Bernhard. (2022). Structural Causal 3D Reconstruction. 10.48550/arXiv.2207.10156. [17] Xing, Zhen & Chen, Yijiang & Ling, Zhixin & Zhou, Xiangdong & Xiang, Yu. (2022). Few-shot Single-view 3D Reconstruction with Memory Prior Contrastive Network. 10.48550/arXiv.2208.00183. [18] Xing, Z., Li, H., Wu, Z., Jiang, YG. (2022). Semi-supervised Single-View 3D Reconstruction via Prototype Shape Priors. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13661. Springer, Cham. https://doi.org/10.1007/978-3-031-19769-7_31 [19] Delcker A, Diener HC. Quantification of atherosclerotic plaques in carotid arteries by three- dimensional ultrasound. Br J Radiol. 1994 Jul;67(799):672-8. doi: 10.1259/0007-1285-67- 799-672. PMID: 8062009. [20] Barratt DC, Ariff BB, Humphries KN, Thom SA, Hughes AD. Reconstruction and quantification of the carotid artery bifurcation from 3-D ultrasound images. IEEE Trans Med Imaging. 2004 May;23(5):567-83. doi: 10.1109/tmi.2004.825601. PMID: 15147010. [21] Ma Y, Wang Z, Dai X, Chen B, Basu A. 3D Reconstruction of Carotid Artery from Ultrasound Images. Annu Int Conf IEEE Eng Med Biol Soc. 2021 Nov;2021:4218-4221. doi: 10.1109/EMBC46164.2021.9630097. PMID: 34892154. [22] X. Wen, J. Zhou, Y. -S. Liu, H. Su, Z. Dong and Z. Han, \"3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow,\" 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 3793-3803, https://doi.org/10.1007/978-3-030-58452-8_15 [23] Latha S, Samiappan D, Kumar R. Carotid artery ultrasound image analysis: A review of the literature. Proc Inst Mech Eng H. 2020 May;234(5):417-443. doi:10.1177/0954411919900720. Epub 2020 Jan 21. PMID: 31960771.Seabra JC, Pedro LM, e Fernandes JF, Sanches JM. A 3-D ultrasound-based framework to characterize the echo morphology of carotid plaques. IEEE Trans Biomed Eng. 2009 May;56(5):1442-53. doi: 10.1109/TBME.2009.2013964. Epub 2009 Feb 6. PMID: doi: 10.1109/CVPR52688.2022.00378.19203880.

Copyright

Copyright © 2023 Nikitha Revanaki, Pankaja Biradar, Prerana S Kabade, Sanjeetha Patil. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.