Survey on Multi-Modal Medical Image Fusion

Abstract

Multi-modality medical or clinical image fusion is a field of study aimed at enhancing diagnostic accuracy and aid in decisions to be taken by medical professional. Various fusion techniques such as pixel-based, region-based, and transform-based approaches are applied in image fusion to provide accurate fusion. Different devices which take scans of body such as MRI, CT, PET, SPECT, Ultrasound hold and carry different features, and different medical sensors obtain different information of the particular part of the body. Each of these imaging modalities offer only specific information that is used for the detection and analysis of specific problem. The idea behind fusion is to achieve and get better contrast and better fused image. The algorithm is making use of the common pyramid type and similarity type fusion algorithm with the neural networks model to achieve a better and more flexible fusion method. The advantages of image fusion medically are widespread. It plays a pivotal role in tumour localization, surgical planning and in treatment assessment.

Introduction

I. INTRODUCTION

Medical image modalities are Computed Tomography known as the CT scan, Magnetic Resonance Imaging known as the MRI scan, Positron Emission Tomography known as the PET scan, and Single-Photon Emission Computed Tomography known as the SPECT have provided medical professionals with overview of the body’s structural features like bones, soft tissue characteristics and many other characteristics.  As each of these modalities use a different imaging principle, they have different advantages which can be used in the domain. Image fusion combines the diagnostic features of two or more modalities to provide a reliable basis for diagnosis of the human body. The composite image will be of better quality and help in achieving a much suitable platform for medical professionals in diagnosing the part of human body. This can be very helpful in various fields such as image recognition in medical imaging or helps in combining different perspectives from different data sources to improves analysis accuracy.

Image fusion is usually of three types, they are pixel-level fusion, feature level fusion, and decision level fusion. If the fusion is through considering each pixel information from input modality images, fusion is considered to be pixel-level. If the fusion is through considering the features from the input images, then the fusion is considered to be of feature level. If the image fusion is through considering the decision parameters from the each source images, then the fusion is considered to be decision level. 

Multi scale transform (MST) technique is one of the highly used methods in the image fusion. The MST fusion technique is based on three fundamental steps. Initially the source images are converted into MST domain by the inbuilt tools from the software. After that the parameters of different scales are merged using a special fusion strategy. At last, composite image is rebuilt through the originally used inverse MST transformation. The MST techniques uses the laplacian pyramid (LP), The wavelet transform (WT), the non subsampled contourlet transform (NSCT) and the non-subsampled shearlet transform (NSST) . It is observed that if the MST method used compared to other fusion measures, some phenomena knows as the block effect is observed. To overcome this disadvantage, some fusion measures along with the MST method is used together. Taking the example, spatial frequency (SF), local variance (LV), the energy of image gradient (EIG) and sum-modified-Laplacian (SML) are commonly used as fusion measures along with MST method. The other disadvantage of these measures applied in the space domain or low-order gradient domain, the fusion map may not be always precise which is expected in the industry. Except for traditionally used MST methods, the edge preserving filtering (EPF) is used along with MST image decomposition are also used frequently. In the EPF-MST techniques, Gaussian filtering and EPF are used to decompose the original image modality into two detailed layers and one base layer components. These three layers are then fused based on suitable fusion techniques. Finally, the fused composite image is represented by a reconstruction algorithm used as the medical fusion algorithm.

As the images used are by the biological nature, artificial neural networks (ANN) has the ability to learn from input image modalities in order to make decisions and fusion through feature processing.

A training dataset is used to adjust a set of parameters to provide accurate predictions without relying much on complex mathematical models and calculations .

Since nature of medical image modalities is subjective to change among different modalities such as contrast, resolution, texture, type of tissues and many other things, adopting ANN to these changes makes it the best solution not only to perform the image fusion, also to assist clinical diagnosis by the medical professionals through the fused images, such as breast cancer and ovarian cancer diagnosis and other medical conditions. We have also noticed that multiple versions of ANN models have covered many modal image fusion such as mapping neural network (MNN) and PCNN. But the PCNN is the most extensively used model in the transform domain and in the spatial domain.

II. LITERATURE REVIEW

Many researchers have worked in fusing medical image modalities. They have opted several mechanisms for developing new algorithms and new models for extracting the features by using these newly created algorithms and models .Most papers are implementing CNN model for defining weight maps and later this same model weight maps will be passed through different layers of CNN for more precise and accurate fusion of the source medical images . Some image fusion papers are reviewed while conducting this survey. Reading through the papers, it is observed that  there are some major algorithms based on which the newer algorithms and newer models are being built upon. Quality improvements is done on already existing methods and algorithms. Some of the papers are reviewed here

A. [Medical Image Fusion using Transform Techniques]

The paper[1] has implemented the edge and energy fusion rule for the decomposed bands of Discrete Wavelet Transformation (DWT) and Stationary Wavelet Transformation (SWT). They have fused MRI scans and PET scans images. Sub-bands were selected depending on edges and energy functions. The image quality of composite image is checked using the entropy method. The fusion proved in accuracy of composite image to be increased by factor of 0.05. From their analysis, Stationary Wavelet Transformation (SWT) showed better results than Discrete Wavelet Transformation (DWT).

B. [Fast curvelet transform through genetic algorithm for multimodal medical image fusion]

The paper[2] uses a fusion algorithm for medical images using the curvelet transform to overcome the  limitations of wavelet transform. They were proved to do a better analysis in dealing with contours compared to previous researches. The curvelet transform has performed better in dealing with the noise in the image modality . The resultant composite image has far less noise compared to the original medical image modalities. The algorithm implemented is considered to be a generic algorithm which had better efficiency in the analysis of all medical image modalities.

C. [Research of Multimodal Medical Image Fusion Based on Parameter-Adaptive Pulse-Coupled Neural Network and Convolutional Sparse Representation]

The paper[3] uses a method in which CNN is applied to decompose the input image modality into detail layers and base layers components. This division was done using the non sub sampled shearlet transform (NSST) algorithm. The detail components are then merged using a pulse-coupled neural network (PCNN) model and the base components are fused by the sparse representation model. The composite images produced were of better contrast, better intensity and better sharpness compared to other fusion algorithms.

D. [Multi-modality medical image fusion using convolutional neural network and contrast pyramid]

The paper[4] uses a fusion method for scan images using convolutional neural network (CNN) to identify weight map generation. Using these weight maps, the image decomposition of image is performed by using a contrast pyramid.  This algorithm is proved to be better in terms of preserving the detailed structural information of input images and retaining the same in the composite image.

E. [Brain Medical Image Fusion Based on Dual-Branch CNNs in NSST Domain]

The paper[5] uses a fusion framework for CT-MRI image modalities fusion in which convolutional neural network (CNN) is used for creating a weight map. Using these weight maps, the image is decomposed by  convolutional neural network (CNN). The composite image is a fusion between the decomposed weight map of high frequency and low  frequency coefficients.

F. [Multimodal Medical Image Fusion using NSCT and DWT Fusion Framework]

The paper[6]  has proposed a fusion method using Nonsubsampled Contourlet Transform  (NSCT) and Discrete Wavelet Transformation (DWT). Initially the scan images are decomposed using the NSCT algorithm and the decomposed images are then merged using the energy rule as used in paper[1]. The original input image modalities are again decomposed using DWT algorithm and the decomposed images are then merged using the same fusion rule. The resultant two fused images are fused by making use of the Edge Strength and also the Orientation Preservation (ESOP) rule. It was found out that the quality of similarity measures based on edges and mutual information was improved for all medical image modalities overall. The efficiency was comparatively better than the previous researches, but the time taken to spew out the composite fused image was much longer compared to other processes.

G. [Multimodal Medical Image Fusion based on Gray Wolf Optimization and Hilbert Transform]

The paper[7] suggested a fusion approach for MRI scans and PET scan image modalities by using Gray wolf optimization algorithm along with the Hilbert transform. As the PET image modality contains information based of colour, these image modalities were transformed from RGB to Intensity- Hue-Saturation (IHS)  . Then 2-D Hilbert Transform is applied for the intensity band of the PET image and also on the MRI scan image. The GWO (Gray wolf optimization) is used to these image modalities to form the composite image. The composite image was improved by combining the  I(Intensity) and Hue/Saturation by applying Inverse Hilbert Transformation. The experimental results showed that GWO shows better results in respect to variations in iterations on the various image modalities and the overall size of the data. This method proved to  achieve better results with higher overall fusion quality, higher spectral resolution and spatial resolution.

H. [Computer Methods and Programs in Biomedicine MRI and PET image fusion using the nonparametric density model and the theory of variable-weight]

The paper[8] came up with MRI scans and PET scans fusion in which multimodal source images were first converted into a general Intensity- Hue-Saturation (IHS) space and then break down the image modality into the detailed layers and base layer coefficients by applying Nonsubsampled Contourlet Transform  (NSCT). A nonparametric density model is made use of to combine base layer components and a maximum fusion strategy depending on the variable weight theory is made use of to fuse detail layer components. Inverse NSCT transformation is made use of to combine the independently fused base layers and detail layers images of each separate modalities. The fused composite image is generated using inverse GIHS transformation. This technique proved to hold the accuracy of colour details from the PET image modality scale and had good spatial resolution.

I. [Multi-Modal Sensor Medical Image Fusion Based on Multiple Salient Features with Guided Image Filter]

The paper[9] suggested an image fusion algorithm using guided image filters to overcome the flaws with respect to low contrast problem which was prominent in the previous researches. The source image modalities were decomposed into detailed layer image and base layer image at several levels using a guided image filter. The prominent features from the base layers and the detailed layers were extracted using the Spectral Residual algorithm and the Visual Saliency model based on graphs. A generalized Intensity- Hue-Saturation (IHS) method was also implemented to join the base layers and detail layers components and their respective prominent features. The fusion of the MRI-SPECT/PET image modalities revealed that this method offers the advantage in terms of contrast, but a low colour contrast compared to the other previous researches.

J. [Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform]

The paper[10] suggested an image fusion algorithm based on Moving Frame Decomposition Framework (MFDF) and non sub sampled shearlet transform (NSST) . Each separate input image modality is broken into base layers and detail layers component. The fusing of the detail layer components, the highest fusion rule is made use of to identify more prominent edge elements. NSST is applied to further decompose the base layer components into sub-base layer and detail layer sub-bands which are joined together by applying the average fusion technique and sum modified laplacian fusion technique. This method was found to have better visual effect compared to other algorithms.

K. [Adaptive decomposition method for multi-modal medical image fusion]

The paper[11] presented a fusion algorithm for medical scan images in which an adaptive algorithm is made use of to decompose the structural images to provide smoothing and texture layers. The structural image's smoothing layer and the functional image's colour information were fused using dynamic fusion rules. Then the texture layer is added on the fused image. This fusion metrics obtained was capable of retaining both colour and structure information and helped in providing colour rich fusion images.

III. METHODOLOGIES

Image pre-processing is a fundamental step before performing any process on the images. This process done to enhance the size, colour and quality of image and help in further processing of particular image. Major reason for pre-processing is to resize the image to adjust an image's size between the different modality images for performing the image fusion. Converting images to grayscale helps to simplify pre-processing activity which reduces the noise and complexity while identifying important information. Other techniques performed are contrast adjustment, noise reduction and intensity normalization edge detection.

After the pre-processing, the images of different modalities processed for feature extraction. The steps involved in feature extraction is identify and isolate important feature or information within an image. These features are wide range of characteristics such as corners, edge, or patterns in the histograms. Feature extraction is a major component for processes such as image fusion, object recognition and image classification.

Feature extraction is mainly performed using Convolutional Neural Networks (CNN). The CNN architecture applies a various methods as convolutional and pooling layers to find the important features through the inputted image of different modalities. The aim is to develop feature maps from more image of different modality and fuse it into a single image which is the resultant fused image. The CNN model used for feature extraction is trained with more than hundreds of images of the modalities. The images can be different type of modalities and it will pass through that model and we will get different feature map for the different input images. By increasing the convolutional layers, more features can be extracted from the image modality resulting in a high-quality fused image.

Once these features are extracted, using the feature maps, we use matlab to output a single fused image. This fusion process combines the extracted features from different images to enhance their total required information content. Matlab software helps to simplify the process for image fusion by offering tools to ensure integration of these feature-rich images. After this we will go through pooling process to modify the size of the composite fused image.

The images decomposed in different levels using the process of Singular Value Decomposition (SVD) . At each level of decomposition process, the complete value of the two singular value is obtained. The detailed feature-rich image will be selected by fusion rules as the detailed feature-rich image is sharper and brighter such as edges and object boundaries. The process subsampled and smoothened the input images.

IV. ACKNOWLEDGEMENT

We would like to offer our sincere gratitude to Dr Kusuma T for her insightful and helpful recommendations during the project's conception and development. We are very grateful for her willingness to give of her time so freely. Additionally, we would like to express our gratitude to all of the KSIT management members, faculty and support staff for their ongoing support and inspiration.

Conclusion

This paper gives a conspectus of the research work of medical image fusion algorithms, techniques and the methodologies used. A comparative study of different image fusion algorithms with their advantages and disadvantages. This paper also discusses different multimodality image fusion methods used previously by researchers. Hybrid algorithms which incorporate spatial and transform domain approaches have been the focus of recent studies. Image fusion processes availing deep learning techniques are growing at a fast rate nowadays. Methods based on Neural Networks like Convolutional Neural Network (CNN) are the focused area of research. Over the past years, numerous researchers have made significant efforts in the field of medical image fusion, using a variety of techniques to fuse the images. However, certain issues, including blurring and the preservation of clear boundary edges, have to be resolved. There is a need to identify and analyse a new fusion technique that can effectively resolve the challenges that existing methods have been unable to satisfy.

References

[1] B. Ashwanth and K. Veera Swamy, \"Medical Image Fusion using Trans- form Techniques\", Presented at ICDCS 2020 2020 5th International Conference on Devices, Circuits and Systems, no. 2, 2020, pp. 303-306. Available at https://ieeexplore.ieee.org/document/9075544 [2] M. Arif and G. Wang. \"Fast curvelet transform through genetic algorithm for multimodal medical image fusion,\" Soft Computing, vol. 24, no. 3. pp. 1815-1836, 2020. [Online]. Available: https://doi.org/10.1007/s00500-019-04011-5 [3] J. Xia, Y. Lu, and L. Tan. \"Research of Multimodal Medical Image Fusion Based on Parameter-Adaptive Pulse-Coupled Neural Network and Convolutional Sparse Representation.\", Published in Computational and Mathematical Methods in Medicine, vol. 2020, 2020. Available at https://doi.org/10.1155/2020/3290136 [4] K. Wang, M. Zheng, H. Wei, G. Qi, and Y. Li, \"Multi-modality medical image fusion using convolutional neural network and contrast pyramid,\" Published in Sensors (Switzerland), vol. 20, no. 8. pp. 1-17, 2020. Available at https://www.mdpi.com/1424-8220/20/8/2169 [5] Z. Ding. D. Zhou, R. Nic, R. Hou, and Y. Liu, \"Brain Medical Image Fusion Based on Dual-Branch CNNs in NSST Domain\" , Published in BioMed Research International, vol. 2020, 2020, Available at https://doi.org/10.1155/2020/6265708 [6] K. Koteswararao and K. V. Swamy, \"Multimodal Medical Image Fusion using NSCT and DWT Fusion Framework\" , Presented at International Journal of Innovative Technology and Exploring Engineering. vol. 9, no. 2, pp. 3643-3648, 2019. Available at https://www.ijitee.org/wp-content/uploads/papers/v9i2/B8036129219.pdf [7] K. Kaur, S. Budhiraja, and N. Sharma, \"Multimodal Medical Image Fusion based on Gray Wolf Optimization and Hilbert Transform\", Published in Biomedical and Pharmacology Journal, vol. 12, no. 4, pp. 2091-2098, 2019, Available at https://dx.doi.org/10.13005/bpj/1844 [8] Z. Liu, Y. Song, V. S. Sheng, C. Xu, C. Maere, K. Xue, and K. Yang, \"Computer Methods and Programs in Biomedicine MRI and PET image fusion using the nonparametric density model and the theory of variable-weight,\" Published in Computer Methods and Programs in Biomedicine, vol. 175, pp. 73-82, 2019. [Online]. Available: https://doi.org/10.1016/j.cmpb.2019.04.010 [9] W Li, L. Jin, and J. Du, \"Multi-Modal Sensor Medical Image Fusion Based on Multiple Salient Features with Guided Image Filter,\" Published in IEEE Access, vol. 7, pp. 173019-173033, 2019, Available at https://doi.org/10.1007/s10489-021-02282-w [10] X. Liu, W. Mei, and H. Du, \"Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform\", Published in Biomedical Signal Processing and Control, vol. 40, pp. 343-350, 2018. [Online]. Available: http://dx.doi.org/10.1016/j.bspc 2017.10.001 [11] J. Wang, X. Li, Y. Zhang, and X. Zhang, \"Adaptive decomposition method for multi-modal medical image fusion,\" pp. 1403-1412, 2018, Available at https://doi.org/10.1049/iet-ipr.2017.1067

Copyright

Copyright © 2023 Dr. Kusuma T, Archit Ganapati Avadhani, Basavesh V L, Bhuvan N Gowda, H. A. Trishala. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.