Comparison of Image Fusion Techniques Using Worldview-3 data

Image fusion is a useful tool for producing a high-resolution multispectral image to be used for land use and land cover mapping. In this study, we use nine pansharpening algorithms namely Color Normalized (CN), Gram-Schmidt (GS), Hyperspherical Color Space (HCS), High Pass Filter (HPF), Nearest-Neighbor Diffusion (NND), Principal Component Analysis (PCA), Resolution Merge (RM), Stationary Wavelet Transform (SWT), and Wavelet Resolution Merge (WRM) to fusion Worldview-3 multispectral Bands and panchromatic band. In spectral and spatial fidelity, several image quality metrics are used to evaluate the performance of pansharpening algorithms. The SWT and PCA algorithms showed better results compared to other pansharpening algorithms while GS and CN algorithms showed the worst results for the original image fusion. The effect of fusion on each band was separately investigated and according to the calculations, we found that the CoastalBlue band and the Blue band showed the best result and the NIR-1 band and NIR-2 band show the worst result for the original image fusion. In the end, we conclude that the choice of fusion method depends on the requirement of remote sensing application.


Introduction
The recent increase in high-resolution images of various types of satellites led to introduction of the images into remote sensing researches widely. The panchromatic band of a satellite data has high spatial resolution and the multispectral bands have a high spectral resolution but lower spatial resolution compared to the panchromatic band. In order to achieve the benefits of spatial capability of panchromatic band and spectral capability of multispectral bands, image pansharpening in used. Different algorithms for image pansharpening have been developed and they may lead to different pansharpening qualities [2][3][4][5][6] . Using the pansharpening algorithms leads to the achievement of the highest possible spatial and spectral information of the earth's surface based on the satellite images of interest. The fusion techniques has become very important in remote sensing industry due to their important role in remote sensing problems such as classification [7], change detection [8], object identification [9], hazard monitoring, stereophotogrammetry [10]. Many studies have been published on evaluation of the performance of fusion methods on Ikonos and Quickbird data [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27] and a few studies have been presented on Geoeye and Worldview2 data [28][29][30][31]. Some widely performed pan-sharpening algorithms are the Intensity-Hue-Saturation (IHS) [32][33][34], the Principal Component Analysis (PCA) [35][36][37] and the Brovey transform [38][39][40]. Other methods such as Gram-Schmidt (GS) [41,42], CN-spectral [43], Ehler [44] and high pass fusion (HPF) [1] relying on the intensity modulation. wavelet transform [45,46] and discrete wavelet transform [47] are used to extract geometric edge information. Du et al. [39] compared Brovey, PCA, MRM, GS, Brovey and UNB pansharp, pansharpening algorithms on QuickBird and IKONOS image and concluded that GS and UNB pansharp produce the best results. Nikolakopoulos [26] compared IHS, Modified IHS, PCA, UNB Pansharp, WRM, LMM, LMVM, Brovey, and MRM pansharpening algorithms on QuickBird and concluded that UNB Pansharp, LMVM and LMM algorithms gave the best results. Yuhendra and Kuze [48] compared GS, Ehlers, Modified IHS, HPF, and WRM on WorldView-2 data and found that the Ehlers method shows a better result for spatial and color reproduction compared to the other methods. Jawak and Luis [31] compared GS, Ehlers, Modified IHS, HPF, Brovey and WRM on WorldView-2 data and concluded that the GS produced the best result and the Brovey presented the worst result. Nikolakopoulos et al. [49] compared CN, Ehlers, GS, Modified IHS, LMM, LMVM, Pansharp, PCA and Wavelet on WorldView-2 data and concluded that Ehlers and HPF presented the best result. Mhangara et al. [50] compared twelve pansharpening algorithms on SPOT6 image and concluded that IMVM and RCS techniques produced better results compared to the other pansharpening methods. This study aims to compare the performance of nine different image pan-sharpening algorithms. In the study, Color Normalized (CN), Gram-Schmidt (GS), Hyperspherical Color Space (HCS) , High Pass Filter (HPF), Nearest-neighbor diffusion (NND), Principal Component Analysis (PCA), Resolution Merge (RM), stationary wavelet transform (SWT) and Wavelet Resolution Merge (WRM) algorithms are applied Worldview-3 images to produce better higher spectral and spatial resolution images. The spectral and spatial quality of the produced pan-sharpened images are assessed based on the correlation coefficient (CC), Erreur Relative Global Adimensionnelle de Synthese (ERGAS), root mean square error (RMSE), Entropy and signal-to-noise ratio (SNR). The assessment methods described by several authors [24,25,45,46,51]. These indicators were selected since they cover the statistical, physical, and similarity-based properties of the extracted fused imagery. The aim of this study is to compare different pansharpening algorithms applied to Worldview-3 multispectral Bands separately, by using the quantitative quality metrics.

Materials and Methods
The Worldview3 multispectral and panchromatic dataset were used for the study. The data obtained in May 2016 and provided by the DigitalGlobe, Inc. (http://www.digitalglobe.com). The image has registered to WGS 84 datum and the Universal Transverse Mercator (UTM) zone 23S projection. The research area is located in the center of Rio De Janeiro, Brazil. The study area is an urban area involving shadows, roads, water, vegetation, and high-density built-up areas with different roof types. The spectral information of optical sensor WorldView-3 is summarized in Table 1.  400  425  450  Blue  450  480  510  Green  510  545  580  Yellow  585  605  625  Red  630  660  690  Red-edge  705  725  745  NIR-1  770  832  895  NIR-2  860  950  1040  Pan  450  625  800 In this study nine algorithms namely the Color Normalized (CN), Gram-Schmidt (GS), Hyperspherical Color Space (HCS), High Pass Filter (HPF), Nearest-neighbor diffusion (NND), Principal Component Analysis (PCA), Resolution Merge (RM), stationary wavelet transform (SWT) and Wavelet Resolution Merge (WRM) were used for the fusion of Worldview-3 multispectral and panchromatic images. These algorithms have been employed to produce images with high spatial and spectral resolutions for various remote sensing applications (i.e. classification, change detection, visual image interpretation) and they are implemented in popular software such as ERDAS Imagine (HCS, 3 of 13 HPF, RM and WRM) and Envi (CN and GS); and they can be used by coding in MATLAB (PCA and SWT).
The color normalized spectral sharpening (CN) algorithm is used to simultaneously sharpen any defined number of bands. In this method, the lower resolution bands are sharpened by the higher resolution bands. The multispectral bands are clustered into spectral segments defined by the spectral range of the panchromatic band. The pansharpened image is generated through multiplying the high-resolution panchromatic with each lower resolution multispectral band and then normalized through dividing it by the sum of the input bands in the segment [16,17,27,37,51]. The Gram Schmidt fusion method enhances the spatial resolution of the multispectral bands throuh merging these bands with high resolution panchromatic band. The GS fusion simulates the PAN band from the lower spatial resolution spectral bands by averaging the MS bands. Then, the high-spatial-resolution panchromatic band is swapped with the first Gram-Schmidt band. Finally, an inverse GS transform is applied to produce the enhanced spatial resolution MS image [14,16,22,24,27,30,41]. Hyperspherical color sharpening (HCS) is specifically designed for WV-2 based on the transformation between any native color space and the hyperspherical color space. In this method, multispectral image transforms to an n-dimension color space and then histogram matching takes place between the square of pan image and the square of intensity image. The histogram matched pan image replaces with intensity component [52,53]. HPF Resolution Merge technique is to extract high spatial detail from the high-resolution Pan image into a bicubically-resampled version of the low-resolution MS image using a high pass filter (HPF). The high frequency information is then added into each band of the multispectral bands with a specified weight [27,38,[54][55][56]. Nearest neighbor diffusion is based on a weighted linear mixture of different bands in its neighborhood pixels. This fusion method enhances the spatial details while preserving spectral fidelity and because of its parallel processing, the processing time of this method decreases [11,[57][58][59]. The Principal Component Analysis (PCA) method is commonly used for spectral transformation of a multivariate data set of correlated variables yielding a new set of uncorrelated principal components. The first PC band is replaced by a high-resolution panchromatic band; then a reverse PCA transform is performed to fusing the panchromatic into multispectral channels. After the reverse PCA transformation, a high-resolution fused image with high spatial resolution of pan image and high spectral resolution of the multispectral image is generated [27,36,37,50,61,62]. Multiplicative Resolution Merge method applies a simple algorithm integrating the two raster images. As this method is computationally simple, it is generally fast, requiring the least system resources. However, the resulting merged image does not retain the radiometry of the input multispectral image. Instead, the intensity component increases, making this technique good for highlighting urban features (tending to be higher reflecting components in an image). SWT method uses a 2-D filter producing two images, the one is an approximation image, and the other is a detailed image called the wavelet plane. In this method, at each level, when the high-pass and low-pass filters are applied to data, two new sequences are produced at the next level. The two new sequences each have the same length as the original sequence. The filters at each level are modified by padding them out with zeroes [63,64]. Wavelet Resolution Merge (WRM) algorithm can be used to sharpen any type of low-resolution multispectral bands by high-resolution panchromatic band. In this method, panchromatic band infusing into each of the multispectral bands by performing a reverse wavelet transform on each MS band together with the corresponding wavelet coefficients. Thus, panchromatic band is decomposing into a set of low-resolution multispectral bands with corresponding spatial details for each level [50,67,68].
In this study two-step procedure were used for the fusion algorithms due to Wald protocol [1,44]. In the first step, multispectral bands and panchromatic band of original image were fused to produce a Pan sharpened MS image with 0.3m spatial resolution. At the second step, the MS image and the pan image of the original image were degraded by a factor of four. The nearest neighbor resampling method was used to change the spatial resolution of the multispectral image from 1.2m to 4.8m and the spatial resolution of panchromatic band from 0.3m to 1.2m. Different fusion algorithms were used to fuse the MS-1.2m and the Pan-0.3m. In order to evaluate the pansharpened image produced by the fusion algorithms, a reference MS image with the spectral resolution of 0.3m was used. Thus, new MS image were produced by resampling the original MS image to a 0.3m spectral resolution MS image. The original MS-1.2m image was used to evaluate the pansharpened images produced by fusing the MS-4.8m and Pan-1.2m.
The CC is one of the most widely used statistical measures of the strength and direction of the linear relationship between two images [76][77][78]. This measure is used to compute the similarity of spectral features and quantify the spectral correspondence between the original image and the fused images [79]. The CC between each band of the original image and the pansharpened image indicates the spectral fidelity of the pansharpened image by values between -1 and 1. The CC of the best fusion is close to 1 as much as possible. ERGAS is a relative dimensional global error in synthesis, A global indicator calculating the amount of spectral distortion by values between 0 and inf. The lower the ERGAS values, the better the spectral quality of the merged images [80]. RMSE measures the changes in the radiance of the pixel values for each band of the original image and calculates the similarity between each band of the original image and fused image. In fact, it measures the average amount of spatial distortion in each pixel to achieve the standard error of fused image. RMSE should be close to zero as much as possible [76,81]. The entropy is used to measure spectral and special information content in the fused image. The ideal value in this measurement is the smallest possible entropy difference between the original image and the pansharpened image [71]. Signal to noise ratio (SNR) is used to measure the ratio between information and noise of the fused image [82]. Summary of quantitative quality metrics, mathematical expression, and the related literature are described in Table  2. Table 2. The spectrum of the Worldview-3 bands.

Results and Discussion
The results for spectral evaluation of fusion Images are presented in Tables 3-14. In Table 3, the correlation scores between the original image -0.3m bands and fused image -0.3m bands are presented. These scores indicate the spectral similarity between the two images. According to the values obtained from the fusion of these two images, bands 1 to 6 scores made by HCS, PCA, RM, and SWT techniques have presented acceptable results. These results have 0.86 and 0.81 scores for RM and PCA respectively. NIR-1 and NIR-2 bands have the lowest scores. Based on the average obtained from the values of all bands, RM presented the best results and after that, PCA method has acceptable result. The worst result was produced by the CN method. Moreover, Coastal Blue, Blue, Green, and Red bands produced acceptable results and the worst result was produced by NIR2. In Table 4, the correlation scores between the resampled original image -1.2m bands and fused image -4.8m bands are presented. Compared to the previous result Table,   In Table 5, the ERGAS scores of fused image -0.3m bands are presented. These scores are indicated the spectral distortions in fused images. This method presented the general quality of fused image in global level. According to the Table 5, the scores range between 1.86 and 142.48. Based on these values, it is proved that SWT method produces the best result when applied to the fused image -0.3m. The CN method presents extremely high values, which are the worst results and not acceptable. The other methods had acceptable average values. According to the average of the calculated values for each band, the CoastalBlue band had the best score average and the green band had the worst score average. In some methods, the band scores are very close to each other, such as the HPF, RM, and WRM. In other methods, the band values fluctuate a lot. In NND method, the green and red bands have more spectral distortion, leading to an increase in the average values of this method. In Table 6, the ERGAS scores of fused image -1.2m bands are presented. According to this Table, the scores range between 5.98 and 272.80. Based on these large changes in the scores, it can be conclude that resampling images have increased spectral distortion of fused images. SWT method produces the best result when applied to the fused image -1.2m. The HCS method presents extremely high values, which is the worst results. In this case, CoastalBlue band has the most spectral distortion and NIR-1 band has the minimum spectral distortion.
In Table 7, the RMSE scores of fused image -0.3m bands are presented and the scores show the spectral quality in the image. These scores indicate that SWT method has the best spectral quality and GS method has the worst spectral quality.
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 28 September 2020 doi:10.20944/preprints202009.0685.v1 In Table 8, the RMSE scores of fused image -1.2m bands are presented. According to the scores, the spectral quality of CoastalBlue is mostly affected by resampling the original image. The scores produced for this band have been significantly changed in all methods and they have reduced the level of spectral quality in these methods. However, SWT method still has the best spectral quality. RM, CN and HCS methods has the worst spectral quality. In Table 9, the Entropy scores of fused image -0.3m bands are presented. According to the scores of Table 9, in all the fusion methods very small changes to the entropy values of the Image bands were presented. These scores indicate that PCA and SWT methods have the best spectral quality, while NND and GS methods have the worst spectral quality. According to Table 10, there was no large change in the total values of fused image -1.2. Only the CoastalBlue band had many changes and the entropy scores in this band increases significantly. Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 28 September 2020 doi:10.20944/preprints202009.0685.v1 In Table 11, the SNR scores of fused image -0.3m bands are presented. Higher values SNR suggest better spectral quality. According to the scores in Table 11, SWT method has the best spectral quality and GS method has the lowest spectral quality. Based on the average of the calculated values for each band, the CoastalBlue band had the best score average and NIR1 band had the worst score average. In Table 12, the SNR scores of fused image -1.2m bands are presented. According to the scores in this Table, HPF method has the best spectral quality and RM method has the lowest spectral quality. Based on the average of the calculated values for each band, the Blue band had the best average score and Coastal Blue band had the worst average score. Since the release of Worldview-3 data, not many studies have been done on the pansharpening of this data. Pandit worked on pansharpening of Worldview-3 by some of popular fusion algorithms, evaluated the results using some spectral and spatial metrics, separated them into two groups to use one of them for the original image, and used the other group for the resampled image. We used this approach to compare the pansharpening performance of more applicable algorithms in Worldview-3 data and used all the quality metrics introduced in this paper for both the original and resampled Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 28 September 2020 doi:10.20944/preprints202009.0685.v1 images. Jinling Zhao et al. worked on NND and brovey pansharpening algorithms for Worldview-3 image. They proved that NNDiffuse pansharpened image could generally maintain spectral and texture information. Entropy was used to evaluate the texture feature and NNDiffuse fused image had extremely similar values in Max, Mean, and SD compared to original panchromatic image. We calculated the entropy for all bands of the image and produced the scores of each band separately. The entropy was then compared with nine fusion methods. This paper was studied on the effectiveness of nine common fusion algorithms when applied to worldview-3 image. All the fusion algorithms were applied on both the original image and on spatial resampled data. The effect of image fusion with most commonly used algorithms was studied. According to the scores produced by ERGAS, RMSE, entropy, and SNR calculations, SWT algorithm has the best spectral quality and the least spectral distortion in original image fusion and the PCA method is ranked in the second level. It was found that CN and GS algorithms have the lowest spectral quality and highest spectral distortion. Based on the scores produced by different methods, the CoastalBlue band and the Blue band shows the best result and the NIR-1 band shows the worst result for the original image fusion. According to the scores produced by different fusion methods for resampled image, the SWT method generally has the best result and the CN method has the worst result.
Since different bands in these nine fusion methods have different accuracy, the choice of fusion method depends on the remote sensing application. For example, in calculating vegetation indices, blue, red, green, and NIR bands are required. Among the various fusion methods performed, PCA and SWT methods have the best values for these four bands.

Conclusion
Nine commonly used algorithms have been applied to Worldview-3 data for fusion of Pan and MS bands. Different quality metrics have been used for the quality assessment of the produced fused images. All the quality assessment methods have proved that all the fusion algorithms produce better results with the original image bands than the resampled ones. Most of the fusion algorithms influenced from the spatial resolution of the input data. Four algorithms CN, GS, HCS, and RM have the most unacceptable results with the resampled images. In this study, as shown in Table 1, eight bands of Worldview-3 were used and the effect of different fusion methods was provided separately for each band. Based on the values obtained from all fusion algorithms, acceptable results were produced for the Blue, Green, Yellow, Red, and Red-edge bands in both the original image fusion and the resampled image fusion. CoastalBlue band has the most effect from resampling as well as the worst result with all fusion algorithms for resampled image. According to the results obtained from studying the values produced by different fusion methods, the results produced from the SWT and PCA applied to the images by MATLAB coding provided better results.  Acknowledgments: In this section you can acknowledge any support given which is not covered by the author contribution or funding sections. This may include administrative and technical support, or donations in kind (e.g., materials used for experiments).