Recently, team of Professor Zhanshan Wang and Professor Xinbin Cheng from IPOE proposed a spatial-spectral dual adaptive graph embedding model (SDAGE) to exploit the spatial nonlocal similarity (SNS) and spectral band correlation (SBC) in high spatial resolution multispectral (HR MS) and low spatial resolution hyperspectral (LR HS) images for the regularization of their fusion with their collaborators. The results were published in Pattern Recognition with the title "Learning Spatial-Spectral Dual Adaptive Graph Embedding for Multispectral and Hyperspectral Image Fusion ".
Hyperspectral (HS) images have been widely applied to both computer vision and remote sensing fields thanks to more abundant spectral information than multispectral (MS) images. However, dozens or even hundreds of bands in HS images imply that their spatial resolution is lower than that of MS images owing to the physical tradeoffs between spatial and spectral resolutions. To obtain high spatial resolution (HR) HS images, researchers have developed many image fusion techniques for the super-resolution of low spatial resolution (LR) HS images with the help of HR MS images. Compared to LR HS images, the fused HR HS images provide more abundant spatial details. Meanwhile, the fused HR HS images have richer spectral information than HR MS images. Thus, using the fused HR HS images can further boost the accuracy of subsequent tasks.
Figure 1. SNS and SBC in images from the Harvard dataset and the neighbor preserving between the HR HS image and source images.
Though many methods have exploited the SNS and SBC in the HR HS image, it is difficult to model the priors explicitly because the HR HS image is unavailable in real scenes. As the low-dimensional degradation versions, HR MS and LR HS images inherit the SNS and SBC in the HR HS image, respectively. But existing methods seldom consider the inheritance of SNS and SBC between the two source images and the HR HS image.
In this paper, we propose a spatial-spectral dual adaptive graph embedding model to exploit the SNS and SBC in HR MS and LR HS images for the regularization of their fusion. Specifically, spatial and spectral graphs are constructed adaptively to describe the SNS in the HR MS image and the SBC in the LR HS image. Then, the two graphs are embedded into the features for the reconstruction of the HR HS image. In this way, it is explicit to ensure the consistency of SNS and SBC between the source images and the HR HS image. Experiments on three benchmark datasets demonstrate the effectiveness of our SDAGE method. The code can be downloaded from https://github.com/RSMagneto/SDAGE.
Figure 2: The architecture of proposed SDAGE.
Compared to state-of-the-art methods, our SDAGE achieves better performance on three datasets thanks to the stronger capacity to model the SNS and SBC in images. As shown in Figures 3-5, local regions (green rectangles) are magnified for further perception. Besides, to reduce the model size of SDAGE, we adopt the weight-sharing strategy and develop a smaller version SDAGE-S for this task. Because the reconstruction of fused HR HS images is constrained by spatial-spectral dual adaptive graphs, SDAGE-S and SDAGE demonstrate comparable fusion performance.
Figure 3. Fusion results on the CAVE dataset. RGB images are composed of the 24th, 15th, and 7th bands of the fusion results.
Figure 4. Fusion results on the Harvard dataset. RGB images are composed of the 24th, 15th, and 7th bands of the fusion results.
Figure 5. Fusion results on the Washington DC Mall dataset. RGB images are composed of the 52th, 30th, and 9th bands of the fusion results.
Associate Professor Kai Zhang (from Shandong Normal University) are corresponding authors of the paper. Xuquan Wang (postdoctoral fellow from Tongji University) is the first author of the paper. Collaborators with outstanding contributions to the paper also include Professor Jiande Sun (from Shandong Normal University), Feng Zhang (Lecturer from University of Jinan), Weijie Wang (PhD. from University of Trento), and Xiong Dun (Distinguished Research Fellow from Tongji University).
Information of the paper: https://doi.org/10.1016/j.patcog.2024.110365
