Two-step ResUp&Down generative adversarial network to reconstruct multispectral image from aerial RGB image

文献类型： 外文期刊

第一作者： Zhang, Yanchao

作者： Zhang, Yanchao;Yang, Wen;Zhang, Wenbo;Yu, Jiya;Zhang, Jianxin;Yang, Yongjie;Lu, Yongliang;Tang, Wei

作者机构：

关键词： Multispectral image reconstruction; Images registration; Generative adversarial network; UAV; ResNet

期刊名称：COMPUTERS AND ELECTRONICS IN AGRICULTURE （ 影响因子：6.757； 五年影响因子：6.817 ）

ISSN： 0168-1699

年卷期： 2022 年 192 卷

页码：

收录情况： SCI

摘要： Convolutional neural network has brought breakthroughs on multispectral image reconstruction research. Previous work has largely focused on reconstructing MSI using the R-G-B channels from the MSI as inputs of the model. However, it's image manipulation rather than practical use. In real application, to reconstruct multispectral image using images from RGB camera is a research that has hardly been studied. In this research, high resolution aerial RGB images are collected by drone with RGB camera and multispectral images are collected by drone with RedEdge-M multispectral Camera. Then a new two-step Generative Adversarial Network (GAN)-based reconstruction method was proposed as follows: At first, MSI and RGB images are carefully registered to make sure that pixels are one-one correspondent. Then two data sources are cropped to form dataset. After that, a novel R-MSI GAN using is proposed. It uses a ResUp&Down block to replace the ResNet block of the Generator network and it outperforms ResNet-based GAN. The experimental results show that: (1) the combination of Mean Square Error and Discriminator (MSE-D) can alleviate the problem of the high-frequency loss of generated images. (2) The root means square error (RMSE), mean relative absolute error (MRAE) and Structural Similarity (SSIM) can only reflect overall error but can't reflect details in reconstructed images, while different bands' statistical histogram can present the total high-frequency loss of generated bands. (3) 3 indexes, which are intersection over union (IoU) based normalized difference vegetation index (NDVI)-IoU, normalized difference red edge (NDRE)-IoU and enhance vegetation index (EVI)-IoU, were defined to verify the effect of the generated MSI and they show good consistence with vegetation index map. 4 In comparisons among ResNet-based GAN, single step ResUp&Down GAN and two-step ResUp&Down GAN(T-GAN) with 3 loss functions (L1, MSE, Discriminator), the two-step ResUp&Down GAN(T-GAN) with MSE-D loss function performs best in reconstructing RGB bands. The T-GAN with L1loss-D (mean absolute error loss) performs best in reconstructing NIR and rededge bands. In summary, the proposed methods can effectively reconstruct MSI using images from RGB camera at drone based remote sensing.

分类号：

  • 相关文献
作者其他论文 更多>>

微信小程序