Submitted:
03 April 2024
Posted:
04 April 2024
You are already at the latest version
Abstract
Keywords:
1. Introduction
- We propose a method to simulate the actual degradation process for underwater images, enabling the network to better learn the mapping between high-resolution and low-resolution images, thereby enhancing the quality of reconstructed images.
- The adaptive residual attention module designed for underwater images automatically assesses image importance using an energy function and, when integrated into dense residual blocks, enhances the precision of key feature extraction and the effectiveness of super-resolution reconstruction.
- Experiments show that our method provides both high PSNR and low LPIPS, which has been considered a trade-off relation.
2. Related Work
2.1. Deep Networks for Image Super-Resolution
2.2. Degradation Models
2.3. Attention-Based Image Super-Resolution
3. Methods
3.1. A Practical Degradation Model
3.1.1. Resize
3.1.2. Noise
3.1.3. Blur
3.1.4. Suspended Particles
3.1.5. Validation of the Degradation Model Efficacy
3.2. Network Architecture
3.2.1. The Overall Structure
3.2.2. Attention Enhanced Residual Dense Block
3.3. Networks and Training
3.3.1. Generator
3.3.2. Discriminator with Spectral Normalization
3.3.3. Loss Function
4. Experiments
4.1. Datasets and Experiments Settings
4.1.1. Datasets
4.1.2. Implementation Details
4.1.3. Evaluation Metrics
4.2. Comparisons of Super-Resolution Results
4.2.1. Quantitative Results
4.2.2. Qualitative Results
4.3. Model Performance Evaluation on Test Datasets
4.4. Ablation Study
5. Conclusions
6. Patents
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Shi, A.; Ding, H. Underwater image super-resolution via dual-aware integrated network. Applied Sciences 2023, 13, 12985. [Google Scholar] [CrossRef]
- Dong, C.; Loy, C.C.; He, K.; Tang, X. Learning a deep convolutional network for image super-resolution. In Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, 2014; pp. 184–199.
- Dong, C.; Loy, C.C.; Tang, X. Accelerating the super-resolution convolutional neural network. In Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, 2016; pp. 391–407.
- Kong, X.; Zhao, H.; Qiao, Y.; Dong, C. Classsr: A general framework to accelerate super-resolution networks by data characteristic. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021; pp. 12016–12025.
- Li, Z.; Liu, Y.; Chen, X.; Cai, H.; Gu, J.; Qiao, Y.; Dong, C. Blueprint separable residual network for efficient image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022; pp. 833–843.
- Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; Fu, Y. Image super-resolution using very deep residual channel attention networks. In Proceedings of the Proceedings of the European conference on computer vision (ECCV), 2018; pp. 286–301.
- Shi, S.; Xiangli, B.; Yin, Z. Multiframe super-resolution of color images based on cross channel prior. Symmetry 2021, 13, 901. [Google Scholar] [CrossRef]
- Liang, J.; Sun, G.; Zhang, K.; Van Gool, L.; Timofte, R. Mutual affine network for spatially variant kernel estimation in blind image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021; pp. 4096–4105.
- Park, S.H.; Moon, Y.S.; Cho, N.I. Perception-oriented single image super-resolution using optimal objective estimation. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023; pp. 1725–1735.
- Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2017; pp. 4681–4690.
- Zhang, K.; Liang, J.; Van Gool, L.; Timofte, R. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021; pp. 4791–4800.
- Wang, X.; Xie, L.; Dong, C.; Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, 2021; pp. 1905–1914.
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30. [Google Scholar]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, 2021; pp. 10012–10022.
- Wu, C.; Wang, D.; Bai, Y.; Mao, H.; Li, Y.; Shen, Q. Hsr-diff: hyperspectral image super-resolution via conditional diffusion models. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023; pp. 7083–7093.
- Zhang, T.; Yang, J. Transformer with Hybrid Attention Mechanism for Stereo Endoscopic Video Super Resolution. Symmetry 2023, 15, 1947. [Google Scholar] [CrossRef]
- Zhao, Z.; Zhang, J.; Gu, X.; Tan, C.; Xu, S.; Zhang, Y.; Timofte, R.; Van Gool, L. Spherical space feature decomposition for guided depth map super-resolution. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023; pp. 12547–12558.
- Choi, H.; Lee, J.; Yang, J. N-gram in swin transformers for efficient lightweight image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023; pp. 2071–2081.
- Sharma, P.; Bisht, I.; Sur, A. Wavelength-based attributed deep neural network for underwater image restoration. ACM Transactions on Multimedia Computing, Communications and Applications 2023, 19, 1–23. [Google Scholar] [CrossRef]
- Zhang, Y.; Yang, S.; Sun, Y.; Liu, S.; Li, X. Attention-guided multi-path cross-CNN for underwater image super-resolution. Signal, Image and Video Processing 2022, 1-9.
- Wang, X.; Yu, K.; Wu, S.; Gu, J.; Liu, Y.; Dong, C.; Qiao, Y.; Change Loy, C. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the Proceedings of the European conference on computer vision (ECCV) workshops, 2018; pp. 0–0.
- Chen, Z.; Zhang, Y.; Gu, J.; Kong, L.; Yang, X.; Yu, F. Dual aggregation transformer for image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, 2023; pp. 12312–12321.
- Liu, Y.; Chu, Z. A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution. Symmetry 2023, 15, 885. [Google Scholar] [CrossRef]
- Gao, Y.; Liu, J.; Li, W.; Hou, M.; Li, Y.; Zhao, H. Augmented Grad-CAM++: Super-Resolution Saliency Maps for Visual Interpretation of Deep Neural Network. Electronics 2023, 12, 4846. [Google Scholar] [CrossRef]
- Mei, Y.; Fan, Y.; Zhou, Y. Image super-resolution with non-local sparse attention. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021; pp. 3517–3526.
- Niu, B.; Wen, W.; Ren, W.; Zhang, X.; Yang, L.; Wang, S.; Zhang, K.; Cao, X.; Shen, H. Single image super-resolution via a holistic attention network. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, 2020; pp. 191–207.
- Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; Aila, T. Analyzing and improving the image quality of stylegan. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020; pp. 8110–8119.
- Lim, B.; Son, S.; Kim, H.; Nah, S.; Mu Lee, K. Enhanced deep residual networks for single image super-resolution. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017; pp. 136–144.
- Zhang, Y.; Tian, Y.; Kong, Y.; Zhong, B.; Fu, Y. Residual dense network for image super-resolution. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2018; pp. 2472–2481.
- Zhou, S.; Zhang, J.; Zuo, W.; Loy, C.C. Cross-scale internal graph neural network for image super-resolution. Advances in neural information processing systems 2020, 33, 3499–3509. [Google Scholar]
- Wang, Z.; Cun, X.; Bao, J.; Zhou, W.; Liu, J.; Li, H.U. A general u-shaped transformer for image restoration. arXiv 2021. arXiv preprint arXiv:2106.03106, arXiv:2106.03106.
- Li, G.; Zhao, L.; Sun, J.; Lan, Z.; Zhang, Z.; Chen, J.; Lin, Z.; Lin, H.; Xing, W. Rethinking Multi-Contrast MRI Super-Resolution: Rectangle-Window Cross-Attention Transformer and Arbitrary-Scale Upsampling. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023; pp. 21230–21240.
- Li, A.; Zhang, L.; Liu, Y.; Zhu, C. Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023; pp. 12514–12524.
- Zhou, Y.; Li, Z.; Guo, C.-L.; Bai, S.; Cheng, M.-M.; Hou, Q. Srformer: Permuted self-attention for single image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023; pp. 12780–12791.
- Liang, J.; Cao, J.; Sun, G.; Zhang, K.; Van Gool, L.; Timofte, R. Swinir: Image restoration using swin transformer. In Proceedings of the Proceedings of the IEEE/CVF international conference on computer vision, 2021; pp. 1833–1844.
- Lai, W.-S.; Huang, J.-B.; Ahuja, N.; Yang, M.-H. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2017; pp. 624–632.
- Zhang, K.; Zuo, W.; Zhang, L. Deep plug-and-play super-resolution for arbitrary blur kernels. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019; pp. 1671–1681.
- Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; Fu, Y. Image super-resolution using very deep residual channel attention networks. In Proceedings of the Proceedings of the European conference on computer vision (ECCV), 2018; pp. 286–301.
- Dai, T.; Cai, J.; Zhang, Y.; Xia, S.-T.; Zhang, L. Second-order attention network for single image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019; pp. 11065–11074.
- Mei, Y.; Fan, Y.; Zhou, Y. Image super-resolution with non-local sparse attention. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021; pp. 3517–3526.
- Park, J.; Son, S.; Lee, K.M. Content-aware local gan for photo-realistic super-resolution. In Proceedings of the Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023; pp. 10585–10594.
- Yang, L.; Zhang, R.-Y.; Li, L.; Xie, X. Simam: A simple, parameter-free attention module for convolutional neural networks. In Proceedings of the International conference on machine learning; 2021; pp. 11863–11874. [Google Scholar]
- Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the Proceedings of the IEEE international conference on computer vision, 2017; pp. 618–626.
- Islam, M.J.; Xia, Y.; Sattar, J. Fast underwater image enhancement for improved visual perception. IEEE Robotics and Automation Letters 2020, 5, 3227–3234. [Google Scholar] [CrossRef]
- Islam, M.J.; Luo, P.; Sattar, J. Simultaneous enhancement and super-resolution of underwater imagery for improved visual perception. arXiv preprint arXiv:2002.01155, arXiv:2002.01155 2020.
- Sharma, P.; Bisht, I.; Sur, A. Wavelength-based attributed deep neural network for underwater image restoration. ACM Transactions on Multimedia Computing, Communications and Applications 2023, 19, 1–23. [Google Scholar] [CrossRef]
- Chen, Z.; Liu, C.; Zhang, K.; Chen, Y.; Wang, R.; Shi, X. Underwater-image super-resolution via range-dependency learning of multiscale features. Computers and Electrical Engineering 2023, 110, 108756. [Google Scholar] [CrossRef]
- Islam, M.J.; Enan, S.S.; Luo, P.; Sattar, J. Underwater image super-resolution using deep residual multipliers. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA); 2020; pp. 900–906. [Google Scholar]
- Zhang, Y.; Yang, S.; Sun, Y.; Liu, S.; Li, X. Attention-guided multi-path cross-CNN for underwater image super-resolution. Signal, Image and Video Processing 2022, 1-9.
- Fang, J.; Lin, H.; Chen, X.; Zeng, K. A hybrid network of cnn and transformer for lightweight image super-resolution. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022; pp. 1103–1112.
- Ren, T.; Xu, H.; Jiang, G.; Yu, M.; Zhang, X.; Wang, B.; Luo, T. Reinforced swin-convs transformer for simultaneous underwater sensing scene image enhancement and super-resolution. IEEE Transactions on Geoscience and Remote Sensing 2022, 60, 1–16. [Google Scholar] [CrossRef]
- Yue, Z.; Zhao, Q.; Xie, J.; Zhang, L.; Meng, D.; Wong, K.-Y.K. Blind image super-resolution with elaborate degradation modeling on noise and kernel. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022; pp. 2128–2138.








| Scale | Method | PSNR(dB)↑ | SSIM↑ | UIQM↑ | LPIPS↓ |
|---|---|---|---|---|---|
| ×2 | SRCNN | 26.81 | 0.76 | 2.59 | 0.56 |
| VDSR | 27.98 | 0.79 | 2.61 | 0.53 | |
| SRGAN | 26.68 | 0.73 | 2.55 | 0.30 | |
| ESRGAN | 28.08 | 0.76 | 2.59 | 0.24 | |
| BSRGAN | 28.15 | 0.79 | 2.63 | 0.20 | |
| Real-ESRGAN | 28.86 | 0.80 | 2.68 | 0.19 | |
| Deep WaveNet | 29.09 | 0.83 | 2.72 | 0.44 | |
| RDLN | 29.76 | 0.82 | 2.74 | 0.29 | |
| DAIN | 29.97 | 0.84 | 2.77 | 0.43 | |
| DAE-GAN(ours) | 29.95 | 0.85 | 2.80 | 0.19 | |
| ×4 | SRCNN | 23.68 | 0.65 | 2.38 | 0.71 |
| VDSR | 24.70 | 0.69 | 2.44 | 0.67 | |
| SRGAN | 23.46 | 0.63 | 2.38 | 0.48 | |
| ESRGAN | 24.50 | 0.67 | 2.45 | 0.40 | |
| BSRGAN | 25.05 | 0.69 | 2.47 | 0.32 | |
| Real-ESRGAN | 25.11 | 0.71 | 2.50 | 0.33 | |
| Deep WaveNet | 25.40 | 0.73 | 2.53 | 0.61 | |
| RDLN | 25.59 | 0.71 | 2.58 | 0.50 | |
| DAIN | 26.16 | 0.73 | 2.64 | 0.63 | |
| DAE-GAN(ours) | 26.23 | 0.75 | 2.68 | 0.31 | |
| ×8 | SRCNN | 19.97 | 0.57 | 2.01 | 0.86 |
| VDSR | 20.15 | 0.61 | 2.09 | 0.83 | |
| SRGAN | 19.83 | 0.54 | 1.98 | 0.61 | |
| ESRGAN | 20.08 | 0.57 | 2.02 | 0.54 | |
| BSRGAN | 20.33 | 0.59 | 2.07 | 0.42 | |
| Real-ESRGAN | 20.45 | 0.62 | 2.10 | 0.44 | |
| Deep WaveNet | 21.70 | 0.63 | 2.13 | 0.72 | |
| RDLN | 22.40 | 0.62 | 2.19 | 0.66 | |
| DAIN | 22.86 | 0.63 | 2.17 | 0.69 | |
| DAE-GAN(ours) | 23.83 | 0.64 | 2.20 | 0.40 |
| Method | PSNR(dB)↑ | SSIM↑ | UIQM↑ | LPIPS↓ | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ×2 | ×3 | ×4 | ×2 | ×3 | ×4 | ×2 | ×3 | ×4 | ×2 | ×3 | ×4 | |
| SRCNN | 24.75 | 22.22 | 19.05 | 0.72 | 0.65 | 0.56 | 2.39 | 2.24 | 2.12 | 0.56 | 0.65 | 0.71 |
| SRGAN | 25.11 | 23.01 | 19.93 | 0.75 | 0.70 | 0.58 | 2.44 | 2.39 | 2.35 | 0.24 | 0.33 | 0.37 |
| Deep WaveNet | 25.71 | 25.23 | 23.26 | 0.77 | 0.76 | 0.73 | 2.89 | 2.86 | 2.85 | 0.40 | - | 0.53 |
| AMPCNet | 25.24 | 25.43 | 25.08 | 0.71 | 0.70 | 0.70 | 2.76 | 2.65 | 2.68 | 0.31 | - | 0.47 |
| ESRGCNN | 25.82 | 25.98 | 24.70 | 0.73 | 0.71 | 0.71 | 2.88 | 2.86 | 2.75 | 0.34 | 0.46 | 0.51 |
| HNCT | 25.73 | 25.86 | 24.91 | 0.72 | 0.73 | 0.70 | 2.76 | 2.78 | 2.64 | 0.27 | 0.40 | 0.47 |
| URSCT | 25.96 | - | 25.37 | 0.81 | - | 0.69 | - | - | - | 0.37 | 0.49 | 0.50 |
| RDLN | 26.20 | 26.13 | 25.56 | 0.78 | 0.74 | 0.73 | 2.87 | 2.84 | 2.83 | 0.29 | 0.37 | 0.39 |
| DAE-GAN(ours) | 26.26 | 26.19 | 25.89 | 0.80 | 0.76 | 0.74 | 2.88 | 2.87 | 2.85 | 0.19 | 0.25 | 0.30 |
| Method | Scale | EUVP | SQUID | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PSNR↑ | SSIM↑ | UIQM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | UIQM↑ | LPIPS↓ | ||
| SRCNN | ×4 | 23.64 | 0.63 | 2.37 | 0.70 | 23.66 | 0.65 | 2.38 | 0.70 |
| SRGAN | 23.31 | 0.59 | 2.38 | 0.48 | 23.37 | 0.63 | 2.40 | 0.45 | |
| ESRGAN | 24.40 | 0.66 | 2.44 | 0.38 | 23.62 | 0.68 | 2.48 | 0.38 | |
| BSRGAN | 24.89 | 0.70 | 2.47 | 0.32 | 25.11 | 0.72 | 2.51 | 0.31 | |
| Real-ESRGAN | 25.01 | 0.73 | 2.48 | 0.33 | 25.23 | 0.75 | 2.53 | 0.33 | |
| PDM-SRGAN | 25.89 | 0.74 | - | 0.29 | 26.04 | 0.74 | - | 0.28 | |
| BSRDM | 26.35 | 0.76 | 2.40 | 0.38 | 26.40 | 0.73 | 2.43 | 0.36 | |
| CAL-GAN | 26.09 | 0.71 | - | 0.31 | 26.11 | 0.69 | - | 0.29 | |
| DAE-GAN(ours) | 26.33 | 0.78 | 2.63 | 0.30 | 26.41 | 0.77 | 2.68 | 0.29 | |
| DM | ARAM | PSNR↑ | SSIM↑ | UIQM↑ | LPIPS↓ |
|---|---|---|---|---|---|
| 24.50 | 0.66 | 2.45 | 0.40 | ||
| √ | 25.02 | 0.68 | 2.64 | 0.34 | |
| √ | 25.89 | 0.72 | 2.47 | 0.37 | |
| √ | √ | 26.23 | 0.75 | 2.68 | 0.31 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).