Transformer-based Multi-scale Underwater Image Enhancement Network

doi:10.12068/j.issn.1005-3026.2024.12.004

Abstract

Abstract:

CNN（convolutional neural network）‑based underwater image enhancement methods neglect global visual perception， leading to color distortion and contrast degradation.

A Transformer‑based multi‑scale underwater image enhancement network （MTransNet） is proposed. To address the problem of lacking global visual perception， a position encoding module is designed based on underwater image priors and a Swin Transformer module which is applicable to underwater scenes is constructed. Furthermore， self‑attention mechanism is built to improve global perception performance. As for the detail blurring that exists in current methods， a CNN module is developed to capture local features such as textures or edges， to improve local perception performance. The transfer fusion module is built to transfer global attention of Swin Transformer to local convolutional feature， achieving full fusion and utilization of global feature and local feature. The PSNR value on subsets of EUVP can reach up to 23.47 dB， which demonstrates the method can significantly enhance global visual perception and increase image visual quality.

Key words: underwater image enhancement, deep learning, Transformer, convolutional neural network, transfer fusion

CLC Number:

TP 391

Ai-ping YANG, Si-jie FANG, Ming-fu SHAO, Teng-fei ZHANG. Transformer-based Multi-scale Underwater Image Enhancement Network[J]. Journal of Northeastern University(Natural Science), 2024, 45(12): 1696-1705.

Figures/Tables 9

Fig.1 Network structure

Fig.2 Position encoding module

Fig.3 Swin Transformer module

Fig.4 Transfer fusion module

Fig.5 Comparison results of Test-910 dataset（f）—U-GAN；（g）—Water-Net；（h）—Ucolor；（i）—MTransNet.

Fig.6 Comparison results of Test-R90 dataset（f）—U-GAN；（g）—Water-Net；（h）—Ucolor；（i）—MTransNet.

Fig.7 Comparison results of Test-UIQS dataset（f）—U-GAN；（g）—Water-Net；（h）—Ucolor；（i）—MTransNet.

Table 1 Objective evaluation results of comparative methods

方法	Test-910		Test-R90		Test-UIQS
方法	PSNR/dB	SSIM	PSNR/dB	SSIM	US	UCIQE	UIQM
Input	17.24	0.79	16.31	0.79	1.00	0.48	0.84
fusion-based^[21]	21.83	0.83	17.83	0.82	2.80	0.62	1.22
Retinex-based^[22]	20.06	0.82	18.09	0.83	2.60	0.71	1.14
MMLE^[23]	22.34	0.85	18.44	0.85	3.00	0.64	1.34
FUnIE-GAN^[9]	23.06	0.85	17.64	0.79	1.60	0.55	0.81
UGAN^[24]	20.76	0.80	17.32	0.77	1.40	0.51	0.77
Water-Net^[10]	21.41	0.83	18.90	0.87	2.40	0.56	0.97
Ucolor^[11]	20.52	0.81	18.41	0.85	2.10	0.53	0.88
MTransNet	23.47	0.84	19.52	0.86	2.50	0.62	1.21

Fig.8 Ablation experiment

References 24

1	Jiang Q P， Gu Y S， Li C Y，et al.Underwater image enhancement quality evaluation：benchmark dataset and objective metric［J］.IEEE Transactions on Circuits and Systems for Video Technology，2022，32（9）：5959-5974.
2	Raveendran S， Patil M D， Birajdar G K.Underwater image enhancement：a comprehensive review，recent trends，challenges and applications［J］.Artificial Intelligence Review，2021，54（7）：5413-5467.
3	Hummel R.Image enhancement by histogram transformation［J］.Computer Graphics and Image Processing，1977，6（2）：184-195.
4	Reza A M.Realization of the contrast limited adaptive histogram equalization （CLAHE） for real‑time image enhancement［J］.The Journal of VLSI Signal Processing Systems for Signal，Image，and Video Technology，2004，38（1）：35-44.
5	Hitam M S， Yussof W N J H W， Awalludin E A，et al.Mixture contrast limited adaptive histogram equalization for underwater image enhancement［C］//2013 International Conference on Computer Applications Technology （ICCAT）.Sousse，2013：1-5.
6	McGlamery B L.A computer model for underwater camera systems［J］.Proceedings of the SPIE，1980，208：221-231.
7	Jaffe J S.Computer modeling and the design of optimal underwater imaging systems［J］.IEEE Journal of Oceanic Engineering，1990，15（2）：101-111.
8	Wang Y， Zhang J， Cao Y，et al.A deep CNN method for underwater image enhancement［C］//2017 IEEE International Conference on Image Processing （ICIP）.Beijing，2017：1382-1386.
9	Islam M J， Xia Y Y， Sattar J.Fast underwater image enhancement for improved visual perception［J］.IEEE Robotics and Automation Letters，2020，5（2）：3227-3234.
10	Li C Y， Guo C L， Ren W Q，et al.An underwater image enhancement benchmark dataset and beyond［J］.IEEE Transactions on Image Processing，2020，29：4376-4389.
11	Li C Y， Anwar S， Hou J H，et al.Underwater image enhancement via medium transmission‑guided multi‑color space embedding［J］.IEEE Transactions on Image Processing，2021，30：4985-5000.
12	Qi Q， Zhang Y C， Tian F，et al.Underwater image co‑enhancement with correlation feature matching and joint learning［J］.IEEE Transactions on Circuits and Systems for Video Technology，2022，32（3）：1133-1147.
13	Vaswani A， Shazeer N， Parmar N，et al. Attention is all you need［J］.Advances in Neural Information Processing Systems，2017，30：6000-6010.
14	Liu Z， Mao H Z， Wu C Y，et al. A ConvNet for the 2020s［C］// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans，2022：11976-11986.
15	Fan S J， Liang W， Ding D R，et al.LACN：a lightweight attention‑guided ConvNeXt network for low‑light image enhancement［J］.Engineering Applications of Artificial Intelligence，2023，117：105632.
16	Liu Z， Lin Y T， Cao Y，et al.Swin Transformer：hierarchical vision Transformer using shifted windows［C］//2021 IEEE/CVF International Conference on Computer Vision.Montreal，2021：9992-10002.
17	Guo C L， Yan Q X， Anwar S，et al.Image dehazing Transformer with transmission‑aware 3D position embedding［C］//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans，2022：5802-5810.
18	Jiang L M， Zhang C X， Huang M Y，et al.TSIT：a simple and versatile framework for image‑to‑image translation［C］//Computer Vision-ECCV 2020：16th European Conference Proceedings，Part III 16.Glasgow，2020：206-222.
19	Wang X T， Yu K， Dong C，et al.Recovering realistic texture in image super‑resolution by deep spatial feature transform［C］//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City，2018：606-615.
20	Liu R S， Fan X， Zhu M，et al.Real‑world underwater enhancement：challenges，benchmarks，and solutions under natural light［J］.IEEE Transactions on Circuits and Systems for Video Technology，2020，30（12）：4861-4875.
21	Ancuti C O， Ancuti C， De Vleeschouwer C，et al.Color balance and fusion for underwater image enhancement［J］.IEEE Transactions on Image Processing，2018，27（1）：379-393.
22	Fu X Y， Zhuang P X， Huang Y，et al.A retinex‑based enhancing approach for single underwater image［C］//2014 IEEE International Conference on Image Processing （ICIP）.Paris，2014：4572-4576.
23	Zhang W D， Zhuang P X， Sun H H，et al.Underwater image enhancement via minimal color loss and locally adaptive contrast enhancement［J］.IEEE Transactions on Image Processing，2022，31：3997-4010.
24	Fabbri C， Islam M J， Sattar J.Enhancing underwater imagery using generative adversarial networks［C］//2018 IEEE International Conference on Robotics and Automation （ICRA）.Brisbane，2018：7159-7165.

[1]	Ji-hong LIU, Xi-xiong HUANG. Face Inpainting Model Based on Denoising Diffusion Probability Models [J]. Journal of Northeastern University(Natural Science), 2024, 45(9): 1227-1234.
[2]	Yan LIU, Qi-jie BU, Hong-chen ZHAO, Xin GUO. Operating Performance Assessment of Flotation Process Based on Multi-source Heterogeneous Information [J]. Journal of Northeastern University(Natural Science), 2024, 45(9): 1217-1226.
[3]	Zhi WANG, Kun WANG, Meng-qing WANG. Super-resolution Reconstruction of Remote Sensing Image Based on Transformer of Multi-scale Feature Fusion [J]. Journal of Northeastern University(Natural Science), 2024, 45(8): 1178-1184.
[4]	Wei-wei LIU, Bing-jun LIU, Huan-qiang LIU, Ze-yuan LIU. Defect Identification Method for Laser Melting Deposition Process [J]. Journal of Northeastern University(Natural Science), 2024, 45(8): 1150-1158.
[5]	Tao REN, Xin-liang LIU, Hong-feng CHEN, Yan-lu MA. Segmented Estimation Method for Early Warning Magnitude Based on Convolutional Neural Network [J]. Journal of Northeastern University(Natural Science), 2024, 45(8): 1073-1079.
[6]	Hong-li LI, Hao-yu LIU, Rong-hua ZHANG, Yi CHENG. Emotional Classification Based on Multidimensional Feature Matrix and Improved Dense Connection Network [J]. Journal of Northeastern University(Natural Science), 2024, 45(7): 928-935.
[7]	Ji-hong LIU, Lü-heng ZHANG, Hai-xu YANG. A Saturation Artifact Inpainting Algorithm for Cell Fluorescence Microscopic Images [J]. Journal of Northeastern University(Natural Science), 2024, 45(7): 921-927.
[8]	Pei-yun XUE, Jing BAI, Nan ZHANG, Jian-xing ZHAO. VMD Based Binary Channels Speech Feature Map Extraction Algorithm for Dysarthria [J]. Journal of Northeastern University(Natural Science), 2024, 45(6): 793-801.
[9]	Dong-hong HAN, Yan-ru KONG, Yi-meng ZHAN, Yuan LIU. Research on Emotion Recognition Method of Music Multimodal Data [J]. Journal of Northeastern University(Natural Science), 2024, 45(6): 776-785.
[10]	Wei-qi ZHANG, Hui-ming WANG. Interpretable Deep Learning Prediction Model for Compressive Strength of Concrete [J]. Journal of Northeastern University(Natural Science), 2024, 45(5): 738-744.
[11]	Peng SHAN, Lin ZHANG, Hong-ming XIAO, Yu-liang ZHAO. CT Diagnosis Method for Coronavirus Pneumonia with Integrated Multi-scale Attention Mechanism [J]. Journal of Northeastern University(Natural Science), 2024, 45(12): 1673-1679.
[12]	Meng-yuan LIU, Zhao-xia WU, Jin-yang WANG, Guang-lei XIA. Air Permeability Prediction of Sinter Layer Based on TST-LSTM Model [J]. Journal of Northeastern University(Natural Science), 2024, 45(10): 1379-1385.
[13]	WEI Jian-yi， WU Jing-jing. Resource Allocation Algorithm in Industrial Internet of Things Based on Edge Computing [J]. Journal of Northeastern University(Natural Science), 2023, 44(8): 1072-1078.
[14]	YANG Xuan， HE Zhan-qi. Improved Two-layer BiLSTM Electrocardiosignal Segmentation Method [J]. Journal of Northeastern University(Natural Science), 2023, 44(12): 1705-1711.
[15]	CHEN Cheng， SHI Pei-xin， WANG Zhan-sheng， JIA Peng-jiao. Shield Load Prediction Method Based on Deep Learning with Multiattention Mechanism [J]. Journal of Northeastern University(Natural Science), 2023, 44(11): 1631-1638.