Face Inpainting Model Based on Denoising Diffusion Probability Models

doi:10.12068/j.issn.1005-3026.2024.09.002

Abstract

Abstract:

A face inpainting model based on the denoising diffusion probability model is proposed aiming at the problems of poor image quality， blurred repair edges， complex model， and difficult training of the mainstream face inpainting model after image inpainting. By improving the denoising diffusion probability model， the U-Net network structure in Guided?diffusion is adopted. The fast Fourier convolution is introduced into the network， and then the model is trained and tested on the CelebA-HQ high?definition face image dataset. The experimental results show that the improved denoising diffusion probability model can achieve a PSNR of 25.01 and a SSIM of 0.886 compared to the original image， when inpainting face images with random mask， both of which are better than the model before improvement and the existing face image inpainting model based on generative adversarial networks.

Key words: deep learning, face inpainting, denoising diffusion probability models, fast Fourier convolution, U-Net network

CLC Number:

TP 391.41

Ji-hong LIU, Xi-xiong HUANG. Face Inpainting Model Based on Denoising Diffusion Probability Models[J]. Journal of Northeastern University(Natural Science), 2024, 45(9): 1227-1234.

Figures/Tables 11

Fig.1 Experimental flowchart

Fig.2 Denoising diffusion probability model

Fig.3 Fast Fourier convolution

Fig.4 Comparison of autoencoder and U-Net structures

Fig.5 Improved U-Net network structure

Fig.6 Model forward propagation process

Fig.7 Comparison of two convolution outputs

Table 1 Comparison of evaluation indicators before and after model improvement

模型	PSNR	SSIM
不使用快速傅里叶卷积	24.28	0.873
使用快速傅里叶卷积	24.93	0.882

Table 2 Comparison of evaluation results with other inpainting models under the center mask

模型	PSNR	SSIM
GMCnn	23.73	0.779
Gated Conv^[4]	23.70	0.780
Edge Connect	23.28	0.776
AOT-GAN	24.47	0.849
本文方法	24.93	0.882

Table 3 Comparison of evaluation results with other inpainting models under the random mask

模型	PSNR	SSIM
GMCnn	23.98	0.781
Gated Conv^[4]	24.10	0.790
Edge Connect	23.57	0.794
AOT-GAN	24.83	0.852
本文方法	25.01	0.886

Fig.8 Face inpainting results

References 23

1	刘颖，佘建初，公衍超，等.基于深度学习的面部修复技术综述［J］.计算机应用研究，2021，38（1）：9-14.
	Liu Ying， She Jian‑chu， Gong Yan‑chao，et al.Survey of facial completion techniques based on deep learning［J］.Application Research of Computers，2021，38（1）：9-14.
2	Goodfellow I J， Pouget‑Abadie J， Mirza M，et al.Generative adversarial nets［C］//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal，2014：2672-2680.
3	Li Y J， Liu S F， Yang J M，et al.Generative face completion［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）.Honolulu，2017：5892-5900.
4	Yu J H， Lin Z， Yang J M，et al.Generative image inpainting with contextual attention［C］//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City，2018：5505-5514.
5	Nazeri K， Eric N， Joseph T，et al.Edge connect：generative image inpainting with adversarial edge learning［J］.arXiv Preprint arXiv，2019：1901.00212.
6	Zeng Y H， Fu J L， Chao H Y，et al.Aggregated contextual transformations for high‑resolution image inpainting［J］.IEEE Transactions on Visualization and Computer Graphics，2023，29（7）：3266-3280.
7	Vaswani A， Shazeer N， Parmar N，et al.Attention is all you need［C］//Proceedings of the 31st International Conference on Neural Information Processing Systems.Long Beach，2017：6000-6010.
8	Lee K， Chang H W， Jiang L，et al.ViTGAN：training GANs with vision transformers［J］.arXiv Preprint arXiv，2021：2107.04589.
9	Ho J， Jain A， Abbeel P.Denoising diffusion probabilistic models［C］//Proceedings of the 34th International Conference on Neural Information Processing Systems.Vancouver，2020：6840-6851.
10	Ronneberger O， Fischer P， Brox T.U‑net：convolutional networks for biomedical image segmentation［C］//Medical I m a g e C o m p u t i n g a n d C o m p u t e r ‑ A s s i s t e d I n t e r v e n t i o n ‑ M I C C A I
	2015．Munich，German：Springer International Publishing，2015：234-241.
11	Dhariwal P， Nichol A.Diffusion models beat gans on image synthesis［J］.Advances in Neural Information Processing Systems，2021，34：8780-8794.
12	Saharia C， Chan W， Chang H W，et al.Palette：image‑to‑image diffusion models［C］//SIGGRAPH ’22：ACM SIGGRAPH 2022 Conference Proceedings.Vancouver，2022：1-10.
13	Chi L， Jiang B， Mu Y.Fast Fourier convolution ［J］.Advances in Neural Information Processing Systems，2020，33：4479-4488.
14	Karras T， Aila T， Laine S，et al.Progressive growing of gans for improved quality，stability，and variation［J］.arXiv Preprint arXiv，2017：1710.10196.
15	Yang L， Zhang Z L， Song Y，et al.Diffusion models：a comprehensive survey of methods and applications［J］.arXiv Preprint arXiv，2022：2209.00796.
16	Suvorov R， Logacheva E， Mashikhin A，et al.Resolution‑robust large mask inpainting with Fourier convolutions［C］//2022 IEEE/CVF Winter Conference on Applications of Computer Vision （WACV）.Waikoloa，2022：3172-3182.
17	He K M， Zhang X Y， Ren S Q，et al.Deep residual learning for image recognition［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）.Las Vegas，2016：770-778.
18	Liu Z W， Luo P， Wang X G，et al.Large‑scale celebfaces attributes （CelebA） dataset［DS/OL］.［2022-11-01］..
19	Saharia C， Ho J， Chan W，et al.Image super‑resolution via iterative refinement［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence，2023，45（4）：4713-4726.
20	Voronin V V， Sizyakin R A， Marchuk V I，et al.Video inpainting of complex scenes based on local statistical model［J］.Electronic Imaging，2016，28（15）：1-6.
21	何凯，牛俊慧，沈成南，等.基于SSIM的自适应样本块图像修复算法［J］.天津大学学报（自然科学与工程技术版），2018，51（7）：763-767.
	He Kai， Niu Jun‑hui， Shen Cheng‑nan，et al.Image inpainting algorithm with adaptive patch using SSIM［J］.Journal of Tianjin University （Science and Technology），2018，51（7）：763-767.
22	Wang Y， Tao X， Qi X J，et al.Image inpainting via generative multi‑column convolutional neural networks［C］//Proceedings of the 32nd International Conference on Neural Information Processing Systems.Montréal，2018：329-338.
23	Song J M， Meng C L， Ermon S.Denoising diffusion implicit models［J］.arXiv Preprint arXiv，2020：2010.02502.

[1]	Yan LIU, Qi-jie BU, Hong-chen ZHAO, Xin GUO. Operating Performance Assessment of Flotation Process Based on Multi-source Heterogeneous Information [J]. Journal of Northeastern University(Natural Science), 2024, 45(9): 1217-1226.
[2]	Ji-hong LIU, Lü-heng ZHANG, Hai-xu YANG. A Saturation Artifact Inpainting Algorithm for Cell Fluorescence Microscopic Images [J]. Journal of Northeastern University(Natural Science), 2024, 45(7): 921-927.
[3]	Hong-li LI, Hao-yu LIU, Rong-hua ZHANG, Yi CHENG. Emotional Classification Based on Multidimensional Feature Matrix and Improved Dense Connection Network [J]. Journal of Northeastern University(Natural Science), 2024, 45(7): 928-935.
[4]	Dong-hong HAN, Yan-ru KONG, Yi-meng ZHAN, Yuan LIU. Research on Emotion Recognition Method of Music Multimodal Data [J]. Journal of Northeastern University(Natural Science), 2024, 45(6): 776-785.
[5]	Wei-qi ZHANG, Hui-ming WANG. Interpretable Deep Learning Prediction Model for Compressive Strength of Concrete [J]. Journal of Northeastern University(Natural Science), 2024, 45(5): 738-744.
[6]	Hao SUN, Zong-sheng DAI, Ai-bing JIN, Yan CHEN. Intelligent Identification and Parameter Extraction of Key Joints in Rock （Mass） Based on AttentionR2U-net [J]. Journal of Northeastern University(Natural Science), 2024, 45(1): 101-110.
[7]	WEI Jian-yi， WU Jing-jing. Resource Allocation Algorithm in Industrial Internet of Things Based on Edge Computing [J]. Journal of Northeastern University(Natural Science), 2023, 44(8): 1072-1078.
[8]	YANG Xuan， HE Zhan-qi. Improved Two-layer BiLSTM Electrocardiosignal Segmentation Method [J]. Journal of Northeastern University(Natural Science), 2023, 44(12): 1705-1711.
[9]	JI Ce， WANG Xin， GENG Rong， LIANG Min-jun. LSTM-Based Channel Estimation Method in Time-Varying Channels [J]. Journal of Northeastern University(Natural Science), 2023, 44(11): 1521-1528.
[10]	CHEN Cheng， SHI Pei-xin， WANG Zhan-sheng， JIA Peng-jiao. Shield Load Prediction Method Based on Deep Learning with Multiattention Mechanism [J]. Journal of Northeastern University(Natural Science), 2023, 44(11): 1631-1638.
[11]	LI Juan-li， WEI Dai-liang， LI Bo， WEN Xiao. Improved SSD Rapid Separation Model of Coal Gangue Based on Deep Learning and Light-Weighting [J]. Journal of Northeastern University(Natural Science), 2023, 44(10): 1474-1480.
[12]	ZHAO Yong， JIAO Shi-hui， ZHAO Qian-bai. Hybrid Recognition Model of Microseismic Signals for Mining Based on Mel Spectrum and LSTM-DCNN [J]. Journal of Northeastern University(Natural Science), 2023, 44(10): 1481-1489.
[13]	ZHANG Xue-feng， WANG Zhao-yi. Automatic Lane Change Decision Model Based on Dueling Double Deep Q-network [J]. Journal of Northeastern University(Natural Science), 2023, 44(10): 1369-1376.
[14]	WANG Ying， WANG Ze-hao， LI Hong， HUANG Wen-jun. Named Entity Recognition in Threat Intelligence Domain Based on Deep Learning [J]. Journal of Northeastern University(Natural Science), 2023, 44(1): 33-39.
[15]	GU De-ying， LUO Yu-lun， LI Wen-chao. Traffic Target Detection in Complex Scenes Based on Improved YOLOv5 Algorithm [J]. Journal of Northeastern University(Natural Science), 2022, 43(8): 1073-1079.