基于U-net边缘生成和超图卷积的两阶段修复算法

doi:10.12068/j.issn.1005-3026.2023.03.004

摘要/Abstract

摘要： 为了有效修复背景复杂、大面积不规则缺失区域，得到合理的结构和精细的纹理，提出了基于U-net边缘生成和超图卷积的两阶段修复算法.首先，将缺失图像输入基于U-net门控卷积的粗修复网络，通过跳跃连接将图像的上下文信息向深层传播，获取丰富的图像细节信息，下采样提取缺失区域边缘特征，上采样还原缺失区域边缘细节，同时使用混合空洞卷积增大信息感受野，获取细节纹理信息.然后，将粗修复结果输入含超图卷积的细修复网络，捕获和学习输入图像中的超图结构，使用空间特征的互相关矩阵捕获空间特征结构，改善结构完整性并提升细节细粒度.最后，将细修复结果输入鉴别器进行判别优化，进一步优化修复结果.在国际公认数据集上进行实验仿真，结果显示:本文提出的算法在修复大面积不规则缺失时，可以生成合理的结构和丰富的纹理细节，修复的视觉效果，PSNR，SSIM和L1损失优于对比算法.

关键词: 图像修复；U-net边缘生成；超图卷积；混合空洞卷积；两阶段网络

Abstract: In order to implement reasonable structure inpainting and fine texture reconstruction for large irregular missing areas with complex background， a two-stage network inpainting algorithm based on U-net edge generation and hypergraphs convolution is proposed. Firstly， the image to be repaired is fed into a coarse inpainting network based on U-net gated convolution where the context information of the image is propagated to a deeper layer through jump connection to obtain rich image detail information. Down-sampling is applied to extract the edge features of the missing area， and up-sampling is performed to restore the edge details of the missing area while hybrid dilated convolution is adopted to increase the information receptive field and further obtain image detailed texture information. Subsequently， the coarse inpainting results are inputted into the refine inpainting network with hypergraphs convolution to capture and learn the hypergraphs structure in the input image， and the cross-correlation matrix of spatial features is implemented to capture the spatial feature structure as well as to further improve the structural integrity and fine-grained details. Finally， the refined inpainting results are input into the discriminator for discrimination optimization to further improve the inpainting results. The experimental simulation is carried out on the internationally published dataset. The experimental results demonstrate that the proposed algorithm can generate a reasonable structure with good color consistency and abundant detail texture under the condition of large-area loss， and the visual effect， PSNR， SSIM and L1 loss are superior over those of the compared algorithms.

Key words: image inpainting; U-net edge generation; hypergraphs convolution; hybrid dilated convolution; two-stage network

中图分类号:

TP391

李海燕，熊立昌，郭磊，李海江. 基于U-net边缘生成和超图卷积的两阶段修复算法[J]. 东北大学学报（自然科学版）, 2023, 44(3): 331-339.

LI Hai-yan， XIONG Li-chang， GUO Lei， LI Hai-jiang. Two-Stage Inpainting Algorithm Based on U-net Edge Generation and Hypergraphs Convolution[J]. Journal of Northeastern University(Natural Science), 2023, 44(3): 331-339.

参考文献

[1]Bertalmio M，Sapiro G，Caselles V，et al.Image inpainting［C］//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM，2000:417-424.
[2]Ballester C，Bertalmio M，Caselles V，et al.Filling-in by joint interpolation of vector fields and gray levels［J］.IEEE Transactions on Image Procession，2001，10(8):1200-1211.
[3]Esedoglu S，Shen J.Digital inpainting based on the Mumford-Shah-Euler image model［J］.European Journal of Applied Mathematics，2002，13(4):353-370.
[4]Wang T，Qin Z，Zhu M.An ELU network with total variation for image denoising［C］//International Conference on Neural Information Processing.Cham:Springer， 2017:227-237.
[5]Barnes C，Shechtman E，Finkelstein A，et al.Patch match:a randomized correspondence algorithm for structural image editing［J］.ACM Transactions on Graphics(TOG)，2009，28(3):1-11.
[6]Pathak D，Krahenbuhl P，Donahue J，et al.Context encoders:feature learning by inpainting［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016:2536-2544.
[7]Iizuka S，Simo-Serra E，Ishikawa H.Globally and locally consistent image completion［J］.ACM Transactions on Graphics(TOG)，2017，36(4):107.1-107.14.
[8]Yang C，Lu X，Lin Z，et al.High-resolution image inpainting using multi-scale neural patch synthesis［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Puekto Rico，2017:6721-6729.
[9]Simonyan K，Zisserman A.Very deep convolutional networks for large-scale image recognition［C］//International Conference on Learning Representations.San Diego:ICLR，2015:1150-1210.
[10]Liu G，Reda F，Shih K，et al.Image inpainting for irregular holes using partial convolutions［C］//Proceedings of the European Conference on Computer Vision(ECCV).Puerto Rito，2018:85-100.
[11]Nazeri K，Ng E，Joseph T，et al.Edge-connect:generative image inpainting with adversarial edge learning［C］//ICCV Workshops.Seoul:ICCV，2019:5833-5841.
[12]Zheng C，Cham T J，Cai J.Pluralistic image completion［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach，2019:1438-1447.
[13]Peng J，Liu D，Xu S，et al.Generating diverse structure for image inpainting with hierarchical VQ-VAE［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Montreal，2021:10775-10784.
[14]Wadhwa G，Dhall A，Murala S，et al.Hyperrealistic image inpainting with hypergraphs［C］//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.New York，2021:3912-3921.
[15]Ronneberger O，Fischer P，Brox T.U-net:convolutional networks for biomedical image segmentation［C］//International Conference on Medical Image Computing and Computer-Assisted Intervention.Athens，2015:234-241.
[16]李海燕，吴自莹，郭磊，等.基于混合空洞卷积网络的多鉴别器图像修复［J］.华中科技大学学报(自然科学版)，2021，49(3):40-45.(Li Hai-yan，Wu Zi-ying，Guo Lei，et al.Multi-discriminator image inpainting algorithm based on hybird dilated convolution network［J］.Journal of Huazhong University of Science and Technology(Nature Science Edition)，2021，49(3):40-45.)
[17]Chen Y，Rohrbach M，Yan Z，et al.Graph-based global reasoning networks［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach，2019:433-442.
[18]Li Y，Gupta A.Beyond grids:learning graph representations for visual recognition［J］.Advances in Neural Information Processing Systems，2018，31:9225-9235.
[19]Liang X，Hu Z，Zhang H，et al.Symbolic graph reasoning meets convolutions［J］.Advances in Neural Information Processing Systems，2018，31:1853-1863.
[20]Yadati N，Nimishakavi M，Yadav P，et al.HyperGCN:a new method of training graph convolutional networks on hypergraphs［C］//Advances in Neural Information Processing Systems(NeurIPS).Puerto Rico，2019:1509-1520.
[21]Nah S，Kim T H，Lee K M.Deep multi-scale convolutional neural network for dynamic scene deblurring［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Puerto Rico，2017:3883-3891.
[22]Pandey R K，Saha N，Karmakar S，et al.MSCE:an edge-preserving robust loss function for improving super-resolution algorithms［C］//International Conference on Neural Information Processing. Cham:Springer，2018:566-575.(上接第322页)(Shao Hao，Wang Lun-wen，Deng Jian.Important node identification method for dynamic networks based on H operation［J］.Journal of Computer Applications，2019，39(9):2669-2674.)
[6]Aghaalizadeh S，Afshord S T，Bouyer A，et al.A three-stage algorithm for local community detection based on the high node importance ranking in social networks［J］.Physica A:Statistical Mechanics and Its Applications，2021，563:125420.
[7]尹梦梦，王磊，姚昌华，等.基于VIKOR模型的复杂网络节点重要度评估［J］.信息网络安全，2022，22(1):87-94.(Yin Meng-meng，Wang Lei，Yao Chang-hua，et al.Evaluation of the importance of complex network nodes based on VIKOR model［J］.Netinfo Security，2022，22(1):87-94.)
[8]李琳，刘雅奇.通信网节点重要性的多指标评价方法［J］.海军工程大学学报，2010，22(5):69-73.(Li Lin，Liu Ya-qi.Node importance evaluation in communication network with respect to multiple network performance measures［J］.Journal of Naval University of Engineering，2010，22(5):69-73.)
[9]Krackhardt D.Assessing the political landscape:structure，cognition，and power in organizations［J］.Administrative Science Quarterly，1990，35(2):342-369.
[10]Dolev S，Elovici Y ，Puzis R.Routing betweenness centrality［J］. Journal of the ACM，2010，57(4):1-27.
[11]Newman M E J，Girvan M.Finding and evaluating community structure in networks［J］.Physical Review E，2004，69(2):026113.
[12]晋建志.复杂网络基于节点重要性的社团探测及社团演化模型研究［D］.武汉:华中师范大学，2014.(Jin Jian-zhi.Research of community detection based on node importance and dynamic community evolution model in complex networks［D］.Wuhan:Central China Normal University，2014.)
[13]Huang Q.A novel important node discovery algorithm based on local community aggregation and recognition in complex networks［J］.International Journal of Wireless Information Networks，2020，27(2):253-260.
[14]Nacher J C，Akutsu T.Analysis on critical nodes in controlling complex networks using dominating sets［C］//2013 International Conference on Signal-Image Technology & Internet-Based Systems.Kyoto，2013:649-654.
[15]Wang J O，Mu Q Z，Meng F Q.Research situation analysis on identifying key nodes in complex networks based on knowledge graph［C］//12th International Conference on Computational Intelligence and Communication Networks(CICN).Bhimtal，2020:198-204.
[16]王胜银，翁怀荣，谢上明.天基综合信息网安全问题及解决方法浅析［J］.信息安全与通信保密，2009(8):172-174.(Wang Sheng-yin，Weng Huai-rong，Xie Shang-ming.Analysis of security problems and suggestions of space-based synthesis information network［J］.Information Security and Communications Privacy，2009(8):172-174.)
[17]Boudries F，Sadouki S ，Tari A.A bio-inspired algorithm for dynamic reconfiguration with end-to-end constraints in web services composition［J］.Service Oriented Computing and Applications，2019，13(3):251-260.
[18]Luo L J，Ren H P.Node importance evaluation of complex network based on M-TOPSIS method［J］.Journal of Physics:Conference Series，2019，1325(1):012016.