Two-Stage Inpainting Algorithm Based on U-net Edge Generation and Hypergraphs Convolution

doi:10.12068/j.issn.1005-3026.2023.03.004

Abstract

Abstract: In order to implement reasonable structure inpainting and fine texture reconstruction for large irregular missing areas with complex background， a two-stage network inpainting algorithm based on U-net edge generation and hypergraphs convolution is proposed. Firstly， the image to be repaired is fed into a coarse inpainting network based on U-net gated convolution where the context information of the image is propagated to a deeper layer through jump connection to obtain rich image detail information. Down-sampling is applied to extract the edge features of the missing area， and up-sampling is performed to restore the edge details of the missing area while hybrid dilated convolution is adopted to increase the information receptive field and further obtain image detailed texture information. Subsequently， the coarse inpainting results are inputted into the refine inpainting network with hypergraphs convolution to capture and learn the hypergraphs structure in the input image， and the cross-correlation matrix of spatial features is implemented to capture the spatial feature structure as well as to further improve the structural integrity and fine-grained details. Finally， the refined inpainting results are input into the discriminator for discrimination optimization to further improve the inpainting results. The experimental simulation is carried out on the internationally published dataset. The experimental results demonstrate that the proposed algorithm can generate a reasonable structure with good color consistency and abundant detail texture under the condition of large-area loss， and the visual effect， PSNR， SSIM and L1 loss are superior over those of the compared algorithms.

Key words: image inpainting; U-net edge generation; hypergraphs convolution; hybrid dilated convolution; two-stage network

CLC Number:

TP391

LI Hai-yan， XIONG Li-chang， GUO Lei， LI Hai-jiang. Two-Stage Inpainting Algorithm Based on U-net Edge Generation and Hypergraphs Convolution[J]. Journal of Northeastern University(Natural Science), 2023, 44(3): 331-339.

References

[1]Bertalmio M，Sapiro G，Caselles V，et al.Image inpainting［C］//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.New York:ACM，2000:417-424.
[2]Ballester C，Bertalmio M，Caselles V，et al.Filling-in by joint interpolation of vector fields and gray levels［J］.IEEE Transactions on Image Procession，2001，10(8):1200-1211.
[3]Esedoglu S，Shen J.Digital inpainting based on the Mumford-Shah-Euler image model［J］.European Journal of Applied Mathematics，2002，13(4):353-370.
[4]Wang T，Qin Z，Zhu M.An ELU network with total variation for image denoising［C］//International Conference on Neural Information Processing.Cham:Springer， 2017:227-237.
[5]Barnes C，Shechtman E，Finkelstein A，et al.Patch match:a randomized correspondence algorithm for structural image editing［J］.ACM Transactions on Graphics(TOG)，2009，28(3):1-11.
[6]Pathak D，Krahenbuhl P，Donahue J，et al.Context encoders:feature learning by inpainting［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016:2536-2544.
[7]Iizuka S，Simo-Serra E，Ishikawa H.Globally and locally consistent image completion［J］.ACM Transactions on Graphics(TOG)，2017，36(4):107.1-107.14.
[8]Yang C，Lu X，Lin Z，et al.High-resolution image inpainting using multi-scale neural patch synthesis［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Puekto Rico，2017:6721-6729.
[9]Simonyan K，Zisserman A.Very deep convolutional networks for large-scale image recognition［C］//International Conference on Learning Representations.San Diego:ICLR，2015:1150-1210.
[10]Liu G，Reda F，Shih K，et al.Image inpainting for irregular holes using partial convolutions［C］//Proceedings of the European Conference on Computer Vision(ECCV).Puerto Rito，2018:85-100.
[11]Nazeri K，Ng E，Joseph T，et al.Edge-connect:generative image inpainting with adversarial edge learning［C］//ICCV Workshops.Seoul:ICCV，2019:5833-5841.
[12]Zheng C，Cham T J，Cai J.Pluralistic image completion［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach，2019:1438-1447.
[13]Peng J，Liu D，Xu S，et al.Generating diverse structure for image inpainting with hierarchical VQ-VAE［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Montreal，2021:10775-10784.
[14]Wadhwa G，Dhall A，Murala S，et al.Hyperrealistic image inpainting with hypergraphs［C］//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.New York，2021:3912-3921.
[15]Ronneberger O，Fischer P，Brox T.U-net:convolutional networks for biomedical image segmentation［C］//International Conference on Medical Image Computing and Computer-Assisted Intervention.Athens，2015:234-241.
[16]李海燕，吴自莹，郭磊，等.基于混合空洞卷积网络的多鉴别器图像修复［J］.华中科技大学学报(自然科学版)，2021，49(3):40-45.(Li Hai-yan，Wu Zi-ying，Guo Lei，et al.Multi-discriminator image inpainting algorithm based on hybird dilated convolution network［J］.Journal of Huazhong University of Science and Technology(Nature Science Edition)，2021，49(3):40-45.)
[17]Chen Y，Rohrbach M，Yan Z，et al.Graph-based global reasoning networks［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach，2019:433-442.
[18]Li Y，Gupta A.Beyond grids:learning graph representations for visual recognition［J］.Advances in Neural Information Processing Systems，2018，31:9225-9235.
[19]Liang X，Hu Z，Zhang H，et al.Symbolic graph reasoning meets convolutions［J］.Advances in Neural Information Processing Systems，2018，31:1853-1863.
[20]Yadati N，Nimishakavi M，Yadav P，et al.HyperGCN:a new method of training graph convolutional networks on hypergraphs［C］//Advances in Neural Information Processing Systems(NeurIPS).Puerto Rico，2019:1509-1520.
[21]Nah S，Kim T H，Lee K M.Deep multi-scale convolutional neural network for dynamic scene deblurring［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Puerto Rico，2017:3883-3891.
[22]Pandey R K，Saha N，Karmakar S，et al.MSCE:an edge-preserving robust loss function for improving super-resolution algorithms［C］//International Conference on Neural Information Processing. Cham:Springer，2018:566-575.(上接第322页)(Shao Hao，Wang Lun-wen，Deng Jian.Important node identification method for dynamic networks based on H operation［J］.Journal of Computer Applications，2019，39(9):2669-2674.)
[6]Aghaalizadeh S，Afshord S T，Bouyer A，et al.A three-stage algorithm for local community detection based on the high node importance ranking in social networks［J］.Physica A:Statistical Mechanics and Its Applications，2021，563:125420.
[7]尹梦梦，王磊，姚昌华，等.基于VIKOR模型的复杂网络节点重要度评估［J］.信息网络安全，2022，22(1):87-94.(Yin Meng-meng，Wang Lei，Yao Chang-hua，et al.Evaluation of the importance of complex network nodes based on VIKOR model［J］.Netinfo Security，2022，22(1):87-94.)
[8]李琳，刘雅奇.通信网节点重要性的多指标评价方法［J］.海军工程大学学报，2010，22(5):69-73.(Li Lin，Liu Ya-qi.Node importance evaluation in communication network with respect to multiple network performance measures［J］.Journal of Naval University of Engineering，2010，22(5):69-73.)
[9]Krackhardt D.Assessing the political landscape:structure，cognition，and power in organizations［J］.Administrative Science Quarterly，1990，35(2):342-369.
[10]Dolev S，Elovici Y ，Puzis R.Routing betweenness centrality［J］. Journal of the ACM，2010，57(4):1-27.
[11]Newman M E J，Girvan M.Finding and evaluating community structure in networks［J］.Physical Review E，2004，69(2):026113.
[12]晋建志.复杂网络基于节点重要性的社团探测及社团演化模型研究［D］.武汉:华中师范大学，2014.(Jin Jian-zhi.Research of community detection based on node importance and dynamic community evolution model in complex networks［D］.Wuhan:Central China Normal University，2014.)
[13]Huang Q.A novel important node discovery algorithm based on local community aggregation and recognition in complex networks［J］.International Journal of Wireless Information Networks，2020，27(2):253-260.
[14]Nacher J C，Akutsu T.Analysis on critical nodes in controlling complex networks using dominating sets［C］//2013 International Conference on Signal-Image Technology & Internet-Based Systems.Kyoto，2013:649-654.
[15]Wang J O，Mu Q Z，Meng F Q.Research situation analysis on identifying key nodes in complex networks based on knowledge graph［C］//12th International Conference on Computational Intelligence and Communication Networks(CICN).Bhimtal，2020:198-204.
[16]王胜银，翁怀荣，谢上明.天基综合信息网安全问题及解决方法浅析［J］.信息安全与通信保密，2009(8):172-174.(Wang Sheng-yin，Weng Huai-rong，Xie Shang-ming.Analysis of security problems and suggestions of space-based synthesis information network［J］.Information Security and Communications Privacy，2009(8):172-174.)
[17]Boudries F，Sadouki S ，Tari A.A bio-inspired algorithm for dynamic reconfiguration with end-to-end constraints in web services composition［J］.Service Oriented Computing and Applications，2019，13(3):251-260.
[18]Luo L J，Ren H P.Node importance evaluation of complex network based on M-TOPSIS method［J］.Journal of Physics:Conference Series，2019，1325(1):012016.