A Real-Time Video Image Background Replacement Method Based on Deep Learning

doi:10.12068/j.issn.1005-3026.2021.11.004

Abstract

Abstract: Aiming at the real-time requirement of video session service， a lightweight deep learning network model is proposed to realize the real-time background replacement function of video images. The network model includes two modules: semantic segmentation and background replacement. The whole architecture of semantic segmentation module adopts encode-decoder structure. Encoder module， dilated convolution pyramid pooling module， attention module， and gain module are used in the encoding terminal to extract features. Decoder module， adjustment module， and encoder module are used in the decoding terminal to recover the image， and the background replacement module is used to complete the background replacement. After the data-set training， the segmentation accuracy of the network model reaches 94.1%， and the segmentation speed reaches 42.5 frames/s， which achieves a good balance between real-time and accuracy， and has a good practical effect.

Key words: real-time video image; background replacement; deep learning; semantic segmentation; encode-decode structure

CLC Number:

TP20

XIE Tian-zhi， LEI Wei-min， ZHANG Wei， LI Zhi-yuan. A Real-Time Video Image Background Replacement Method Based on Deep Learning[J]. Journal of Northeastern University(Natural Science), 2021, 42(11): 1540-1546.

References

[1]Zhu D.The application of computer image processing technology in the fine arts creation［C］//2014 IEEE Workshop on Advanced Research and Technology in Industry Applications(WARTIA).Ottawa:IEEE，2014:790-792.
[2]韩思奇，王蕾.图像分割的阈值法综述［J］.系统工程与电子技术，2002，24(6):91-94.(Han Si-qi，Wang Lei.A survey of thresholding methods for image segmentation［J］.Systems Engineering and Electronics，2002，24(6):91-94.)
[3]Zhu S，Xia X，Zhang Q，et al.An image segmentation algorithm in image processing based on threshold segmentation［C］//2007 Third International IEEE Conference on Signal-Image Technologies and Internet-Based System.Shanghai:IEEE，2007:673-678.
[4]任洪娥，刘冕，董本志.基于改进形态学算子的木粉边缘检测算法［J］.计算机工程与应用，2015，51(5):183-186.(Ren Hong-e，Liu Mian，Dong Ben-zhi.Edge detection algorithm of wood-flour based on modified morphological operator［J］.Computer Engineering and Applications， 2015，51(5):183-186.)
[5]Zhang Y，Qin L，Yao H，et al.Beyond particle flow:bag of trajectory graphs for dense crowd event recognition［C］//2013 IEEE International Conference on Image Processing.Melbourne:IEEE，2013:3572-3576.
[6]Csurka G，Perronnin F.An efficient approach to semantic segmentation［J］.International Journal of Computer Vision， 2011，95(2):198-212.
[7]He Y，Wang H，Zhang B.Color-based road detection in urban traffic scenes［J］.IEEE Transactions on Intelligent Transportation Systems， 2005，5(4):309-318.
[8]安喆，徐熙平，杨进华，等.结合图像语义分割的增强现实型平视显示系统设计与研究［J］.光学学报，2018，38(7):85-91.(An Zhe，Xu Xi-ping，Yang Jin-hua，et al.Design of augmented reality head-up display system based on image semantic segmentation［J］.Acta Optica Sinica，2018，38(7):85-91.)
[9]Long J，Shelhamer E，Darrell T.Fully convolutional networks for semantic segmentation［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston，2015:3431-3440.
[10]Ronneberger O，Fischer P，Brox T.U-net:convolutional networks for biomedical image segmentation［C］//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham:Springer，2015:234-241.
[11]Chen L C，Papandreou G，Kokkinos I，et al.DeepLab:semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFs［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018，40(4):834-848.
[12]Chen L C，Papandreou G，Schroff F，et al.Rethinking atrous convolution for semantic image segmentation［J］.arXiv Preprint arXiv:1706.05587，2017.
[13]Caelles S，Maninis K K，Pont-Tuset J，et al.One-shot video object segmentation［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii，2017:221-230.
[14]Oh S W，Lee J Y，Xu N，et al.Video object segmentation using space-time memory networks［C］//Proceedings of the IEEE/CVF International Conference on Computer Vision.Long Beach，2019:9226-9235.
[15]Chen X，Li Z，Yuan Y，et al.State-aware tracker for real-time video object segmentation［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle，2020:9384-9393.
[16]Yu C，Wang J，Peng C，et al.BiseNet:bilateral segmentation network for real-time semantic segmentation［C］// Proceedings of the European Conference on Computer Vision.Munich，2018:325-341.
[17]Howard A G，Zhu M，Chen B，et al.MobileNets:efficient convolutional neural networks for mobile vision applications［J］.arXiv Preprint arXiv:1704.04861，2017.
[18]Sandler M，Howard A，Zhu M，et al.MobileNetv2:inverted residuals and linear bottlenecks［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City，2018:4510-4520.
[19]Pinheiro P O，Lin T Y，Collobert R，et al.Learning to refine object segments［C］//European Conference on Computer Vision.Cham:Springer，2016:75-91.
[20]Rhemann C，Rother C，Wang J，et al.A perceptually motivated online benchmark for image matting［C］//2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami:IEEE，2009:1826-1833.
[21]Supervisely.Supervisely person dataset［DB/OL］.(2018-04-12)［2020-11-04］.http://github.com/superwisely/supervisely.
[22]Aisegmen.com.Aisegmention［DB/OL］.(2019-05-30)［2020-11-04］.https://github.com/aisegmentcn/matting_human_datasets.
[23]Xu N，Price B，Cohen S，et al.Deep image matting［C］// IEEE Conference on Computer Vision & Pattern Recognition.Hawaii:IEEE Computer Society，2017:311-320.
[24]He K M，Zhang X Y，Ren S Q，et al.Deep residual learning for image recognition［C］ //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society，2016:770-778.