一种基于深度学习的实时视频图像背景替换方法

doi:10.12068/j.issn.1005-3026.2021.11.004

东北大学学报（自然科学版） ›› 2021, Vol. 42 ›› Issue (11): 1540-1546.DOI: 10.12068/j.issn.1005-3026.2021.11.004

一种基于深度学习的实时视频图像背景替换方法

谢天植，雷为民，张伟，李志远

(东北大学计算机科学与工程学院，辽宁沈阳110169)

修回日期:2021-01-27 接受日期:2021-01-27 发布日期:2021-11-19
通讯作者: 谢天植
作者简介:谢天植(1996－)，男，河北石家庄人，东北大学硕士研究生; 雷为民(1969－)，男，山西平遥人，东北大学教授，博士生导师.
基金资助:
国家重点研发计划项目(2018YFB1702000); 中央高校基本科研业务费专项资金资助项目(N2016014).

A Real-Time Video Image Background Replacement Method Based on Deep Learning

XIE Tian-zhi， LEI Wei-min， ZHANG Wei， LI Zhi-yuan

School of Computer Science & Engineering， Northeastern University， Shenyang 110169， China.

Revised:2021-01-27 Accepted:2021-01-27 Published:2021-11-19
Contact: ZHANG Wei
About author:-
Supported by:
-

摘要/Abstract

摘要： 针对视频会话业务的实时性需求，提出一种轻量级深度学习网络模型实现视频图像的实时背景替换功能.网络模型包含语义分割和背景替换两个模块.语义分割模块整体架构采用编解码结构，编码端使用编码器模块、空洞卷积金字塔池化模块、注意力模块以及增益模块提取特征；解码端使用解码器模块、调整模块以及编码器模块恢复图像，再传入背景替换模块完成背景替换.该网络模型在本文设定的数据集训练后分割精确度达到94.1%，分割速度达到42.5帧/s，在实时性和准确性上达到较好的平衡，具有很好的实用效果.

关键词: 实时视频图像;背景替换;深度学习;语义分割;编解码结构

Abstract: Aiming at the real-time requirement of video session service， a lightweight deep learning network model is proposed to realize the real-time background replacement function of video images. The network model includes two modules: semantic segmentation and background replacement. The whole architecture of semantic segmentation module adopts encode-decoder structure. Encoder module， dilated convolution pyramid pooling module， attention module， and gain module are used in the encoding terminal to extract features. Decoder module， adjustment module， and encoder module are used in the decoding terminal to recover the image， and the background replacement module is used to complete the background replacement. After the data-set training， the segmentation accuracy of the network model reaches 94.1%， and the segmentation speed reaches 42.5 frames/s， which achieves a good balance between real-time and accuracy， and has a good practical effect.

Key words: real-time video image; background replacement; deep learning; semantic segmentation; encode-decode structure

中图分类号:

TP20

谢天植，雷为民，张伟，李志远. 一种基于深度学习的实时视频图像背景替换方法[J]. 东北大学学报（自然科学版）, 2021, 42(11): 1540-1546.

XIE Tian-zhi， LEI Wei-min， ZHANG Wei， LI Zhi-yuan. A Real-Time Video Image Background Replacement Method Based on Deep Learning[J]. Journal of Northeastern University(Natural Science), 2021, 42(11): 1540-1546.

参考文献

[1]Zhu D.The application of computer image processing technology in the fine arts creation［C］//2014 IEEE Workshop on Advanced Research and Technology in Industry Applications(WARTIA).Ottawa:IEEE，2014:790-792.
[2]韩思奇，王蕾.图像分割的阈值法综述［J］.系统工程与电子技术，2002，24(6):91-94.(Han Si-qi，Wang Lei.A survey of thresholding methods for image segmentation［J］.Systems Engineering and Electronics，2002，24(6):91-94.)
[3]Zhu S，Xia X，Zhang Q，et al.An image segmentation algorithm in image processing based on threshold segmentation［C］//2007 Third International IEEE Conference on Signal-Image Technologies and Internet-Based System.Shanghai:IEEE，2007:673-678.
[4]任洪娥，刘冕，董本志.基于改进形态学算子的木粉边缘检测算法［J］.计算机工程与应用，2015，51(5):183-186.(Ren Hong-e，Liu Mian，Dong Ben-zhi.Edge detection algorithm of wood-flour based on modified morphological operator［J］.Computer Engineering and Applications， 2015，51(5):183-186.)
[5]Zhang Y，Qin L，Yao H，et al.Beyond particle flow:bag of trajectory graphs for dense crowd event recognition［C］//2013 IEEE International Conference on Image Processing.Melbourne:IEEE，2013:3572-3576.
[6]Csurka G，Perronnin F.An efficient approach to semantic segmentation［J］.International Journal of Computer Vision， 2011，95(2):198-212.
[7]He Y，Wang H，Zhang B.Color-based road detection in urban traffic scenes［J］.IEEE Transactions on Intelligent Transportation Systems， 2005，5(4):309-318.
[8]安喆，徐熙平，杨进华，等.结合图像语义分割的增强现实型平视显示系统设计与研究［J］.光学学报，2018，38(7):85-91.(An Zhe，Xu Xi-ping，Yang Jin-hua，et al.Design of augmented reality head-up display system based on image semantic segmentation［J］.Acta Optica Sinica，2018，38(7):85-91.)
[9]Long J，Shelhamer E，Darrell T.Fully convolutional networks for semantic segmentation［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston，2015:3431-3440.
[10]Ronneberger O，Fischer P，Brox T.U-net:convolutional networks for biomedical image segmentation［C］//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham:Springer，2015:234-241.
[11]Chen L C，Papandreou G，Kokkinos I，et al.DeepLab:semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFs［J］.IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018，40(4):834-848.
[12]Chen L C，Papandreou G，Schroff F，et al.Rethinking atrous convolution for semantic image segmentation［J］.arXiv Preprint arXiv:1706.05587，2017.
[13]Caelles S，Maninis K K，Pont-Tuset J，et al.One-shot video object segmentation［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Hawaii，2017:221-230.
[14]Oh S W，Lee J Y，Xu N，et al.Video object segmentation using space-time memory networks［C］//Proceedings of the IEEE/CVF International Conference on Computer Vision.Long Beach，2019:9226-9235.
[15]Chen X，Li Z，Yuan Y，et al.State-aware tracker for real-time video object segmentation［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle，2020:9384-9393.
[16]Yu C，Wang J，Peng C，et al.BiseNet:bilateral segmentation network for real-time semantic segmentation［C］// Proceedings of the European Conference on Computer Vision.Munich，2018:325-341.
[17]Howard A G，Zhu M，Chen B，et al.MobileNets:efficient convolutional neural networks for mobile vision applications［J］.arXiv Preprint arXiv:1704.04861，2017.
[18]Sandler M，Howard A，Zhu M，et al.MobileNetv2:inverted residuals and linear bottlenecks［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City，2018:4510-4520.
[19]Pinheiro P O，Lin T Y，Collobert R，et al.Learning to refine object segments［C］//European Conference on Computer Vision.Cham:Springer，2016:75-91.
[20]Rhemann C，Rother C，Wang J，et al.A perceptually motivated online benchmark for image matting［C］//2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami:IEEE，2009:1826-1833.
[21]Supervisely.Supervisely person dataset［DB/OL］.(2018-04-12)［2020-11-04］.http://github.com/superwisely/supervisely.
[22]Aisegmen.com.Aisegmention［DB/OL］.(2019-05-30)［2020-11-04］.https://github.com/aisegmentcn/matting_human_datasets.
[23]Xu N，Price B，Cohen S，et al.Deep image matting［C］// IEEE Conference on Computer Vision & Pattern Recognition.Hawaii:IEEE Computer Society，2017:311-320.
[24]He K M，Zhang X Y，Ren S Q，et al.Deep residual learning for image recognition［C］ //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society，2016:770-778.

一种基于深度学习的实时视频图像背景替换方法

A Real-Time Video Image Background Replacement Method Based on Deep Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	耿蓉，吴亚倩，肖倩倩，徐赛. 基于改进GRU算法的天基信息网资源预测研究[J]. 东北大学学报（自然科学版）, 2023, 44(3): 305-314.
[2]	罗忠，吴东泽，李雷，葛长闯. 转子系统动力学仿真平台设计与实验验证[J]. 东北大学学报（自然科学版）, 2023, 44(3): 375-381.
[3]	侯振隆，郑玉君，巩恩普，程浩. 基于深度加权的重力梯度数据联合相关成像反演[J]. 东北大学学报:自然科学版, 2020, 41(11): 1628-1632.
[4]	吴超，李元辉，徐帅，戴星航. 地下工程岩体节理迹长统计下限值的确定方法[J]. 东北大学学报:自然科学版, 2020, 41(8): 1167-1173.
[5]	温守钦，唐铁乔，谢伟，付建飞. 氧、硫逸度对岫岩红旗铅锌矿床矿物组合共生分异的制约[J]. 东北大学学报:自然科学版, 2020, 41(7): 999-1007.
[6]	齐锡晶，唐梁，康伟鑫，秦娇娇. 高速公路桥梁桥面板养护方案多目标决策方法[J]. 东北大学学报:自然科学版, 2020, 41(7): 1033-1040.
[7]	蔡志辉，张德良，周彦君，文光奇. 应变速率对Fe-11Mn-2Al-0.2C中锰钢变形行为的影响[J]. 东北大学学报:自然科学版, 2020, 41(7): 1041-1047.
[8]	马树军，王霄霄. 吸附微粒对微悬空桥传感器的影响[J]. 东北大学学报:自然科学版, 2020, 41(1): 108-112.
[9]	马树军，玄航，孙嘉蔚，杨磊. 在非黏性流体域中微悬臂梁振动的建模与仿真[J]. 东北大学学报:自然科学版, 2018, 39(9): 1272-1276.
[10]	马树军，修强. 考虑表面应力的微悬臂梁质量传感器的建模与仿真[J]. 东北大学学报:自然科学版, 2018, 39(2): 237-241.
[11]	陈大力，沈岩涛，谢槟竹，马颖异. 基于余弦相似度模型的最佳教练遴选算法[J]. 东北大学学报:自然科学版, 2014, 35(12): 1697-1700.
[12]	潘峰，刘禄，薛定宇. 最优分数阶PIλDμ网络时延控制器[J]. 东北大学学报:自然科学版, 2014, 35(10): 1382-1385.
[13]	刘禄，潘峰，薛定宇. 一种分数阶卡尔曼滤波器[J]. 东北大学学报:自然科学版, 2014, 35(8): 1069-1072.
[14]	刘金海，冯健，马大中. 流体管道压力信号的高精度实时滤波方法[J]. 东北大学学报:自然科学版, 2013, 34(1): 9-12.
[15]	孙艳蕊. 带时间和成本约束的随机流网络可靠度的计算[J]. 东北大学学报（自然科学版）, 2013, 34(11): 1537-1541.