基于双流多关系GCNs的骨架动作识别方法

doi:10.12068/j.issn.1005-3026.2021.06.002

东北大学学报（自然科学版） ›› 2021, Vol. 42 ›› Issue (6): 768-774.DOI: 10.12068/j.issn.1005-3026.2021.06.002

基于双流多关系GCNs的骨架动作识别方法

刘芳^1,2，乔建忠¹，代钦³，石祥滨²

(1. 东北大学计算机科学与工程学院，辽宁沈阳110169； 2. 沈阳航空航天大学计算机学院，辽宁沈阳110136；3. 沈阳工程学院信息学院，辽宁沈阳110136)

修回日期:2020-09-18 接受日期:2020-09-18 发布日期:2021-06-23
通讯作者: 刘芳
作者简介:刘芳(1981-)，女，辽宁鞍山人，东北大学博士研究生，沈阳航空航天大学讲师；乔建忠(1964-)，男，辽宁葫芦岛人，东北大学教授，博士生导师；石祥滨(1963-)，男，辽宁大连人，沈阳航空航天大学教授.
基金资助:
辽宁省自然科学基金资助项目(2019-MS-254，20180550337)；辽宁省教育厅科研项目(JYT2020050，JL-2019)；中央引导地方科技发展专项资金资助项目(2021JH6/10500127).

Skeleton-based Action Recognition Method with Two-Stream Multi-relational GCNs

LIU Fang^1,2， QIAO Jian-zhong¹， DAI Qin³， SHI Xiang-bin²

1. School of Computer Science & Engineering， Northeastern University， Shenyang 110169， China; 2. School of Computer Science， Shenyang Aerospace University， Shenyang 110136， China; 3. College of Information， Shenyang Institute of Engineering， Shenyang 110136， China.

Revised:2020-09-18 Accepted:2020-09-18 Published:2021-06-23
Contact: QIAO Jian-zhong
About author:-
Supported by:
-

摘要/Abstract

摘要： 基于图卷积网络(graph convolutional networks，GCNs)的骨架动作识别方法只能对关节点间的单一关系进行建模，缺少描述多种关系的能力.借鉴知识图谱描述实体之间不同关系的思想，提出一种基于关节点流和肢体流的双流多关系GCNs人体骨架动作识别方法，对图结点间的自然连接关系、对称关系和全局关系进行建模，各种特征在网络中同步传输并有效融合.运动的全身协作过程中，每个部位的交互范围有限且依赖于具体动作，提出基于Non-local机制的topK全局邻接关系自适应计算方法，为每个结点动态选择交互强度较大的前K个结点作为全局关系邻接点.实验结果表明，所提出的双流多关系网络在Kinetics和NTU-RGB+D数据集上取得了较好的动作识别效果.

关键词: 动作识别；骨架；图卷积网络；多关系；topK

Abstract: The interaction of human body parts in motion is diverse， but the existing skeleton-based action recognition methods with GCNs(graph convolutional networks) can only model a single relationship between joints. The idea of knowledge graphs was used to describe the different relationships between entities and a two-stream multi-relational GCNs action recognition method was proposed based on joints and body parts， by which the models of the natural connection relationship， symmetric relationship， and global relationship among nodes were established. The features of each relationship were synchronously transmitted and effectively fused in the network. In the process of global cooperation of human body parts， the interaction range of each part was limited and depends on specific actions. An adaptive topK global adjacency relationship calculation method was proposed based on Non-local algorithm， in which the nodes with the topK interaction strength were selected dynamically as the adjacent nodes for each node. The experimental results show that the proposed two-stream multi-relational network achieves good action accuracy on the Kinetics dataset and the NTU-RGB+D dataset.

Key words: action recognition； skeleton； GCNs(graph convolutional networks)； multi-relation； topK

中图分类号:

TP18

刘芳，乔建忠，代钦，石祥滨. 基于双流多关系GCNs的骨架动作识别方法[J]. 东北大学学报（自然科学版）, 2021, 42(6): 768-774.

LIU Fang， QIAO Jian-zhong， DAI Qin， SHI Xiang-bin. Skeleton-based Action Recognition Method with Two-Stream Multi-relational GCNs[J]. Journal of Northeastern University(Natural Science), 2021, 42(6): 768-774.

参考文献

[1]Wang L，Koniusz P，Huynh D.Hallucinating IDT descriptors and I3D optical flow features for action recognition with CNNs［C］//2019 IEEE International Conference on Computer Vision.Seoul，2019: 8697-8707.
[2]Yan S，Xiong Y，Lin D.Spatial temporal graph convolutional networks for skeleton-based action recognition［C］//Thirty-Second AAAI Conference on Artificial Intelligence.New Orleans，2018:7444-7452.
[3]Shahroudy A，Liu J，Ng T，et al.NTU RGB+D: a large scale dataset for 3D human activity analysis［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016:1010-1019.
[4]Ke Q，Bennamoun M，An S，et al.A new representation of skeleton sequences for 3D action recognition［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu，2017: 4570-4579.
[5]Shi L，Zhang Y，Cheng J，et al.Two-stream adaptive graph convolutional networks for skeleton-based action recognition［C］//2019 IEEE Conference on Computer Vision and Pattern Recognition.Long Beach，2019:12026-12035.
[6]Hu G，Cui B，Yu S.Skeleton-based action recognition with synchronous local and non-local spatio-temporal learning and frequency attention［C］//2019 IEEE International Conference on Multimedia and Expo.Shanghai，2019:1216-1221.
[7]Zhang P，Lan C，Zeng W，et al.Semantics-guided neural networks for efficient skeleton-based human action recognition［C］//2020 IEEE Conference on Computer Vision and Pattern Recognition.Seattle，2020: 1109-1118.
[8]Gao J，Zhang T，Xu C.I know the relationships: zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs［C］//The Thirty-Third AAAI Conference on Artificial Intelligence.Honolulu，2019: 8303-8311.
[9]Schlichtkrull M S，Kipf T N，Bloem P，et al.Modeling relational data with graph convolutional networks［C］//The Semantic Web 15th International Conference.Heraklion，2018: 593-607.
[10]Ye R，Li X，Fang Y，et al.A vectorized relational graph convolutional network for multi-relational network alignment［C］//28th International Joint Conference on Artificial Intelligence.Macao，2019: 4135-4141.
[11]Vashishth S，Sanyal S，Nitin V，et al.Composition based multi-relational graph convolutional networks［C/OL］//8th International Conference on Learning Representations.［2020-12-21］.https://arxiv.org/pdf/1911.03082.pdf.
[12]Wang X，Girshick R B，Gupta A，et al.Non-local neural networks［C］//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Computer Society，2018: 7794-7803.
[13]Kay W，Carreira J，Simonyan K，et al.The Kinetics human action video dataset［J/OL］.Computing Research Repository.［2020-12-21］.https://arxiv.org/pdf/1705.06950.pdf.
[14]Li S，Li W，Cook C，et al.Independently recurrent neural network (INDRNN):building a longer and deeper RNN［C］//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Computer Society，2018: 5457-5466.
[15]Kim T S，Reiter A.Interpretable 3D human action analysis with temporal convolutional networks［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops.Honolulu，2017: 1623-1631.
[16]Li C，Zhong Q，Xie D，et al.Skeleton-based action recognition with convolutional neural networks［C］//2017 IEEE International Conference on Multimedia and Expo Workshops.Hong Kong，2017: 597-600.
[17]Li B，Dai Y，Cheng X，et al.Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN ［C］//2017 IEEE International Conference on Multimedia and Expo Workshops.Hong Kong，2017: 601-604.
[18]Tang Y，Tian Y，Lu J，et al.Deepprogressive reinforcement learning for skeleton-based action recognition［C］//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Computer Society，2018: 18-22.

基于双流多关系GCNs的骨架动作识别方法

Skeleton-based Action Recognition Method with Two-Stream Multi-relational GCNs

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李婵，曲璐渲，信俊昌，王之琼. 基于潜在调控因子筛选的高阶动态贝叶斯建模方法[J]. 东北大学学报（自然科学版）, 2023, 44(3): 323-330.
[2]	张之敏，乔建忠，林树宽，王品贺. 一种基于深度网络的视图重建方法[J]. 东北大学学报:自然科学版, 2020, 41(8): 1065-1069.
[3]	赵建喆，吴辰铌，王兴伟，裴丽亚. 基于马尔可夫毯的贝叶斯网络结构学习算法[J]. 东北大学学报:自然科学版, 2020, 41(4): 464-470.
[4]	崔雪婷，李颖，范嘉豪. 全局混沌蝙蝠优化算法[J]. 东北大学学报:自然科学版, 2020, 41(4): 488-492.
[5]	曲璐渲，郭上慧，王之琼，信俊昌. 基因调控网络的父节点筛选贝叶斯建模方法[J]. 东北大学学报:自然科学版, 2020, 41(2): 158-162.
[6]	孔芝，李事成，赵杰. 珊瑚礁算法的改进研究[J]. 东北大学学报:自然科学版, 2020, 41(2): 163-169.
[7]	翟莹莹，左丽，张恩德. 基于参数优化的RBF神经网络结构设计算法[J]. 东北大学学报:自然科学版, 2020, 41(2): 176-182.
[8]	张华伟，郑晓涛. 基于遗传算法优化神经网络的拼焊板压边力预测[J]. 东北大学学报:自然科学版, 2020, 41(2): 241-245.
[9]	于长永，何鑫，祁欣，马海涛. GMSDenseNet:基于组多结构卷积的轻量级DenseNet[J]. 东北大学学报:自然科学版, 2020, 41(1): 12-16.
[10]	李占山，刘兆赓，俞寅，鄢文浩. 量子化信息素蚁群优化特征选择算法[J]. 东北大学学报:自然科学版, 2020, 41(1): 17-22.
[11]	李壮，刘磊，张桐搏，吕帅. 基于重启策略的学习子句优化方法[J]. 东北大学学报:自然科学版, 2020, 41(1): 44-48.
[12]	徐林，郑晓彤，付博，田歌. 基于改进GAN算法的电机轴承故障诊断方法[J]. 东北大学学报:自然科学版, 2019, 40(12): 1679-1684.
[13]	马铁民，周福才，王爽. 基于用户相似度的随机游走社交网络事件推荐算法[J]. 东北大学学报:自然科学版, 2019, 40(11): 1533-1538.
[14]	蒋芳芳，徐敬傲，李任，徐礼胜. 基于CNN的心冲击信号阵发性房颤自动检测方法[J]. 东北大学学报:自然科学版, 2019, 40(11): 1539-1543.
[15]	刘鹏，杜佳芝，吕伟刚，窦明武. 面向不平衡数据集的一种改进的k-近邻分类器[J]. 东北大学学报:自然科学版, 2019, 40(7): 932-936.