Skeleton-based Action Recognition Method with Two-Stream Multi-relational GCNs

doi:10.12068/j.issn.1005-3026.2021.06.002

Abstract

Abstract: The interaction of human body parts in motion is diverse， but the existing skeleton-based action recognition methods with GCNs(graph convolutional networks) can only model a single relationship between joints. The idea of knowledge graphs was used to describe the different relationships between entities and a two-stream multi-relational GCNs action recognition method was proposed based on joints and body parts， by which the models of the natural connection relationship， symmetric relationship， and global relationship among nodes were established. The features of each relationship were synchronously transmitted and effectively fused in the network. In the process of global cooperation of human body parts， the interaction range of each part was limited and depends on specific actions. An adaptive topK global adjacency relationship calculation method was proposed based on Non-local algorithm， in which the nodes with the topK interaction strength were selected dynamically as the adjacent nodes for each node. The experimental results show that the proposed two-stream multi-relational network achieves good action accuracy on the Kinetics dataset and the NTU-RGB+D dataset.

Key words: action recognition； skeleton； GCNs(graph convolutional networks)； multi-relation； topK

CLC Number:

TP18

LIU Fang， QIAO Jian-zhong， DAI Qin， SHI Xiang-bin. Skeleton-based Action Recognition Method with Two-Stream Multi-relational GCNs[J]. Journal of Northeastern University(Natural Science), 2021, 42(6): 768-774.

References

[1]Wang L，Koniusz P，Huynh D.Hallucinating IDT descriptors and I3D optical flow features for action recognition with CNNs［C］//2019 IEEE International Conference on Computer Vision.Seoul，2019: 8697-8707.
[2]Yan S，Xiong Y，Lin D.Spatial temporal graph convolutional networks for skeleton-based action recognition［C］//Thirty-Second AAAI Conference on Artificial Intelligence.New Orleans，2018:7444-7452.
[3]Shahroudy A，Liu J，Ng T，et al.NTU RGB+D: a large scale dataset for 3D human activity analysis［C］//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，2016:1010-1019.
[4]Ke Q，Bennamoun M，An S，et al.A new representation of skeleton sequences for 3D action recognition［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu，2017: 4570-4579.
[5]Shi L，Zhang Y，Cheng J，et al.Two-stream adaptive graph convolutional networks for skeleton-based action recognition［C］//2019 IEEE Conference on Computer Vision and Pattern Recognition.Long Beach，2019:12026-12035.
[6]Hu G，Cui B，Yu S.Skeleton-based action recognition with synchronous local and non-local spatio-temporal learning and frequency attention［C］//2019 IEEE International Conference on Multimedia and Expo.Shanghai，2019:1216-1221.
[7]Zhang P，Lan C，Zeng W，et al.Semantics-guided neural networks for efficient skeleton-based human action recognition［C］//2020 IEEE Conference on Computer Vision and Pattern Recognition.Seattle，2020: 1109-1118.
[8]Gao J，Zhang T，Xu C.I know the relationships: zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs［C］//The Thirty-Third AAAI Conference on Artificial Intelligence.Honolulu，2019: 8303-8311.
[9]Schlichtkrull M S，Kipf T N，Bloem P，et al.Modeling relational data with graph convolutional networks［C］//The Semantic Web 15th International Conference.Heraklion，2018: 593-607.
[10]Ye R，Li X，Fang Y，et al.A vectorized relational graph convolutional network for multi-relational network alignment［C］//28th International Joint Conference on Artificial Intelligence.Macao，2019: 4135-4141.
[11]Vashishth S，Sanyal S，Nitin V，et al.Composition based multi-relational graph convolutional networks［C/OL］//8th International Conference on Learning Representations.［2020-12-21］.https://arxiv.org/pdf/1911.03082.pdf.
[12]Wang X，Girshick R B，Gupta A，et al.Non-local neural networks［C］//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Computer Society，2018: 7794-7803.
[13]Kay W，Carreira J，Simonyan K，et al.The Kinetics human action video dataset［J/OL］.Computing Research Repository.［2020-12-21］.https://arxiv.org/pdf/1705.06950.pdf.
[14]Li S，Li W，Cook C，et al.Independently recurrent neural network (INDRNN):building a longer and deeper RNN［C］//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Computer Society，2018: 5457-5466.
[15]Kim T S，Reiter A.Interpretable 3D human action analysis with temporal convolutional networks［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops.Honolulu，2017: 1623-1631.
[16]Li C，Zhong Q，Xie D，et al.Skeleton-based action recognition with convolutional neural networks［C］//2017 IEEE International Conference on Multimedia and Expo Workshops.Hong Kong，2017: 597-600.
[17]Li B，Dai Y，Cheng X，et al.Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN ［C］//2017 IEEE International Conference on Multimedia and Expo Workshops.Hong Kong，2017: 601-604.
[18]Tang Y，Tian Y，Lu J，et al.Deepprogressive reinforcement learning for skeleton-based action recognition［C］//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City:IEEE Computer Society，2018: 18-22.