基于几何注意力机制的三维手部姿势估计算法

doi:10.12068/j.issn.1005-3026.2025.20240079

摘要/Abstract

摘要：

在Transformer编码-解码的基础架构上设计了手势识别网络，在自注意力机制的基础上引入了优化的偏移注意力机制来提取手部特征.同时为了更好地提取手部结构的局部特征，设计了邻域聚合策略.手部结构自身的三维复杂性导致其不同区域的平滑程度不同，进行手部姿势估计时，忽略这种特征会使手部结构的局部关键信息丢失，为了解决这一问题，对手部结构进行了几何分解，分别用锐变成分和柔变成分来表示手部结构的尖锐区域和平坦区域，通过注意力机制对这两种成分的特征给予不同的关注.在MSRA,ICVL和NYU数据集上的实验验证了此算法的准确度与SOTA算法相当.

关键词: 手势识别, 三维点云, 注意力机制, Transformer模型, 深度学习

Abstract:

A gesture recognition network based on the coding and decoding infrastructure of Transformer was designed， and an optimized offset attention mechanism was introduced to extract hand features based on the self-attention mechanism. At the same time， in order to extract the local features of the hand structure better， a neighborhood aggregation strategy was designed. The three-dimensional （3D） complexity of the hand structure itself led to different levels of smoothness in different regions. When estimating gestures， ignoring this feature usually leads to the loss of local key information of the hand structure. In order to solve this problem， geometric decomposition of the hand structure was carried out， and sharp and flexible components were used to represent the sharp and flat regions of the hand structure， respectively. Different attention was paid to the characteristics of these two components through the attention mechanism. Experiments on MSRA， ICVL， and NYU datasets demonstrate that the accuracy of this algorithm is comparable to that of SOTA.

Key words: gesture recognition, 3D point cloud, attention mechanism, Transformer model, deep learning

中图分类号:

TP 391.4

邹慧, 佘黎煌, 陈烨涵, 乐意. 基于几何注意力机制的三维手部姿势估计算法[J]. 东北大学学报（自然科学版）, 2025, 46(10): 44-50.

Hui ZOU, Li-huang SHE, Ye-han CHEN, Yi YUE. 3D Gesture Estimation Algorithm Based on Geometric Attention Mechanism[J]. Journal of Northeastern University(Natural Science), 2025, 46(10): 44-50.

图/表 9

图1 算法总体框图

Fig.1 Overall block diagram of algorithm

图2 手部几何分解示意图

Fig.2 Schematic diagram of geometric decomposition of hand

图3 SG采样分组示意图

Fig.3 Schematic diagram of SG sampling grouping

图4 手势估计结果示意图（a）—MSRA；（b）—NYU；（c）—ICVL.

Fig.4 Schematic diagram of gesture estimation results

表1 平均误差距离的对比

Table 1 Comparison of mean error distance

方法	平均误差/mm	方法	平均误差/mm	方法	平均误差/mm
方法	NYU	方法	ICVL	方法	MSRA
3DCNN^[13]	14.1	DeepMode1^[17]	11.6	Cascade	15.2
Pose-REN	11.8	Cascade	9.9	Occlusion^[19]	12.8
DeepPrior++	12.2	CrossingNets^[18]	10.2	CrossingNets	12.2
REN-9×6×6^[14]	12.7	DeepPrior++	8.1	REN-9×6×6	9.7
Feedback^[15]	16.0	LRF	12.6	DeepPrior++	9.5
Hand3D^[16]	17.6	Hand PointNet	6.9	Hand PointNet	8.5
本文	12.0	本文	6.7	本文	8.1

图5 各关节点的平均误差距离的对比（a）—MSRA；（b）—NYU；（c）—ICVL.

Fig.5 Comparison of average error distance of each node

图6 模型质量分析

Fig.6 Model quality analysis

表2 互补注意效果

Table 2 Complementary attention effect

自身融合	锐变分量	柔变分量	误差/mm
√			8.6
	√		8.3
		√	8.4
	√	√	8.1

图7 均方误差

Fig.7 Mean squared error

参考文献 19

[1]	Ahmet G， Neslihan K， Gerhard R， et al. Real-time hand gesture detection and classification using convolutional neural networks ［C］// 2019 IEEE International Conference on Automatic Face and Gesture Recognition. Lille， 2019： 1-8.
[2]	Chen X H， Wang G J， Guo H K， et al. Pose guided structured region ensemble network for cascaded hand pose estimation ［J］. Neurocomputing， 2020， 395： 138-149.
[3]	Moon G， Chang J Y， Lee K M， et al. V2V PoseNet： voxel-to-voxel prediction network for accurate 3D hand and human pose estimation from a single depth map ［C］// IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Salt Lake City， 2018： 5079-5088.
[4]	Ge L H， Cai Y J， Weng J W， et al. Hand PointNet： 3D hand pose estimation using point set ［C］// 2018 IEEE International Conference on Computer Vision and Pattern Recognition. Salt Lake City， 2018： 8417-8426.
[5]	Chen Y J， Tu Z G， Ge L H， et al. So-HandNet： self-organizing network for 3D hand pose estimation with semi-supervised learning ［C］// 2019 International Conference on Computer Vision. Seoul， 2019： 6960-6969.
[6]	Qi C R， Yi L， Su H， et al. PointNet++： deep hierarchical feature learning on point sets in a metric space ［J］. Advances in Neural Information Processing Systems， 2017， 30（10）： 5105-5114.
[7]	Xu M T， Zhang J H， Zhou Z P. Learning geometry-disent angled representation for complementary understanding of 3D object point cloud ［C］// 2021 AAAI Conference on Artificial Intelligence. Vancouver， 2021， 35（4）： 3056-3064.
[8]	Guo M H， Liu Z J. Point cloud transformer ［J］. Computational Visual Media，2021， 7（2）： 187-199.
[9]	Sun X， Wei Y C， Liang S， et al. Cascaded hand pose regression ［C］// 2015 IEEE International Conference on Computer Vision and Pattern Recognition. Boston， 2015： 824-832.
[10]	Tompson J， Stein M， Lecun Y， et al. Real-time continuous pose recovery of human hands using convolutional networks ［J］. ACM Transactions on Graphics， 2014， 33（5）： 1-10.
[11]	Tang D H， Chang C J， Alykhan， et al. Latent regression forest： structured estimation of 3D articulated hand posture ［C］// 2014 IEEE International Conference on Computer Vision and Pattern Recognition. Columbus， 2014： 3786-3793.
[12]	Markus O， Vincent L. Deepprior++： improving fast and accurate 3D hand pose estimation. ［C］// 2017 IEEE International Conference on Computer Vision Workshop. Venice， 2017： 585-594.
[13]	Ge L H， Liang H， Yuan J S， et al. 3D convolutional neural networks for efficient and robust hand pose estimation from single depth images ［C］// 2017 IEEE International Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 5679-5688.
[14]	Wang G J， Chen X H， Guo H K， et al. Region ensemble network： towards good practices for deep 3D hand pose estimation ［J］. Journal of Visual Communication and Image Representation，2018， 55（8）： 404-414.
[15]	Markus O， Wohlhart P， Lepetit V. Training a feedback loop for hand pose estimation ［C］// 2015 IEEE/CVF International Conference on Computer Vision. Santiago， 2015： 3316-3324.
[16]	Deng X M， Yang S， Zhang Y D， et al. Hand3D： hand pose estimation using 3D neural network ［C］// 2017 IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 549-557.
[17]	Zhou X Y， Wan Q F， Zhang W， et al. Model-based deep hand pose estimation ［C］// 2016 International Joint Conference on Artificial Intelligence. New York， 2016： 2421-2427.
[18]	Wan C D， Probst T， Luc V G， et al. Crossing nets： combining GANs and VAEs with a shared latent space for hand pose estimation ［C］// 2017 IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 1196-1205.
[19]	Madadi M， Escalera S， Carruesco A， et al. Occlusion aware hand pose recovery from ` sequences of depth images ［C］// 2017 International Conference on Automatic Face & Gesture Recognition. Washington DC， 2017： 230-237.