基于面部掩码引导的多人场景图像伪造定位算法

doi:10.12068/j.issn.1005-3026.2025.20249048

摘要/Abstract

摘要：

为解决现有伪造定位算法在小区域面部篡改的多人场景图像时性能下降、鲁棒性不足的问题，提出一种基于面部掩码引导的伪造定位模型FMG-L.首先，为了减轻多人场景图像中背景信息的干扰，设计面部掩码引导模块，鼓励FMG-L关注重要的面部区域；其次，为了提升FMG-L面对图像质量退化的鲁棒性，设计三通道特征提取模块提取多维特征，结合基于双重注意力网络的特征融合模块，增强模型学习到的伪造线索；最后，使用伪造区域定位模块进行伪造定位.在OpenForensics，ManulFake，FFIW和DiffSwap数据集上的实验结果表明，FMG-L能够有效进行伪造定位，具有面对多种图像退化和不同在线社交平台的强鲁棒性.

关键词: 深度伪造, 深度伪造定位, 多人场景图像, 小区域篡改, 面部掩码引导

Abstract:

To address the performance degradation and lack of robustness in existing forgery localization models when dealing with small region facial manipulations in multi-person scene images， a FMG-L model based on facial mask guidance for forgery localization is proposed. Firstly， to mitigate interference from background information in multi-person scene images， a facial mask guidance module is designed to encourage the model to focus on critical facial regions. Secondly， to enhance the robustness against image degradations， a three-channel feature extraction module is developed to extract multi-dimensional features， and a feature fusion module based on a dual attention network is also designed to enhance the forgery clues. Finally， a forgery localization module is used for forgery localization. Experimental results on the OpenForensics， ManulFake， FFIW， and DiffSwap datasets demonstrate that the FMG-L effectively localizes forgery regions and shows strong robustness against various image degradations and different online social platforms.

Key words: DeepFakes, DeepFake localization, multi-person scene images, small region manipulations, facial mask guidance

中图分类号:

TP 391.4

刘佳彤, 王丽娜, 汪润, 叶茜. 基于面部掩码引导的多人场景图像伪造定位算法[J]. 东北大学学报（自然科学版）, 2025, 46(5): 10-19.

Jia-tong LIU, Li-na WANG, Run WANG, Xi YE. Facial Mask Guidance Based Multi-person Scene Images Forgery Localization Algorithm[J]. Journal of Northeastern University(Natural Science), 2025, 46(5): 10-19.

图/表 12

图1 FMG-L模型网络结构

Fig.1 Network architecture of FMG-L model

图2 单人脸数据集与多人场景数据集的伪造区域对比（a）—单人脸数据集；（b）—多人场景数据集.

Fig.2 Comparison of forgery regions between single-face dataset and multi-face scene dataset

图3 Block网络结构

Fig.3 Network architecture of Block

表 1 数据集内伪造定位性能 (%)

Table 1 Forgery localization performance within the datasets

方法	OF			MF			FW			DS
方法	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$
GradCAM	91.0	90.7	95.0	97.9	97.9	98.9	92.5	92.3	96.0	82.3	76.9	86.8
Patch	91.7	91.5	95.4	97.9	97.9	98.9	92.9	92.9	96.2	81.8	76.4	86.4
HiFi-Net	91.6	91.6	95.2	97.3	97.2	98.8	90.5	90.4	95.0	67.8	67.6	78.8
Attention	91.0	91.0	95.0	98.0	98.0	99.0	93.0	92.9	95.6	82.2	76.8	86.7
FMG-F	94.6	94.5	95.3	98.2	98.2	99.8	94.2	94.1	96.6	86.4	84.4	85.8
FMG-L	94.8	94.7	95.4	98.3	98.3	99.8	94.5	94.4	96.6	87.7	85.5	85.4

表 1 数据集内伪造定位性能 (%)

Table 1 Forgery localization performance within the datasets

方法	OF			MF			FW			DS
方法	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$
GradCAM	91.0	90.7	95.0	97.9	97.9	98.9	92.5	92.3	96.0	82.3	76.9	86.8
Patch	91.7	91.5	95.4	97.9	97.9	98.9	92.9	92.9	96.2	81.8	76.4	86.4
HiFi-Net	91.6	91.6	95.2	97.3	97.2	98.8	90.5	90.4	95.0	67.8	67.6	78.8
Attention	91.0	91.0	95.0	98.0	98.0	99.0	93.0	92.9	95.6	82.2	76.8	86.7
FMG-F	94.6	94.5	95.3	98.2	98.2	99.8	94.2	94.1	96.6	86.4	84.4	85.8
FMG-L	94.8	94.7	95.4	98.3	98.3	99.8	94.5	94.4	96.6	87.7	85.5	85.4

图4 伪造定位结果可视化

Fig.4 Visualization of forgery localization results

表 2 跨数据集伪造定位结果 (%)

Table 2 Forgery localization results across the datasets

方法	训练	MF		FW		DS		训练	OF		FW		DS
方法	训练	Acc	IoU	Acc	IoU	Acc	IoU	训练	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	OF	91.2	91.2	87.5	87.5	68.2	68.0	MF	87.4	87.2	90.2	90.1	73.2	71.6
Patch		92.3	92.2	88.6	89.6	68.5	68.2		89.7	89.5	90.4	90.3	74.2	71.0
HiFi-Net		93.2	93.1	90.7	90.6	67.9	67.9		90.9	90.7	89.4	89.3	70.7	69.6
Attention		92.4	92.5	89.6	89.6	68.4	68.2		88.3	88.2	91.7	91.7	72.5	71.0
FMG-F		95.3	95.3	92.6	92.6	70.5	69.2		92.9	92.7	94.3	94.2	74.9	72.0
FMG-L		95.5	95.5	92.7	92.8	73.2	71.4		93.0	92.8	94.4	94.3	75.6	72.6
方法	训练	OF		MF		DS		训练	OF		MF		FW
方法	训练	Acc	IoU	Acc	IoU	Acc	IoU	训练	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	FW	88.4	88.2	90.4	90.3	71.1	69.5	DS	88.3	88.8	89.7	89.6	89.4	89.3
Patch		90.9	90.7	92.0	93.0	71.4	69.6		84.0	83.3	91.9	91.9	90.0	90.9
HiFi-Net		89.2	89.1	91.8	91.8	67.9	67.9		88.2	88.2	90.9	90.8	88.8	88.8
Attention		90.2	90.2	92.7	92.7	70.7	70.0		89.2	88.7	91.8	91.8	91.1	91.9
FMG-F		91.1	91.0	94.2	94.2	71.1	69.9		90.9	90.8	94.4	94.3	94.2	94.1
FMG-L		91.2	91.1	95.0	94.7	71.5	70.1		91.1	90.9	94.4	94.5	94.3	94.2

图5 不同程度图像退化对伪造定位Acc的影响（a）—JPEG压缩对Acc影响；（b）—高斯噪声对Acc影响；（c）—高斯模糊对Acc影响；（d）—块扰动对Acc影响；（e）—颜色饱和度对Acc影响；（f）—颜色对比度对Acc影响.

Fig.5 Impact of various degrees of image degradations on the Acc of forgery localization

表 3 面对不同图像退化的伪造定位性能 (%)

Table 3 Forgery localization performance for different image degradations

方法	原始图像		JPEG		GN		GB		BW		CS		CC
方法	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	91.0	90.7	88.9	88.6	87.9	87.8	88.4	88.3	89.9	89.8	89.8	89.7	90.2	89.9
Patch	91.7	91.5	88.9	88.8	87.8	87.8	88.5	88.4	89.6	89.5	87.7	87.7	89.5	89.4
HiFi-Net	91.6	91.6	91.0	91.0	90.6	90.9	90.9	90.8	91.1	91.1	90.8	90.8	91.0	91.6
Attention	91.0	91.0	89.1	89.0	86.6	86.5	88.4	88.4	89.4	89.3	86.9	87.1	89.6	89.6
FMG-F	94.6	94.5	94.3	94.2	93.9	93.8	94.4	94.4	93.2	93.1	93.6	93.6	92.6	92.4
FMG-L	94.8	94.7	94.5	94.4	94.3	94.2	94.5	94.5	93.4	93.3	93.7	93.7	92.8	92.7

表 4 面对不同在线社交平台的伪造定位性能 (%)

Table 4 Forgery localization performance for different online social platforms

方法	原始图像		Facebook		TikTok		WeChat		WeChat(PC)		WhatsApp		YouTube
方法	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	97.9	97.9	94.5	94.5	95.6	95.5	95.0	94.9	96.3	96.3	95.1	95.0	96.0	95.9
Patch	97.9	97.9	94.5	94.4	95.5	95.5	95.0	94.9	96.2	96.2	95.0	95.0	96.0	95.9
HiFi-Net	97.3	97.2	96.8	96.7	96.4	96.3	96.3	96.2	96.5	96.5	96.4	96.3	96.5	96.5
Attention	98.0	98.0	94.7	94.7	95.7	95.7	95.1	95.1	96.5	96.4	95.2	95.2	96.1	96.1
FMG-F	98.2	98.2	97.3	97.2	97.2	97.1	97.1	97.0	97.0	97.5	97.1	97.0	97.2	97.2
FMG-L	98.3	98.3	97.8	97.7	97.3	97.2	97.4	97.3	97.5	98.0	97.4	97.4	97.4	97.3

表 5 不同Block数量在OF数据集上的伪造定位性能

Table 5 Forgery localization performance with different number of Blocks on OF dataset %

Block	Acc	IoU
1	93.3	93.2
2	94.8	94.7
3	93.4	93.3
4	92.2	92.0

表 6 不同模块在OF数据集上的伪造定位性能 (different modules on OF dataset %)

Table 6 Forgery localization performance with

算子	Acc	IoU
$C 2$ + $C 3$ +DAN	92.4	92.5
$C 1$ + $C 3$ +DAN	89.7	89.6
$C 1$ + $C 2$ +DAN	91.2	91.0
$C 1$ + $C 2$ + $C 3$	93.9	93.8
$C 1$ + $C 2$ + $C 3$ +DAN	94.8	94.7

表 6 不同模块在OF数据集上的伪造定位性能 (different modules on OF dataset %)

Table 6 Forgery localization performance with

算子	Acc	IoU
$C 2$ + $C 3$ +DAN	92.4	92.5
$C 1$ + $C 3$ +DAN	89.7	89.6
$C 1$ + $C 2$ +DAN	91.2	91.0
$C 1$ + $C 2$ + $C 3$	93.9	93.8
$C 1$ + $C 2$ + $C 3$ +DAN	94.8	94.7

表 7 不同损失函数在OF数据集上的伪造定位性能 (different loss functions on OF dataset %)

Table 7 Forgery localization performance with

算子	Acc	IoU
$L b c e$	92.8	92.6
$L f o c a l$	94.3	94.2
$L b c e + L d i c e$	93.7	93.5
$L f o c a l + L d i c e$	94.8	94.7

表 7 不同损失函数在OF数据集上的伪造定位性能 (different loss functions on OF dataset %)

Table 7 Forgery localization performance with

算子	Acc	IoU
$L b c e$	92.8	92.6
$L f o c a l$	94.3	94.2
$L b c e + L d i c e$	93.7	93.5
$L f o c a l + L d i c e$	94.8	94.7

参考文献 22

[1]	Wang T C， Liu M Y， Zhu J Y， et al. High-resolution image synthesis and semantic manipulation with conditional gans［C］//IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City， 2018： 8798-8807.
[2]	Dhariwal P， Nichol A. Diffusion models beat gans on image synthesis［J］. Advances in Neural Information Processing Systems， 2021， 34： 8780-8794.
[3]	Mirsky Y， Lee W K. The creation and detection of deepfakes： a survey［J］. ACM Computing Surveys （CSUR）， 2021， 54（1）： 1-41.
[4]	Le T N， Nguyen H H， Yamagishi J， et al. Openforensics： large-scale challenging dataset for multi-face forgery detection and segmentation in-the-wild［C］// IEEE/CVF International Conference on Computer Vision. Montreal， 2021： 10117-10127.
[5]	Agarwal A， Ratha N. Deepfake Catcher： can a simple fusion be effective and outperform complex DNNs？［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， 2024： 3791-3801.
[6]	He K M， Gkioxari G， Dollár P， et al. Mask r-CNN［C］// IEEE International Conference on Computer Vision. Venice， 2017： 2961-2969.
[7]	Zhou P， Han X T， Morariu V I， et al. Learning rich features for image manipulation detection［C］// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City， 2018： 1053-1061.
[8]	Fu J， Liu J， Tian H J， et al. Dual attention network for scene segmentation［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， 2019： 3146-3154.
[9]	Mao A Q， Mohri M， Zhong Y T. Cross-entropy loss functions： theoretical analysis and applications［C］// The 40th International Conference on Machine Learning. Honolulu， 2023： 23803-23828.
[10]	Ross T Y， Dollár G. Focal loss for dense object detection［C］// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 2980-2988.
[11]	Kumar A， Guo Y L， Huang X Y， et al. SeaBird： segmentation in bird's view with dice loss improves monocular 3D detection of large objects［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， 2024： 10269-10280.
[12]	Wu H W， Zhou J T， Zhang S L， et al. Exploring spatial-temporal features for deepfake detection and localization［EB/OL］.（2022-10-28）［2024-08-13］. .
[13]	Zhou T F， Wang W G， Liang Z Y， et al. Face forensics in the wild［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， 2021： 5778-5788.
[14]	Chen Z X， Sun K， Zhou Z Y， et al. DiffusionFace： towards a comprehensive dataset for diffusion-based face forgery analysis［EB/OL］. （2024-03-27）［2024-08-13］. .
[15]	Selvaraju R R， Cogswell M， Das A， et al. Grad-CAM： visual explanations from deep networks via gradient-based localization［EB/OL］. （2024-03-27）［2024-08-13］. .
[16]	Chollet F. Xception： deep learning with depth wise separable convolutions［C］// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 1251-1258.
[17]	Chai L， Bau D， Lim S N， et al. What makes fake images detectable？ understanding properties that generalize［C］// European Conference on Computer Vision. Glasgow， 2020： 103-120.
[18]	Guo X， Liu X H， Ren Z Y， et al. Hierarchical fine-grained image forgery detection and localization［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver， 2023： 3155-3165.
[19]	Țânțaru D C， Oneață E， Oneață D. Weakly-supervised deepfake localization in diffusion-generated images［C］//IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa， 2024： 6258-6268.
[20]	Huang Y H， Xu J F， Wang R， et al. Fakelocator： robust localization of GAN-based face manipulations［J］. IEEE Transactions on Information Forensics and Security， 2022， 17： 2657-2672.
[21]	Loshchilov I. Decoupled weight decay regularization［EB/OL］.（2017-11-14）［2024-08-13］..
[22]	Jiang L M， Li R， Wu W， et al. Deeperforensics-1.0： a large-scale dataset for real-world face forgery detection［C］//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， 2020： 2889-2898.