Facial Mask Guidance Based Multi-person Scene Images Forgery Localization Algorithm

doi:10.12068/j.issn.1005-3026.2025.20249048

Abstract

Abstract:

To address the performance degradation and lack of robustness in existing forgery localization models when dealing with small region facial manipulations in multi-person scene images， a FMG-L model based on facial mask guidance for forgery localization is proposed. Firstly， to mitigate interference from background information in multi-person scene images， a facial mask guidance module is designed to encourage the model to focus on critical facial regions. Secondly， to enhance the robustness against image degradations， a three-channel feature extraction module is developed to extract multi-dimensional features， and a feature fusion module based on a dual attention network is also designed to enhance the forgery clues. Finally， a forgery localization module is used for forgery localization. Experimental results on the OpenForensics， ManulFake， FFIW， and DiffSwap datasets demonstrate that the FMG-L effectively localizes forgery regions and shows strong robustness against various image degradations and different online social platforms.

Key words: DeepFakes, DeepFake localization, multi-person scene images, small region manipulations, facial mask guidance

CLC Number:

TP 391.4

Jia-tong LIU, Li-na WANG, Run WANG, Xi YE. Facial Mask Guidance Based Multi-person Scene Images Forgery Localization Algorithm[J]. Journal of Northeastern University(Natural Science), 2025, 46(5): 10-19.

Figures/Tables 12

Fig.1 Network architecture of FMG-L model

Fig.2 Comparison of forgery regions between single-face dataset and multi-face scene dataset

Fig.3 Network architecture of Block

Table 1 Forgery localization performance within the datasets

方法	OF			MF			FW			DS
方法	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$
GradCAM	91.0	90.7	95.0	97.9	97.9	98.9	92.5	92.3	96.0	82.3	76.9	86.8
Patch	91.7	91.5	95.4	97.9	97.9	98.9	92.9	92.9	96.2	81.8	76.4	86.4
HiFi-Net	91.6	91.6	95.2	97.3	97.2	98.8	90.5	90.4	95.0	67.8	67.6	78.8
Attention	91.0	91.0	95.0	98.0	98.0	99.0	93.0	92.9	95.6	82.2	76.8	86.7
FMG-F	94.6	94.5	95.3	98.2	98.2	99.8	94.2	94.1	96.6	86.4	84.4	85.8
FMG-L	94.8	94.7	95.4	98.3	98.3	99.8	94.5	94.4	96.6	87.7	85.5	85.4

Table 1 Forgery localization performance within the datasets

方法	OF			MF			FW			DS
方法	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$	Acc	IoU	$F 1$
GradCAM	91.0	90.7	95.0	97.9	97.9	98.9	92.5	92.3	96.0	82.3	76.9	86.8
Patch	91.7	91.5	95.4	97.9	97.9	98.9	92.9	92.9	96.2	81.8	76.4	86.4
HiFi-Net	91.6	91.6	95.2	97.3	97.2	98.8	90.5	90.4	95.0	67.8	67.6	78.8
Attention	91.0	91.0	95.0	98.0	98.0	99.0	93.0	92.9	95.6	82.2	76.8	86.7
FMG-F	94.6	94.5	95.3	98.2	98.2	99.8	94.2	94.1	96.6	86.4	84.4	85.8
FMG-L	94.8	94.7	95.4	98.3	98.3	99.8	94.5	94.4	96.6	87.7	85.5	85.4

Fig.4 Visualization of forgery localization results

Table 2 Forgery localization results across the datasets

方法	训练	MF		FW		DS		训练	OF		FW		DS
方法	训练	Acc	IoU	Acc	IoU	Acc	IoU	训练	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	OF	91.2	91.2	87.5	87.5	68.2	68.0	MF	87.4	87.2	90.2	90.1	73.2	71.6
Patch		92.3	92.2	88.6	89.6	68.5	68.2		89.7	89.5	90.4	90.3	74.2	71.0
HiFi-Net		93.2	93.1	90.7	90.6	67.9	67.9		90.9	90.7	89.4	89.3	70.7	69.6
Attention		92.4	92.5	89.6	89.6	68.4	68.2		88.3	88.2	91.7	91.7	72.5	71.0
FMG-F		95.3	95.3	92.6	92.6	70.5	69.2		92.9	92.7	94.3	94.2	74.9	72.0
FMG-L		95.5	95.5	92.7	92.8	73.2	71.4		93.0	92.8	94.4	94.3	75.6	72.6
方法	训练	OF		MF		DS		训练	OF		MF		FW
方法	训练	Acc	IoU	Acc	IoU	Acc	IoU	训练	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	FW	88.4	88.2	90.4	90.3	71.1	69.5	DS	88.3	88.8	89.7	89.6	89.4	89.3
Patch		90.9	90.7	92.0	93.0	71.4	69.6		84.0	83.3	91.9	91.9	90.0	90.9
HiFi-Net		89.2	89.1	91.8	91.8	67.9	67.9		88.2	88.2	90.9	90.8	88.8	88.8
Attention		90.2	90.2	92.7	92.7	70.7	70.0		89.2	88.7	91.8	91.8	91.1	91.9
FMG-F		91.1	91.0	94.2	94.2	71.1	69.9		90.9	90.8	94.4	94.3	94.2	94.1
FMG-L		91.2	91.1	95.0	94.7	71.5	70.1		91.1	90.9	94.4	94.5	94.3	94.2

Fig.5 Impact of various degrees of image degradations on the Acc of forgery localization

Table 3 Forgery localization performance for different image degradations

方法	原始图像		JPEG		GN		GB		BW		CS		CC
方法	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	91.0	90.7	88.9	88.6	87.9	87.8	88.4	88.3	89.9	89.8	89.8	89.7	90.2	89.9
Patch	91.7	91.5	88.9	88.8	87.8	87.8	88.5	88.4	89.6	89.5	87.7	87.7	89.5	89.4
HiFi-Net	91.6	91.6	91.0	91.0	90.6	90.9	90.9	90.8	91.1	91.1	90.8	90.8	91.0	91.6
Attention	91.0	91.0	89.1	89.0	86.6	86.5	88.4	88.4	89.4	89.3	86.9	87.1	89.6	89.6
FMG-F	94.6	94.5	94.3	94.2	93.9	93.8	94.4	94.4	93.2	93.1	93.6	93.6	92.6	92.4
FMG-L	94.8	94.7	94.5	94.4	94.3	94.2	94.5	94.5	93.4	93.3	93.7	93.7	92.8	92.7

Table 4 Forgery localization performance for different online social platforms

方法	原始图像		Facebook		TikTok		WeChat		WeChat(PC)		WhatsApp		YouTube
方法	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU	Acc	IoU
GradCAM	97.9	97.9	94.5	94.5	95.6	95.5	95.0	94.9	96.3	96.3	95.1	95.0	96.0	95.9
Patch	97.9	97.9	94.5	94.4	95.5	95.5	95.0	94.9	96.2	96.2	95.0	95.0	96.0	95.9
HiFi-Net	97.3	97.2	96.8	96.7	96.4	96.3	96.3	96.2	96.5	96.5	96.4	96.3	96.5	96.5
Attention	98.0	98.0	94.7	94.7	95.7	95.7	95.1	95.1	96.5	96.4	95.2	95.2	96.1	96.1
FMG-F	98.2	98.2	97.3	97.2	97.2	97.1	97.1	97.0	97.0	97.5	97.1	97.0	97.2	97.2
FMG-L	98.3	98.3	97.8	97.7	97.3	97.2	97.4	97.3	97.5	98.0	97.4	97.4	97.4	97.3

Table 5 Forgery localization performance with different number of Blocks on OF dataset %

Block	Acc	IoU
1	93.3	93.2
2	94.8	94.7
3	93.4	93.3
4	92.2	92.0

Table 6 Forgery localization performance with

算子	Acc	IoU
$C 2$ + $C 3$ +DAN	92.4	92.5
$C 1$ + $C 3$ +DAN	89.7	89.6
$C 1$ + $C 2$ +DAN	91.2	91.0
$C 1$ + $C 2$ + $C 3$	93.9	93.8
$C 1$ + $C 2$ + $C 3$ +DAN	94.8	94.7

Table 6 Forgery localization performance with

算子	Acc	IoU
$C 2$ + $C 3$ +DAN	92.4	92.5
$C 1$ + $C 3$ +DAN	89.7	89.6
$C 1$ + $C 2$ +DAN	91.2	91.0
$C 1$ + $C 2$ + $C 3$	93.9	93.8
$C 1$ + $C 2$ + $C 3$ +DAN	94.8	94.7

Table 7 Forgery localization performance with

算子	Acc	IoU
$L b c e$	92.8	92.6
$L f o c a l$	94.3	94.2
$L b c e + L d i c e$	93.7	93.5
$L f o c a l + L d i c e$	94.8	94.7

Table 7 Forgery localization performance with

算子	Acc	IoU
$L b c e$	92.8	92.6
$L f o c a l$	94.3	94.2
$L b c e + L d i c e$	93.7	93.5
$L f o c a l + L d i c e$	94.8	94.7

References 22

[1]	Wang T C， Liu M Y， Zhu J Y， et al. High-resolution image synthesis and semantic manipulation with conditional gans［C］//IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City， 2018： 8798-8807.
[2]	Dhariwal P， Nichol A. Diffusion models beat gans on image synthesis［J］. Advances in Neural Information Processing Systems， 2021， 34： 8780-8794.
[3]	Mirsky Y， Lee W K. The creation and detection of deepfakes： a survey［J］. ACM Computing Surveys （CSUR）， 2021， 54（1）： 1-41.
[4]	Le T N， Nguyen H H， Yamagishi J， et al. Openforensics： large-scale challenging dataset for multi-face forgery detection and segmentation in-the-wild［C］// IEEE/CVF International Conference on Computer Vision. Montreal， 2021： 10117-10127.
[5]	Agarwal A， Ratha N. Deepfake Catcher： can a simple fusion be effective and outperform complex DNNs？［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， 2024： 3791-3801.
[6]	He K M， Gkioxari G， Dollár P， et al. Mask r-CNN［C］// IEEE International Conference on Computer Vision. Venice， 2017： 2961-2969.
[7]	Zhou P， Han X T， Morariu V I， et al. Learning rich features for image manipulation detection［C］// IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City， 2018： 1053-1061.
[8]	Fu J， Liu J， Tian H J， et al. Dual attention network for scene segmentation［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， 2019： 3146-3154.
[9]	Mao A Q， Mohri M， Zhong Y T. Cross-entropy loss functions： theoretical analysis and applications［C］// The 40th International Conference on Machine Learning. Honolulu， 2023： 23803-23828.
[10]	Ross T Y， Dollár G. Focal loss for dense object detection［C］// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 2980-2988.
[11]	Kumar A， Guo Y L， Huang X Y， et al. SeaBird： segmentation in bird's view with dice loss improves monocular 3D detection of large objects［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， 2024： 10269-10280.
[12]	Wu H W， Zhou J T， Zhang S L， et al. Exploring spatial-temporal features for deepfake detection and localization［EB/OL］.（2022-10-28）［2024-08-13］. .
[13]	Zhou T F， Wang W G， Liang Z Y， et al. Face forensics in the wild［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， 2021： 5778-5788.
[14]	Chen Z X， Sun K， Zhou Z Y， et al. DiffusionFace： towards a comprehensive dataset for diffusion-based face forgery analysis［EB/OL］. （2024-03-27）［2024-08-13］. .
[15]	Selvaraju R R， Cogswell M， Das A， et al. Grad-CAM： visual explanations from deep networks via gradient-based localization［EB/OL］. （2024-03-27）［2024-08-13］. .
[16]	Chollet F. Xception： deep learning with depth wise separable convolutions［C］// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， 2017： 1251-1258.
[17]	Chai L， Bau D， Lim S N， et al. What makes fake images detectable？ understanding properties that generalize［C］// European Conference on Computer Vision. Glasgow， 2020： 103-120.
[18]	Guo X， Liu X H， Ren Z Y， et al. Hierarchical fine-grained image forgery detection and localization［C］// IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver， 2023： 3155-3165.
[19]	Țânțaru D C， Oneață E， Oneață D. Weakly-supervised deepfake localization in diffusion-generated images［C］//IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa， 2024： 6258-6268.
[20]	Huang Y H， Xu J F， Wang R， et al. Fakelocator： robust localization of GAN-based face manipulations［J］. IEEE Transactions on Information Forensics and Security， 2022， 17： 2657-2672.
[21]	Loshchilov I. Decoupled weight decay regularization［EB/OL］.（2017-11-14）［2024-08-13］..
[22]	Jiang L M， Li R， Wu W， et al. Deeperforensics-1.0： a large-scale dataset for real-world face forgery detection［C］//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， 2020： 2889-2898.