SupCon-MPL-DP: Supervised Contrastive Learning with Meta Pseudo Labels for Deepfake Image Detection

1. Introduction

Recently, with the advancement of generative artificial intelligence models [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15], deepfakes have become increasingly similar to real images/videos, making them difficult to distinguish. Deepfakes can be broadly categorized into three generations based on the evolution of generative models. First-generation deepfake generative models[1,2,3,4,5] typically attempt to generate simple and low-resolution images/videos based on probability distributions or synthesize multiple images/videos by exploiting features in tasks such as Face2Face and FaceSwap. In particular, GAN-based models such as CGAN [2], WGAN [4], WGAN-GP [5], as well as AutoEncoder-based models like VAE [6], Conditional VAE [7], have enabled the generation of various deepfake images/videos. However, first-generation deepfakes often exhibit noticeable artifacts that can be discerned by the human eye. With the transition to second-generation deepfake generative models [6,7,8], there has been progress in generating high-resolution deepfake images that are more difficult to distinguish compared to first-generation ones, along with performance improvements in various tasks. In particular, second-generation deepfake generative models like StyleGAN, proposed by T. Karras et al. [8], produce deepfake images that are difficult for the human eye to distinguish, excluding some flaws such as artifacts in hair. Second-generation deepfakes can be generated using deepfake generation tools such as DeepfaceLab [10], DeepSwap [11], Synthesia [12] and others. Finally, third-generation deepfake generative models [9,13,14,15] produce images/videos that are even more flexible and difficult to distinguish than those generated by second-generation models, across various tasks. In particular, Stable Diffusion, proposed by R. Rombach et al. [13], is currently being used for the generation of various human and artwork images, raising concerns related to copyright and human rights issues. Furthermore, the Consistency model proposed by Y. Song et al. [15] has enabled state-of-the-art deepfake generative model training at a lower cost by reducing the extensive iteration process required by previous Diffusion models for restoring original images from noise. As deepfakes increasingly become difficult to distinguish from real images/videos, they are being utilized in various criminal activities.

To address issues caused by deepfakes, methods have been proposed to identify flaws in landmarks that occur when deepfake generative models create images, aiding in the detection of deepfakes [16,17,18,19]. Meanwhile, recent advancements in deepfake detection for single models have shown improvement in detecting deepfake videos and images. D.A. Coccomini et al. [20] enhanced the detection performance of deepfake videos and images by combining EfficientNet [21] and Vision Transformer [22] when training a single model. In other words, existing deepfake detection models [16,17,18,19,20,21,22,23] verify flaws in facial landmarks during the preprocessing stage and construct large models for flexible predictions.

Previous studies have primarily focused on the detection performance of labeled known domain (Known Domain) tasks in deepfake detection. However, deepfake generation models are rapidly evolving, and similar generations of deepfake generation models are also being developed diversely. Therefore, detecting deepfake images in unknown domains (Unknown Domain) is also crucial. A few studies have proposed generalized deepfake detection models using techniques such as Contrastive Learning, Meta Learning, and others [24,25,26,27,28,29,30,31,32,33,34,35,36].

In this paper, we propose SupCon-MPL, combination of the Meta Pseudo Labels (MPL) [37] with Supervised Contrastive Learning (SupCon) [38] to further train the model with unlabeled images/videos, simultaneously enhancing the model's generalization ability to distinguish deepfakes in unknown domains. The proposed SupCon-MPL utilizes the basic structure of MPL, where two models, namely teacher and student, are simultaneously trained. Each model influences the other during training. The teacher model constructs pseudo-labels for unlabeled images and transfers them to the student model. Through this approach, the student model learns from unlabeled data, providing the potential to train effectively with limited labeled data. Furthermore, during the training process, we apply the SupConLoss [38] to the encoder of each model, enabling Contrastive Representation Learning, thereby inducing generalized model training.

The performance evaluation experiments were conducted in two parts: model validation experiments and deepfake detection experiment based on scenario. In the model validation experiments, we utilized the data from five domains within FaceForensics++ [39]. We evaluated the detection performance in labeled Known Domains by combining the data in various ways and assessed the generalized detection performance in Unknown Domains. The deepfake detection experiment based on scenario involves training the model with first-generation’s deepfake datasets (FaceForensics++ [39], DFDC [40], Celeb-DF [41]) and evaluating the detection performance on first- and second-generation unknown deepfake datasets (StyleGAN [8], NeuralTextures [39]). The experimental results showed that SupCon-MPL achieved performance improvements of 1.58%, 1.32%, and 8.74% over the baseline MPL model in the proposed evaluation scenario, respectively.

2. Related Works

As deepfake generation models advance, various detection methods have also been researched. A common approach in deepfake detection is to explore flaws in facial images [16,17,18,19]. However, the continual development of new generative models has led to the problem of being unable to train detection models using data from all generative models. To address this problem, a few studies have explored training generalized deepfake detection models [24,25,26,27,28,29,30,31,32,33,34,35,36].

2.1. Generalized Deepfake Detection

The generalization of deepfake detection implies the ability to detect deepfake videos generated not only by the models used during training but also by unseen or new generative models. In other word, as generative models progress from GANs, VAEs to Diffusion and Consistency models, achieving the ability to detect deepfakes generated by various and new generative models simultaneously is the main goal of generalized deepfake detection techniques. Recently, research has been conducted on detecting deepfake videos that are unknown from both the data and training perspectives.

On the data perspective, SBL [29] and OST [30] enhanced the generalization of deepfake detection by synthesizing additional training data by combining original images from each generative model with various other images and selectively using them. On the training perspective, A. Jain et al. [25] utilized datasets from Google, Jigsaw, FaceForensics++ [39], Celeb-DF [41], Deepfake-TIMIT [42], and their own database DF-Mobio to train a generalized deepfake detection model using Contrastive Representation Learning across various domains. A. Nadimpalli et al. [26] proposed a hybrid learning technique combining supervised learning and reinforcement learning. In particular, during the training process, the reinforcement learning agent selects the top k augmentations that have the most significant impact on performance improvement when training through CNN and uses them for testing, enabling the training of a generalized deepfake detection model. We employed the meta-learning technique Meta Pseudo Labels in the deepfake training process, appling it after domain splitting for each data, resulting in training a model with higher performance in the same model training [24].

2.2. Contrastive Representation Learning in Deepfake Detection

Currently, research applying Contrastive Representation Learning (CRL) for training generalized deepfake detection models is conducted. CRL enables learning similar features in the feature space between a specific image and from the same domain (positive images), while also learning features that differentiate from different domain (negative images). H. Chih-Chung et al. [27] utilized Contrastive Loss [43] to train the encoder, following which they trained the classifier to generalize the discriminative performance on deepfake images generated by various GAN-based models. S. Fung et al. [36] trained the encoder using unsupervised CRL with image pairs that include random augmentation applied to the same image during the training process. Following this, they trained the classifier using labeled images to develop a generalized deepfake detection model. X. Ying et al. [35] addressed the issue of conventional CRL techniques not utilizing label information of deepfake images by applying Supervised Contrastive Learning (SupCon) [38], which use label information. However, CRL requires a large amount of data, especially a significant number of negative samples. In this paper, we performed CRL using SupCon [38], while simultaneously combining the SupCon model with meta-learning method, MPL [37]. This approach also allows for additional CRL training on unlabeled data, even when using the same labeled data, enabling the training of a more generalized deepfake detection model compared to conventional MPL.

2.3. Meta Pseudo Labels

Meta Pseudo Labels (MPL) [37] trains the model using unlabeled images, and when the same model is trained on an image classification task, it has shown improved performance compared to conventional models. MPL gained significant attention by achieving over 90% Top 1 score on the ImageNet [44] classification task, marking a significant milestone. Figure 1a shows the MPL facilitates the learning of the Teacher model through feedback from the Student model, thus enhancing the conventional learning techniques such as Knowledge Distillation [45] or Noisy Student[46], where the Teacher model passes on information to the Student model. This improvement addresses the issue of inadequate learning of the student model when the performance of the Teacher model is subpar. The training process of the MPL is shown in Figure 1b. The student model in MPL learns through the Pseudo Labels inferred by the Teacher model. Subsequently, it imparts the feedback value regarding the learning to the Teacher model. The Teacher model learns through the Labeled Loss from the labeled data, UDA Loss [47], feedback from the student model, and MPL Loss from the unlabeled data. But it consumes substantial computing resources due to the simultaneous training of the two models.

From the perspective of training a generalized deepfake detection model, MPL can enhance the performance of generalized deepfake detection by enabling additional learning through unlabeled data, compared to models trained solely with labeled data. In this paper, we experiment with the enhancement of detection capabilities for unknown domains and post-generation deepfakes, using both MPL [37] and SupCon [38].

3. Proposed SupCon-MPL Based Deepfake Detection

To detect deepfake videos in the Deepfake Unknown Domain, the proposed method introduces SupCon-MPL, a meta-learning model based on CRL, utilizing unlabeled Images from the Deepfake Known Domain.

3.1. Proposed Training Strategy

3.1.1. Known Domain and Unknown Domain in Deepfake

A deepfake domain can be defined as a collection of images and their features, generated from a "specific deepfake generative model". In this paper, we distinguish deepfake domains into known domain ($\mathit{K}$) and Unknown Domain ($\mathit{U}$). Known domain ($\mathit{K}$) refers to a collection of deepfake images that are labeled when training models. The data in $\mathit{K}$ is labeled and therefore can be directly used for training. Meanwhile, unknown domain ($\mathit{U}$) refers to data created by unknown deepfake generative models. The data in $\mathit{U}$ is not labeled, hence it is not possible to determine whether the image is real or fake. Also, as they are created from various generative models, they can involve various features. Known domain $\mathit{K}$ can be defined as $\mathrm{K}=\{{\mathrm{K}}_1,{\mathrm{K}}_2,\cdot \cdot \cdot \}$ where $K_i$is $i-th$ known deepfake generative model, and the deepfake dataset ${\mathbf{D}}_{\mathbf{K}}=\{D_{K_1},D_{K_2},\cdot \cdot \cdot \}$ consists of a dataset ${\mathit{D}}_{{\mathit{K}}_{\mathit{i}}}=\left\{\left(x_i,y_i\right)\right\}$ composed with a set of deepfake images $x_i$ and labels $y_i$ generated by $K_i$. On the other hand, unknown domain $\mathit{U}$ can be defined as $\mathbf{U}=\{U_1,U_2,\cdot \cdot \cdot \}$ where $U_i$ is $i-th$ unknown deepfake generative model, and the deepfake dataset ${\mathbf{D}}_{\mathbf{U}}=\{D_{U_1},D_{U_2},\cdot \cdot \cdot \}$ consists of a dataset $D_{U_i}=\left\{\left(x_i\right)\right\}$ composed with a set of deepfake images $x_i$ generated by $U_i$.

In this paper, to address $\mathit{U}$, we first experiment by distinguishing $D_K$ into a labeled dataset ($D_L$) and an Unlabeled Dataset ($D_{UL}$) as shown in Figure 2(a). Subsequently, to verify the influence of ${\mathit{D}}_{\mathit{U}}$ on the training process, we assume ${\mathit{D}}_{\mathit{U}}$ as $D_{UL}$ and perform experiments as shown in Figure 2(b). In the deepfake training scenario, from the perspective of generative models by generation, both $D_L$ and $D_{UL}$ constitute with first-generation ${\mathit{D}}_{\mathit{K}}$, and evaluation is conducted using the first and second generation ${\mathit{D}}_{\mathit{U}}$.

3.1.2. Training Strategy for Deepfake Unknown Domain Detection

The training process is employed based on a comparison between the base model and the student model of the MPL (SupCon-MPL). Upon completion of training the base model with the entire dataset ${\mathit{D}}_{\mathit{K}}$, the model is subsequently employed as the Teacher model to train Student model. In other words, we aim to verify performance improvement when training the model under the same conditions. If performance enhancement is validated at this method, it suggests that superior performing models can be trained under identical learning conditions, even when employing larger or state-of-the-art (SOTA) models.

The training images are constructed considering the problems of existing deepfake detection. While deepfakes by known generative models exist in $\mathit{K}$, deepfake images by unknown generative models also exist in $\mathit{U}$. Therefore, during training phase, we enhance the deepfake detection performance in $\mathit{K}$ using labeled data and contribute to the generalization of the learning model by using data ${\mathit{D}}_{\mathit{K}}$ and ${\mathit{D}}_{\mathit{U}}$ from $\mathit{K}$ and $\mathit{U}$ as unlabeled data, respectively. Consistent with this approach, the data $D_L$ and $D_{UL}$ are structured into ${\mathit{D}}_{\mathit{K}}$, with images from dataset ${\mathit{D}}_{\mathit{U}}$ serving as $D_{UL}$.

In the proposed method, we combine data in three strategies to detect the Unknown Domain Dataset ${\mathit{D}}_{\mathit{U}}$. The first strategy is to use the data from $D_L$ and $D_{UL}$ as the same domain, aiming to verify whether unlabeled data from a specific $\mathit{K}$ contributes to the improvement of model performance. Figure 2a illustrates the training strategy of using ${\mathit{D}}_{\mathit{K}}$ as unlabeled data. The second strategy aims to solve the realistic deepfake problem by experimenting with the impact of unlabeled data on the detection performance of the corresponding domain. Figure 2b illustrates the feasibility of improving model performance by employing dataset ${\mathit{D}}_{\mathit{K}}$ as labeled data $D_L$ and dataset ${\mathit{D}}_{\mathit{U}}$ as unlabeled data $D_{UL}$. Finally, in the deepfake scenario experiment, after training the model using the first-generation deepfake dataset as $D_L$ and $D_{UL}$, the generalized deepfake detection model learning is assessed through the first-generation ${\mathit{D}}_{\mathit{K}}$ and the first and second-generation ${\mathit{D}}_{\mathit{U}}$.

3.2. SupCon-MPL: Supervised Contrastive Learning with Meta-Pseudo Labels

In the proposed method, following the strategy in Figure 2b, the MPL model is trained for the detection of deepfakes in the Unknown Domain $\mathit{U}$. SupCon-MPL allows supplementary training utilizing unlabeled videos, and with the aid of CRL, it enhances the deepfake detection in feature space. Furthermore, it affords the flexibility to employ diverse encoder models during the training phase and enables fine-tuning of the SupCon-MPL-trained model.

In particular, the limitations of deepfake detection with limited labeled data can be mitigated by using unlabeled data, and a generalized detection model can be trained through CRL. Another notable advantage lies in the capability to conduct concurrent learning via feedback from the student model, even if the performance of the teacher model is low. The details of the proposed method are elucidated in Figure 3. The most significant distinction from the conventional MPL and SupCon model training is that with learning through unlabeled data not only resolves the training issue of CRL due to limited data but also enhances detection capabilities in both $\mathit{K}$ and $\mathit{U}$. Ultimately, the final goal is to enhance the detection capabilities of deepfakes in domains that are not targeted, especially in a situation where new deepfake models in $\mathit{U}$ continue to be developed.

The training of the SupCon-MPL is conducted by first having the student model $T$ use unlabeled data to perform CRL, followed by fine-tuning with labeled data. In this process, the teacher model $T$'s classifier learns through the feedback from $S$, while $S$ learns dependently on $T$.

3.3. SupCon-MPL Loss Function

SupCon-MPL, as shown in Figure 3, is composed of a teacher model ($T$) and a student model (S), each of which consists of an encoder and a linear classifier. SupCon-MPL has two loss functions in order to sequentially train each model. One involves the Teacher model $T$ distilling knowledge to the student model $S$, while the other entails the Teacher model $T$ training from the feedback factor provided by $S$ on the labeled data. The knowledge distilled by $T$ includes previously learned content about deepfakes.

In SupCon-MPL, let the parameters of $T$ classifier and $S$ classifier be ${\theta }_T$, ${\theta }_S$ respectively, and denote the batch of images and labels on the labeled data as $\left(x_l,y_l\right)\in D_K$, and the batch of images on the unlabeled data as $x_u\in D_U$. The goal of SupCon-MPL is to minimize the parameters ${\theta }_S^{PL}$ of the generalized deepfake detection model $S$.

$${\theta }_S^{PL}=\underset{{\theta }_S}{\mathit{argmi}\mathrm{n}}\underset{L_u≔\left({\theta }_T,{\theta }_S\right)}{\underbrace{E_{x_u}\left[CE\left(T\left(x_u;{\theta }_T\right),S\left(x_u;{\theta }_S\right)\right)\right]}}$$

(1)

Hence, the objective function of SupCon-MPL is defined as follows.

$$\begin{array}{c}{L}_l\mathrm{ }\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{ }\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{ }\mathrm{t}\mathrm{o}\mathrm{ }{\theta }_T:\\ \underset{{\mathsf{\theta }}_{\mathrm{T}}}{\min }\mathrm{ }{L}_l\left({\theta }_S^{PL}\left({\theta }_T\right)\right),\\ \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{ }{\theta }_S^{PL}\left({\theta }_T\right)=\underset{{\theta }_S}{\mathrm{argmin}}\mathrm{ }{L}_u\left({\theta }_T,{\theta }_S\right).\end{array}$$

(2)

For optimization, SupCon-MPL approximates ${\theta }_S^{PL}\left({\theta }_T\right)$ by the learning rate $\mathsf{\eta }s$, and then,

$${\theta }_S^{PL}\left({\theta }_T\right)\approx {\theta }_S-\eta s\cdot {\nabla }_{{\theta }_S}{L}_u\left({\theta }_T,{\theta }_S\right)$$

(3)

defines the final objective function as follows.

$$\begin{array}{c}{L}_l\mathrm{ }\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{ }\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{ }\mathrm{t}\mathrm{o}\mathrm{ }{\mathsf{\theta }}_T:\\ \underset{{\mathsf{\theta }}_T}{\min }\mathrm{ }{L}_l\left({\theta }_S-\eta s\cdot {\nabla }_{{\theta }_S}{L}_u\left({\theta }_T,{\theta }_S\right)\right),\\ \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{ }{\theta }_S^{PL}\left({\theta }_T\right)=\underset{{\mathsf{\theta }}_S}{\mathrm{argmin}}\mathrm{ }{L}_u\left({\mathsf{\theta }}_T,{\mathsf{\theta }}_S\right).\end{array}$$

(4)

Both $T$ and $S$ consist of an encoder and a classifier, and are trained according to their respective loss functions. The loss function of $EN{C}_T$, the encoder of $T$, is composed of SupConLoss [38], and the loss function of $Cl{F}_T$, the classifier, is composed of Labeled Loss for the labeled data $x_l$ and MPL Loss reflecting the feedback from $S$. First and foremost, $g_{T,contrast}^{\left(t\right)}$the loss function of $EN{C}_T$, receives image pairs $\left(RandAugmen{t}_a\left(x_l\right),RandAugmen{t}_b\left(x_l\right),y_l\right)$ as inputs that reflect different random augmentations on the same image $x_l$ and the label $y_l$. Subsequently, the loss value is obtained by passing image pairs through SupConLoss [38]. At this juncture, given the similarity between the current training process and that of the original MPL's UDA Loss, the utilization of the UDA Loss is no more continued.

$$g_{T,contrast}^{\left(t\right)}={\nabla }_{{\theta }_T}SupConLoss\left(RandAugmen{t}_a\left(x_l\right),RandAugmen{t}_b\left(x_l\right),y_l\right){\left.|\right|}_{{\theta }_T={\theta }_T^{\left(t\right)}}$$

(5)

$EN{C}_T$ is promptly updated following the computation of the $g_{T,contrast}^{\left(t\right)}$.

$${\mathsf{\theta }}_{T,ENC}^{\left(t+1\right)}={\mathsf{\theta }}_{T,ENC}^{\left(t\right)}-{\mathsf{\eta }}_S\cdot g_{T,contrast}^{\left(t\right)}$$

(6)

The Labeled Loss of $Cl{F}_T$, $g_{T,supervised}^{\left(t\right)}$, measures the difference between $y_l$ and the label predicted by $T$ through Crossentropy Loss (CE Loss). Here, $em{b}_l^T$ denotes the embedding value derived by passing the labeled data $x_l$ through $EN{C}_T$.

$$g_{T,supervised}^{\left(t\right)}={\nabla }_{{\theta }_T}CE\left(y_l,CL{F}_T\left(em{b}_l^T;{\theta }_T\right)\right){|}_{{\theta }_T={\theta }_T^{\left(t\right)}}$$

(7)

The MPL Loss $g_T^{\left(t\right)}$ calculates the difference between the hard pseudo label $y_u$, which is the maximum value extracted from the pseudo labels generated by $T$ through $x_u$, and the logit. Here, $em{b}_u^T$ denotes the embedding value derived by passing the labeled data $x_u$ through $EN{C}_T$.

$$g_T^{\left(t\right)}=h\cdot {\nabla }_{{\theta }_T}CE\left({\widehat{y}}_u,CL{F}_T\left(em{b}_u^T;{\theta }_T\right)\right){|}_{{\theta }_T={\theta }_T^{\left(t\right)}}$$

(8)

The feedback factor $h$ of $S$ was calculated in the same way as the original Meta Pseudo Labels [37], using Taylor Expansion to calculate the difference before and after the training of $S$. In the proposed method, we approximated $h$ using the difference from the CE Loss value for the labeled data after $S$ was trained to the value before training. This allows the final loss value to converge as the training progresses.

$$h=CE\left(y_l,S\left(x_l;{\theta }_S^{\left(t+1\right)}\right)\right)-CE\left(y_l,S\left(x_l;{\theta }_S^{\left(t\right)}\right)\right)$$

(9)

The final loss function of $Cl{F}_T$, is composed of the sum of each loss function value.

$${\theta }_T^{\left(t+1\right)}={\theta }_T^{\left(t\right)}-\eta s\cdot \left(g_T^{\left(t\right)}+g_{T,supervised}^{\left(t\right)}\right)$$

(10)

$S$ is trained through unlabeled data. The loss function of student model’s encoder $EN{C}_S$, denoted as $g_{S,contrast}^{\left(t\right)}$, is trained utilizing SupConLoss [38], akin to $EN{C}_T$. It leverages $\left(x_u,{\widehat{y}}_u\right)$, comprising an unlabeled image $x_u$ paired with pseudo labels ${\widehat{y}}_u$, generated by $T$.

$$g_{S,contrast}^{\left(t\right)}={\nabla }_{{\theta }_T}SupConLoss\left(RandAugmen{t}_a\left(x_u\right),RandAugmen{t}_b\left(x_u\right),{\widehat{y}}_u\right){\left.|\right|}_{{\theta }_S={\theta }_S^{\left(t\right)}}$$

(11)

$EN{C}_S$ is also promptly updated following the computation of the $g_{S,contrast}^{\left(t\right)}$

$${\theta }_{S,ENC}^{\left(t+1\right)}={\theta }_{S,ENC}^{\left(t\right)}-{\eta }_S\cdot g_{S,contrast}^{\left(t\right)}$$

(12)

The loss function of $CL{F}_S$ is calculated using CE Loss for the Hard Pseudo Label ${\widehat{y}}_u$ of $T$ for $x_u$ and the prediction of $S$. Here, $em{b}_u^S$ denotes the embedding value derived by passing the labeled data $x_u$ through $EN{C}_S$.

$${\theta }_S^{\left(t+1\right)}={\theta }_S^{\left(t\right)}-\eta s\cdot {\nabla }_{{\theta }_S}CE\left({\widehat{y}}_u,CL{F}_S\left(em{b}_u^S;{\theta }_S\right)\right){|}_{{\theta }_S={\theta }_S^{\left(t\right)}}$$

(13)

The SupConLoss in $g_{T,contrast}^{\left(t\right)}$, $g_{S,contrast}^{\left(t\right)}$on the Teacher model $T$ and Student model $S$ are as follows.

$$SupConLoss={\sum }_{i\in I}SupConLos{s}_i={\sum }_{i\in I}\frac{-1}{\left|P\left(i\right)\right|}{\sum }_{p\in P\left(i\right)}log\frac{exp\left(z_i\cdot z_p/\tau \right)}{{\sum }_{a\in A\left(i\right)}exp\left(z_i\cdot z_a/\tau \right)}$$

(14)

Here, $i\in I\equiv \{1,\cdot \cdot \cdot ,2N\}$ is the index of the randomly augmented data, and $P\left(i\right)\equiv \{p\in A\left(i\right):y_p=y_i\}$ is the set of indices for all positives in the batch (Since $y_p$ and $y_i$ are labels of images that have been randomly augmented from $y_l$, they are the same as $y_l$). ${\mathrm{z}}_{\mathrm{i}}$ and $z_p$ are the embedding values of each randomly augmented image in $\left(RandAugmen{t}_a\left(x_l\right),RandAugmen{t}_b\left(x_l\right)\right)$ passed through the encoder $ENC$, and $A\left(i\right)\equiv I\backslash \left\{i\right\}$, and $\mathsf{\tau }$ is the temperature parameter. In other words, the inner product between positive pairs ($i$ and $p$ are the same class but different samples) is maximized through $exp\left(z_i\cdot z_p/\tau \right)$, and the inner product between negative pairs is minimized through $exp\left(z_i\cdot z_a/\tau \right)$, so that the SupConLoss is minimized.

The training process of the proposed SupCon-MPL model for deepfake detection is as shown in Figure 4.

4. Experiment

4.1. Experiment Setup

The experiments are conducted using NVIDIA Tesla V100 32 and NVIDIA RTX-3090. The single-domain experiment and the multi-domain experiment are existing outputs of the Meta Learning-based Deepfake Detection Project [24]. The Pretrained model, Meta Pseudo Labels model (MPL model), SupCon model, and SupCon-MPL model are experimented for their training performance under the same conditions and hyperparameters. The training dataset uses the videos of the Deepfakes(DF), Face2Face(F2F), FaceSwap(FS), NeuralTextures(NT), and Real videos in FaceForensics++ [39], with DFDC [40], and Celeb-DF [41]. In scenario evaluation, deepfake videos of first-generation’s unknown domain are NeuralTextures(NT)[39] with Real videos, and for post-generation’s unknown domain, we selected StyleGAN [8] images with CelebA [48] videos. We used MTCNN [49] to extract face images frame by frame of Each video.

In the single-domain experiment, 260,000 real and 340,000 fake data from each domain in the FaceForensics++ [39] are used. During training, the amount of validation data used is 20% of the training data, and the evaluation dataset uses 150,000 per each data domain. In the multi-domain experiment, 200,000 labeled data are randomly extracted from 4 domains, and 180,000 unlabeled data are extracted from a single domain for use.

The generational deepfake scenario trains using 170,000 each of the 1st generation known domain's FaceForensics++ (DF, F2F, FS, Real) [39], DFDC[40], Celeb-DF [41] data, and then evaluates using 51,200 each of 1st and 2nd generation Unknown Domain data. Backbone model in scenario evaluation, we used ResNet50 [50] due to lack of computational resources.

The hyperparameters used in the experiment are a learning rate of 1e-4, an image size of 64, and a batch size of 512. In the MPL, SupConMPL models, the batch sizes of labeled and unlabeled images are 64 and 448, respectively. Finally, the threshold is used at 0.95. The training models used were ResNet50 [50], ResNet101 [50], ResNext50 [51], EfficientNet-b5 [21], and WideResNet50 [52].

Experiment data and evaluation data use a mix of fake and real data. In the experiment in Section 4.2.1, video data from one domain is used as labeled and unlabeled data, and in the experiment in Section 4.2.2, videos from multiple domains are used as labeled data, and one domain is used as unlabeled data.

4.2. Single-domain Experiment

In the experiment using only one domain, as shown in Table 1, the performance of the known domain increased in most of situation. Furthermore, it was confirmed that the performance in unknown domains also increased in most of situation. Based on EfficientNet-b5 [21] in Table 1, ACC and AUC in K improved by an average of 4.47% and 4.53% respectively, and ACC in U improved by an average of 0.20%, but AUC decreased by an average of 0.13%. However, overall, out of a total of 64 ACC and AUC validations recorded in Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8, 44 and 41 case improved, respectively. Through this, it was confirmed that when using the same data, the performance of the MPL model is higher than the pretrained model.

4.3. Multi-domain Experiment

In the multi-domain experiment, the combination of data is configured considering the actual deepfake situation. The situation is assumed to have $\mathit{K}$ deepfake data from multiple domains and $D_{UL}$ data from $\mathit{U}$. Afterwards, the evaluation is conducted through the $\mathit{U}$ data used as $D_{UL}$. Therefore, MPL trains with deepfake videos from multiple domains, and after training, MPL experiments the ability to detect unlabeled data using unlabeled data. As a result of the experiment, ACC and AUC increased by an average of 1.59%, 1.26% in 2 models in Table 9 and Table 10. Through this, it was confirmed that the MPL model improved the deepfake detection ability of the unknown model by learning with unlabeled data.

4.4. SupCon-MPL Experiment

The experiment uses the labeled data identically to the multi-domain experiment and evaluates according to each combination of known domain and unknown domain. At this time, Celeb-DF [41] is used as unlabeled data in the training of the SupCon-MPL model. As a result of the experiment, shown in Table 11, when evaluating FS data as an Unknown domain compared to the SupCon model [35], ACC and AUC decreased, but in other validations, the performance of the SupCon-MPL model was similar or higher than the performance of the two models being compared. This shows that the SupCon-MPL model has been trained as a generalized detection model compared to the existing deepfake detection model.

4.5. Deepfake Scenario Experiment

In this section, we construct a training scenario for a deepfake detection model in the real world and train the model. The scenario involves training a deepfake model with first-generation deepfake data, then experiment with the detection of first-generation deepfake (NT) that was not participated while training, and post-generation(second-generation) deepfakes (StyleGAN) that are newly developed and unknown. Table 12 shows the training results of various models according to the scenario. As a result of the experiment, among various models, the SupCon-MPL model achieved highest performance in all scenario evaluations compared to other models.

4.5. Limitations

The main goal of the SupCon-MPL is to enhance the deepfake detection performance in unknown domains using meta-learning. Therefore, in this paper, we conducted experiments by reconfiguring a limited deepfake dataset into scenarios.

The main limitation is related to computing resources. As the training in Section 4.3. and Section 4.4. was conducted using NVIDIA RTX-3090, only ResNet50 [50] could be used as the backbone model in SupCon-MPL, which uses two models. Subsequent experiments are needed with various backbone models and larger image sizes through more computing resources.

The next limitation is that we could not find a verified deepfake dataset for the third generation and higher. Further experiments are needed through the corresponding dataset in the future.

5. Conclusion

With the development of various deepfake generative models, it has become important to develop a generalized deepfake detection model that guarantees the detection performance of unknown domain deepfakes, not just the data of the domain used for training. The proposed SupCon-MPL improved all detection performances over other model's known and unknown domains in scenario evaluations by utilizing unlabeled deepfake images/videos and Contrastive Learning. Indeed, one of the significant features of SupCon-MPL is its ability to train models using a large amount of unlabeled data. This provides a method to enhance the model's performance utilizing the countless images and videos available on the internet and so on.

Future research will focus on improving the detection performance of higher generation deepfake images/videos using these methods. Additionally, studies on reducing the training cost of SupCon-MPL will be conducted. The goal is to develop a more efficient and economical deepfake detection model through these efforts.