Detection of Forged Images Using a Combination of Passive Methods Based on Neural Networks

1. Introduction

Fake news is the term used to describe false information disseminated on social networks. This type of news can deceive viewers and be shared more often than factual news since it can be specially crafted to elicit a strong reaction [1]. An example of false information confusing the population explored by Higgs Boson in 2012, which generated as many as 600 tweets per minute [2]. The use of tampered images to support fake news is particularly troublesome for its capacity to implicitly give a sense of veracity to information [3], harming public trust [4].

Most of the papers analyzed in the related works section use datasets that do not have images with copy-move or splicing tampering. However, other papers use more specific types of manipulation; for example, Liu and Pun [5] mentioned a manipulation able to remove parts of an image and replace it with surroundings, and Rocha et al. [6] mentioned healing as a manipulation capable of softening features.

This paper uses the nomenclature of manipulations used by [7,8], which in addition to copy-move and splicing, acknowledges the existence of retouching as a manipulation to slightly alter the content of an image without totally hiding big patches of information, this includes filters, highlights, resizing or rotating parts of an image.

With the intention to present a method capable of generalizing to more types of manipulation, as is the case in images present in social media, but without losing accuracy, this paper proposes to combine two detection methods to guide a neural network to estimate the authenticity of an image. Moreover, this paper analyzes the results with a dataset containing retouching manipulation to better estimate the accuracy in real-world applications.

There are two general approaches to identifying image manipulation, active and passive. Active techniques, also known as preemptive, aim to preemptively insert structures in an image that can be used to detect any future changes. These structures can be visible to the human eye, like in watermarking, or hidden using steganography techniques. Manipulation can be reliably detected by verifying the integrity of the inserted structure; however, those methods only allow modification detection after the structure has been inserted and therefore need a trustworthy image capture that inserts the structure correctly before any manipulation [9].

Meanwhile, passive methods, also known as blind methods, use only the content already present in an image and do not require any prior action, making them better suited for use in social media. To detect image forgery passive methods can either analyze artifacts intrinsic to a digital image or search for inconsistencies in the content of an image. According to Lubna and Chowdhury [9] the artifacts present in digital images can be divided into three types, acquisition, format, and manipulation.

Acquisition artifacts are introduced in the image either by imperfections created by the manufacture of camera sensors called fixed-pattern noise or by programs used by cameras to process sensor data before storage, the introduction of artifacts in this manner usually follows a predictable pattern and any divergences to it is evidence of manipulation.

Format artifacts, introduced in the digital storage of images by algorithms like the jpeg compression, by removing information less relevant to human eyes, areas of the image with artifacts of this type different than expected indicate the image was manipulated. Manipulation artifacts, that get introduced during image manipulation by a program, like the application of a blur filter that alters areas of an image in predictable patterns, the presence of those known artifacts in areas of an image is evidence of manipulation.

The other possible approach involves analyzing the content of an image for inconsistencies. Some inconsistencies are duplicated portions of an image, unnaturally sharp edges, shadow inconsistencies, and perspective inconsistencies [10]. Unfortunately, all those methods of detection have their pros and cons. There is no absolute best method capable of detecting all types of manipulation. Therefore, to better understand detection methods and their limitations, it is essential to understand how images can be manipulated in the first place [11].

There has yet to be a consensus on classifying all image manipulations; however, most papers recognize the existence of at least two types, copy-move, and splicing. In copy-move forgery, an area of an image is duplicated and pasted over another area of the same image. In splicing, however, an area of another image is pasted over the image. This manipulation can add new meaning to the image or conceal information; the resulting image is a fusion of two different images.

Besides this introductory section, the following sections aim to explain better how the proposed approach works; in Section 2, we discuss a few related works. Section 3 presents the methods, the dataset used, and the model architecture, then in Section 4 we showcase the results we obtained and try to explain them. Finally, in Section 5, we presented the final considerations and elaborate on how this work could be expanded.

2. Related Works

A problem faced in the definition of the search string consisted of the significant variability of terms used to refer to the addressed subject. So some terms were tested. For example, terms sometimes used to refer to counterfeits are "falsification", "tampering", "counterfeiting", "adulteration", "forgery", "manipulation", "edited", "doctored" or "altered". After tests the following search string was defined: "image AND (tamper OR forged OR forgery) AND (detect OR localize) AND NOT video". This search returned a considerable amount of work. To delimit the search, using the first search string we adopted the following criteria:

• The presented detection method must follow the passive approach;
• The Detection method should primarily focus on verifying digital images’ authenticity;
• Must be in the top five most relevant results of each base that follows the other criteria;

With this research, we find 15 works. However, most of the papers returned on this initial search did not claim to be capable of detecting all types of manipulation and mostly focused on a single type, therefore, to include terms related to general identification, terms such as "global" or "universal" were tested. Unfortunately, not all works that detect all types of manipulation used those terms, so the search key had to be made less specific: "image AND (tamper OR forged OR forgery) AND (detect OR localize)" and to limit the search the following criteria were employed:

• Introduce a method of detecting general-purpose manipulated images in their text;
• The presented detection method must follow the passive approach;
• The presented method should not require file formats with data compression;
• The Detection method should primarily focus on verifying digital images’ authenticity;
• paper was published in the last five years;
• Must be the most relevant results of each base that follows the other criteria.

This search returned three more works, for a total of 18 works. The works returned have different functioning particularities in their detection approach. Therefore, we divided them according to their specific characteristics stages in the: detected manipulations; color space; feature extraction; and detection method, the comparative analysis can be found in Table 1.

The detection methods found can be categorized as classifiers capable of determining alterations in JPEG artifacts, classifiers to detect duplicated areas, and methods that compare detected features to determine correlation or fixed pattern noise (FPN).

To understand methods based on fixed pattern noise, it is first necessary to understand the image capture process. Image acquisition aims to transform an image into a discrete and numerical representation that can be stored and processed by a computer. This requires a sensor capable of capturing a range of energy from the electromagnetic spectrum and generating as output an electric signal proportional to the captured energy level, then a digitizer must convert the analog signal into digital information that can be represented in binary form.

The manufacturing process of most sensors responsible for capturing images in digital cameras introduces imperfections that cause small differences in light sensitivity [30] The divergences of all the sensors present in a camera introduce a variation in the values of the pixels of images registered by these cameras, resulting in unevenness similar to a signature in all the images generated by it [31].

The FPN of a sensor is constant, however, it varies from sensor to sensor. In Sensors of the Charged-Coupled Device (CCD) type, the FPN varies randomly while in sensors of the Complementary Metal Oxide Semiconductor (CMOS) type, due to its perpendicular capture system, the FPN forms vertical bars, as can be seen in Figure 1 taken from [32].

Virtually all sensors, including the two popular examples presented earlier, produce patterns in their images that can be used for image manipulation detection. This detection is carried out by identifying areas that do not present the pattern correctly, so before detection, it is necessary to first determine the FPN produced by the camera that captured the image, a possible approach for this is to take the average of several images captured by the same camera [30].

The work by [33] assumes that the FPN information of the camera used for capture is previously known and then calculates the FPN of the image for validation by comparing the two and marking significantly different areas as being possibly altered.

On the other hand, methods based on classifiers use filtered features to determine if there is a correlation between parts of the image or alteration of the compression through machine learning [34]. In deep learning [35,36,37], convolutional neural networks (CNNs) are increasingly being used for image classification [38,39,40].

In this field, several authors are working to reduce the complexity of the classification considering the use of big data [41,42,43] and make this evaluation more efficient [44,45,46] Finally, correlation-based methods compare a set of features by similarity to determine whether they contain Duplication, often using a final step to eliminate a portion of the found correlations. In this context unsupervised learning methods are also applied [47].

All datasets found in the literature review focus on detecting Duplication and Splicing manipulations. Their most significant divergences are the images used for alteration, the size of the altered area, and the application of subsequent modifications to hide falsification on the manipulated areas. The work by [48], which is the only one focused on detecting manipulation retouching, had to create its dataset for validation. The datasets used by analyzed works focused only on the detection of Duplication type are MICC-F220, MICC-F600, MICC-F, MICC-F2000, MICC-F8multi, IMD, CMH, CoMoFoD, and UCID.

The datasets containing Duplication and Splicing manipulations used were: CASIA v1.0, CASIA v2.0, Columbia, and Image Manipulation Database, Columbia and CASIA v1.0 datasets. Finally, the Image Manipulation Database contains a set of unaltered images and software to perform Duplication and Splicing manipulations, as well as Retouch under the areas, to be altered to hide modifications automatically, so the final adulterated images of the dataset may vary depending on usage of the software.

The problem in comparing methods for passively detecting falsified images was the limitations found in the literature. For example, some methods only detect one type of manipulation, others only work on specific types of image files, and some require or assume additional image information, such as the FPN of the camera used in the capture.

In addition to these limitations of scope, another impacting factor is the robustness of the methods; performing multiple manipulations on images tends to affect the Accuracy of different methods in different ways. This brings another problem: many datasets are available for testing with different focuses, even when trying to detect a specific type of manipulation, in addition to creating new datasets by some authors to evaluate their results.

From the comparison of the works analyzed we conclude that there is no consensus regarding ideal methods of feature extraction and image authenticity classification. Multiple approaches are possible even for identical types of manipulation, such as methods with feature extraction blocks and key points being successfully employed by different authors.

In addition, the lack of a standard dataset in all these works makes it difficult to compare the effectiveness of all methods using only the Accuracy reported by the authors, demonstrating the importance of the present work.

The problem in comparing methods for passively detecting falsified images was the limitations found in the literature. For example, some methods only detect one type of manipulation, others only work on specific types of image files, and some require or assume additional image information, such as the FPN of the camera used in the capture.

In addition to these limitations of scope, another impacting factor is the robustness of the methods. For example, multiple manipulations of images affect the Accuracy of different methods in different ways. Another problem is the number of datasets available for testing with different focuses, even when trying to detect a specific type of manipulation.

From the comparison of the works analyzed, we conclude that there needs to be more consensus in the literature regarding ideal methods of feature extraction and image authenticity classification.

In addition, the lack of a standard dataset in all these works makes it difficult to compare the effectiveness of all methods using only the Accuracy reported by the authors, demonstrating the importance of the present work.

3. Materials and Methods

3.1. Error-Level Analysis

Error-Level Analysis is a passive detection method traditionally used by human forensics specialists to make differences in format artifacts of jpeg images more evident, it works by taking the difference between a jpeg image at different quality levels, making any difference in compression rate more evident on the resulting image [48], an example of this process can be seen in the Figure 2 and the code used to generate the images is presented in Listing 1.

Listing 1. Error-Level analysis implementation in pseudocode

# Static method that performs Error Level Analysis (ELA) on an image using JPEG compression. # Returns a normalized difference image between the original image and a JPEG-compressed version of the image. FUNCTION method_1_ela(image, quality)     # Create another image with jpeg compression of the given quality     save_image_as_jpeg(image, temp_image, quality)     compressed_image = open_image(temp_image)       # Calculate the image difference between the original and the JPEG-compressed image     difference_image = image - compressed_image       # Normalize the difference image for contrast by assigning a value of 255 to the brightest points, while proportionally adjusting the values of all other points based on their distance from the brightest point.     normalized_difference  = difference_image.normalizeContrast()     RETURN normalized_difference END FUNCTION

In this example, the resulting image on the bottom is mostly black, but edited areas have more color in them, however, ELA results aren’t always so easily interpreted and traditionally need a forensics specialist to look at the results, however, this paper proposed using ELA as a feature extraction step and feeding it to a Convolutional Neural Network in order to perform the authenticity analysis of an image automatically.

3.2. Discrete Wavelet Transform

Discrete Wavelet Transform (DWT) stands for denoising [49], which is a mathematical technique used for analyzing signals that can be applied to images [50]. It is a way of decomposing an image into a set of frequency components, with each component representing a different level of detail or resolution [51].

In image manipulation detection, DWT is often used for feature selection as it allows for efficient compression of image data while preserving important image features. The DWT algorithm works by dividing an image into four smaller blocks or "sub-bands" of different frequencies: The LL (low-low) sub-band, which contains the low-frequency information, and the LH (low-high), HL (high-low), and HH (high-high) sub-bands, which contain the high-frequency information [52].

The equations to perform DWT can be found in Eq. 1 and Eq. 2 and the equation to reverse the process known as inverse DWT is presented in Eq. 3.

$$W_{\phi }(j_0,k)=\frac{1}{\sqrt[]{M}}\sum _M^{}f\left(x\right){\phi }_{j_0,k}\left(x\right)$$

(1)

$$W_{\psi }(j,k)=\frac{1}{\sqrt{M}}\sum _{k}f\left(x\right){\psi }_{j,k}\left(x\right)$$

(2)

$$\begin{array}{cc}\hfill f\left(x\right)=& \frac{1}{\sqrt{M}}\sum _k{W}_{\phi }\left(j_0,k\right){\phi }_{j_0,k}\left(x\right)\hfill \\ \hfill & +\frac{1}{\sqrt{M}}\sum _{i=i_0}^{\infty }\sum _k{W}_{\psi }(j,k){\psi }_{j,k}\left(x\right).\hfill \end{array}$$

(3)

This work uses the technique proposed by [53], referred to in the rest of this work as DWT method for abbreviation purposes, which makes use of DWT, Bilateral Filters, and the Laplace operator to remove less meaningful features of an image, resulting in an image only with features containing sharp pixel variation, which is a common indicator of forgery.

This technique works by initially the image is converted to grayscale, then DWT is applied to decompose the image information into sub-bands, then the image is reconstructed after discarding the LL band and a bilateral filter, and the Laplace operator is applied, the result is an image where sharp pixel transitions are more easily visible, however when filters are used to mask the manipulation this method fails to highlight the manipulated areas. This work uses the image resulting from this method as a feature selection step for a Convolutional Neural Network to teach it to better detect large pixel transitions.

An example of the result of this process can be seen in Figure 3 that was taken from the RTD and the code used to generate the images is presented in Listing 2.

Listing 2. DWT based method implementation in pseudocode

FUNCTION method_2_dwt(image)     # Convert image to grayscale and perform discrete wavelet transform     gray_image = convertToGrayscale(image)     coeffs = discreteWaveletTransform(gray_image)     (LL, (LH, HL, HH)) = coeffs       # Reconstruct the image using only the high-frequency components     high_freq_components = (None, (LH, HL, HH))     joinedLhHlHh = inverseDiscreteWaveletTransform(high_freq_components)       # Apply bilateral filter to smooth the image while preserving edges     blurred = bilateralFilter(joinedLhHlHh, 9, 75, 75)       # Apply Laplacian edge detection to highlight edges     kernel_size = 3     imgLapacian = laplacianEdgeDetection(blurred, kernel_size)       # Convert negative values to zero     final_image = convertScaleToAbs(imgLapacian)       RETURN final_image END FUNCTION

3.3. Proposed Method

The proposed approach first consists of applying the ELA method and the method proposed by [53], both used as a feature selection step to generate two extra sets of images.

Next, the original dataset and the two new sets of images are shuffled and used for training and evaluation of three different CNNs; the three models are then frozen to preserve the acquired knowledge and merged. Finally, we added two new layers and trained them to combine the knowledge of the three models. These steps are summarized in Figure 4, showing the overview of the system.

This approach aims to explore the Accuracy of each method individually in addition to their combination. For this, we individually trained three distinct models and a model composed of all three, plus additional learning layers. Model A uses only the original images without alterations as input, Model B uses only images with feature selection by ELA, and Model C uses only images with feature selection by the DWT-based method. Finally, the models are combined and four layers are added to generate the Merged model, which uses images of all three types for input.

Initially, we considered the use of pre-trained models for combining the passive methods in this research. To get an initial idea of their performance concerning custom models the MobileNetV2 pre-trained model was selected for initial tests for being considered lightweight with other pre-trained models. A comparison of model performance between model "A" detailed later and MobileNetV2 plus four dense layers with l2 regularization, each followed by a batch normalization layer can be seen in the Figure 5 and Figure 6.

Those tests show a tendency of pre-trained models to over-fit to the training images, unfreezing some of the pre-trained layers was also attempted with similar results, the use of another pre-trained model in InceptionResNetV2 also did not yield better results than the custom model, this leads the researchers to believe pre-trained are not a good fit for image manipulation detection.

With that in mind, we decided to create three identical custom CNNs and implement regularization methods to minimize overfitting to produce results capable of generalizing to a more extensive set of images and facilitating the comparison of the three individual models.

The primary challenge in developing the neural networks for this study was addressing the issue of overfitting, where the models tend to perform well on training data but struggle to generalize to new, unseen data. To mitigate this, various experiments were conducted to assess the impact of different model architectures on accuracy.

After testing multiple models with numerous layers, it was observed that increasing the number of layers did not significantly improve accuracy. Instead, it led to a higher risk of overfitting. As a result, a decision was made to design a model that consists of five convolutional layers, each followed by a batch normalization layer and a max pooling layer. The outputs from these layers were then flattened and passed through four dense layers, each accompanied by a batch normalization layer. Furthermore, L2 regularization was applied to both the convolutional and dense layers to prevent overfitting. Figure 7 provides a visual representation of this model architecture.

The values used for the size of convolutional layers are 16,32,64,128,64 all with a 3 by 3 kernel, and the values for the dense layers are 128,128,64,32.

The final merged model that is proposed here consists of merging the individual models, initially, models A and B last non-output layer are combined through a concatenation layer, and then this combination is further merged with model C last non-output layer through another concatenated layer, then two dense layers with 64 neurons were added for learning, each followed by a batch normalization layer, all layers of the individual models were frozen to preserve the knowledge they acquired during training.

The use of Dropout layers was explored, however, Batch Normalization had a better performance given the same training time [54]. The optimizer used was Adam, the metric during training was accuracy, and the Loss function used was Binary Cross Entropy. The activation function used was Rectified Linear Unit in all layers except in output, where we use the sigmoid Function.

3.4. Dataset Assembly

The first step to executing the experiments we carried out was assembling the final dataset, to accomplish this, we gathered authentic and manipulated images from four different datasets as explained below:

• CASIA V2.0: proposed in [55], contains 7491 authentic images and 5123 manipulated images containing Splicing and or Duplication operations with retouching operations applied on top to mask alterations;
• Realistic Tampering Dataset: Proposed by [56,57] containing 220 authentic and 220 Splicing and or Duplication manipulations made to the original images with the objective of being realistic. Retouching operations are sometimes applied to help hide Compositing and Duplication manipulations. In addition, this dataset provides masks of tampered areas and information about capture devices used;
• IMD2020: Proposed by [58], it consists of four parts, first a dataset containing 80 authentic images manipulated to generate 1930 images tampered realistically and using all types of manipulation, with their respective manipulation masks. Then the second part consists of 35,000 authentic images captured by 2,322 different camera models, the images were collected online and reviewed manually by the authors. The third has 35,000 algorithmically generated images with retouching manipulations. Finally, the last part has 2759 authentic images acquired by the authors with 19 different camera models designed for sensor noise analysis;
• CASIA V1.0: Proposed in [55], Contains 800 authentic images, 459 Duplicate-type manipulation images, and 462 Splicing images. This dataset has no retouching operations applied.

To balance the dataset and use images manipulated realistically, we separated the images into two folders. The first contains 7491 authentic images from the CASIA V2.0 dataset. The second with manipulated images contains 7491 tampered images taken from the part from the IMD2020 dataset with Realistic Images, Realistic Tampering Dataset (RTD), CASIA V2.0, and 218 images selected from CASIA V1.0 to balance the dataset. Table 2 shows the distribution of images used from each dataset.

Therefore, as explained in Table 2 the final dataset used in this paper consists of 14982 images in total, half original and half Manipulated, since some of the images were not supported by model all images were converted to .jpg format.

The second step of the final program is to apply the methods described to the dataset, as both methods generate an image as output. The result is two new sets of images with specific inconsistencies enhanced, using methods from the TensorFlow library whenever applicable.

The dataset was divided into three parts: 70% for training, 20% for validation, and 10% for testing. Each image was resized to 224 by 224 pixels to match the input size required by MobileNetV2, a pre-trained neural network that was initially considered but ultimately not used in favor of a custom model due to improved performance.

The experiment consisted of first creating two additional sets of images using the two selected passive methods. Subsequently, three convolutional neural networks were trained: Model A using the original dataset, and Models B and C utilizing the outputs of the selected passive methods. These models were then combined by concatenating them at the penultimate layer. Additionally, an extra dense layer followed by a batch regularization layer was introduced so the model could learn how to combine the results, this combination is the final merged model.

To enhance accuracy and mitigate overfitting, three techniques were employed. First, a model checkpoint callback was implemented, which saved the model weights corresponding to the minimum validation loss achieved during training. Second, an early stopping condition was defined, terminating training after 50 epochs without any improvement in validation loss.

The third technique used to reduce overfitting was the use of dataset augmentation techniques applied at the end of every epoch to the training images, which randomly flipped the image in the four-axis and changed the brightness, contrast, saturation, and hue of the images randomly.

The experiments were conducted on a system with the following specifications: Windows 11 operating system, Intel(R) Core(TM) i5-8300H CPU 2.30GHz, 16GB RAM, GeForce GTX 1050 graphics card, Python version 3.10, CUDA version 11.2, cuDNN version 8.1.1, and TensorFlow version 2.10.0. The training was performed with a batch size of 32 over a total of 500 epochs, as the dataset exhibited significant variation in manipulations, leading to fluctuation in the loss of validation images.

4. Results and Discussion

After training, the final Accuracy obtained by the merged model was 89,59% in the set of test images, higher than the model trained just with original images, which obtained 78.64%.

Figure 8, Figure 9, Figure 10 and Figure 11, show the area under curve (ROC) of the models when evaluated on the validation and test datasets.

Figure 12, Figure 13, Figure 14 and Figure 15, show graphics of the Accuracy and loss during training of models A, B, C, and the proposed merged model, respectively.

The final results obtained after the stop conditions were triggered are on the Table 3 below.

5. Conclusion and Future Work

This work offered a review of how forensic methods for detecting manipulated images work and applied two detection methods and a mixture of both using CNNs to a dataset containing a diverse set of manipulations.

After training, the proposed merged method obtained an accuracy of 89.59% in the validation images, significantly higher than the model trained with original images, which obtained 78.64%, and the model that enhances compression differences in images using ELA which obtained 68.02% and the worst performing model individually that uses the method proposed by [53] with 50.70%.

The combination of models trained differently but on the same set of images in this work demonstrates the possibility of using techniques from the detection of falsified images to guide Neural Network models to learn different detection techniques. We observed this in the final model that obtained an Accuracy higher than all the models used for its creation.

With the proposed approach, the research could serve as a basis for the elaboration of future works about the detection of manipulated images that contain different approaches, nomenclatures, and possible methods that were generalized to structure the knowledge available in this field.

The proposed method opens several possibilities for future work. For example, it is possible to cite them as the combination and comparison of more trained models with different selection or feature extraction methods, testing other model architectures like you only look once (YOLO), modifications to the model to localize the manipulated area and tests of the proposed method on other sets of images. In future work, we also intend to optimize the algorithms used, compare the results, and evaluate them in other scenarios. When it comes to the safety of the data used, it will also be necessary to carry out studies and define future criteria and parameters for this issue. Energy consumption, quantity, and size of data used is something interesting to be investigated in future work, with this and other various hypotheses that may arise, several improvements and future contributions may result.