Video Quality Preservation Based on Deep Neural Network

1. Introduction

Nowadays, the demanded amount of memory needed to store data is an important issue in the field of Information Technology. One of the methods to solve this problem is the use of lossless and lossy compression. Video codecs are specialized data compression for video purposes. Lossless compression is characterized by a large amount of memory needed to store data. On the other hand lossy compression allows to store video data in a form that is imperceptibly changed or slightly noticeable to the human eye by maintaining the appropriate degree of video compression, while offering a reduction in the need to store video on data carriers with limited storage capacity. Examples of popular video codecs that offer both lossless and lossy compression are: H.264, H.265 and AV1. With the advent of lossy video compression, the natural course of things was a need to recover previously lost video quality which was compressed in order to reduce the amount of memory needed to store it.

The purpose of this publication is to present the developed method that enables improving the quality of compressed video by applying the process of enhancing image characteristics before compression rather than after decompression, which is commonly used in most methods. The problem of improving image quality after use of compression codec is challenging and complexity of it increases with the greater levels of compression (QP coefficient), which, at its highest values, makes it difficult to correctly recognize the details of objects or generally makes them less visible in the image. The presented method is based on the use of DNN to preprocess the image, which will be subjected to AVC and HEVC compression to increase the similarity of the compressed image to the original image.

The structure of the manuscript is as follows: The literature related to this publication is presented at the beginning. Next, the idea of the proposed method will be presented and the Artificial Neural Network (ANN) learning method will be discussed. Subsequently, the results of the method will be presented and discussed. The final part of the manuscript contains a summary of the research results and suggestions for further development. Appendix A shows preview of video compression comparison with the use of the proposed method and without it.

2. Related Work

In reviewing the literature related to video quality improvement, attention should be paid to the following issues: video enhancement [1,2,3,4,5,6], video restoration [7,8,9], video deblurring [10,11,12], video denoising [13,14,15,16], video dehazing [17,18] and video super-resolution [19,20,21]. The above-mentioned issues overlap and share and have a common feature of improving video quality.

The Video Super-Resolution methods are designed to improve video quality by increasing image resolution while maintaining image quality. The Video Deblurring and Video Denoising methods are designed to improve video quality by removing unwanted graphic effects. The Video Restoration and Video Enhancement methods, on the other hand, are intended to improve the overall video quality.

There are two main trends in the available literature on restoring video quality. The first one is based on the use of a sequence of video frames [7,8,9,10,11,22,23,24]. The second one is based on the principle of improving a single video frame [25,26,27,28].

Many of the methods available in the literature perform video quality restoration through methods based on the processing of a single video frame [26,27,28,29]. The use of this type of methods enables the parallelization of calculations.

Methods using a sequence of video frames can be divided into: sliding-window-based [7,8,9,10,11,23,24,25] and recurrent [30,31,32,33,34,35,36,37] methods. Sliding window-based methods in general use multiple input frames to generate a single improved video frame. Methods based on recursion utilize recurrence for their computations and require more time than sliding window-based methods.

Among the methods aimed at improving the quality of compressed video, there are methods that can be applied to various types of compression [3,5] and as well as dedicated methods to a given type of video codec [2,3,4,23,38].

An important aspect of this work is the fact that the focus is mainly on improving the quality of compressed video frames after passing through a process of compression using the H.264 and H.265 codec. Therefore, learning of the features of the HEVC codec as best as possible is crucial for proposed neural network. A good step is to analyze the architecture of the neuron-based networks used in conjunction with the encoding of the HEVC codec. Without trying to obtain the characteristics of the HEVC codec through simulation/reconstruction of its actions or methods such as deep reinforcement learning [39] or black box gradient estimation [40,41,42,43,44], a problem known as no continuity in the gradient can be encountered (non-differentiability of certain processing block), which destabilizes the training process or makes it impossible.

In the case of processing video using the HEVC codec, deep neural networks are used to replace encoding HEVC through end-to-end systems completely based on processing video by subsequent blocks of neural network. In case of [45] authors replaced processing blocks of HEVC algorithm by corresponding to them DNN blocks. Transform and Inverse transform blocks were replaced by Residual Encoder/Decoder Net, Motion Compensation block was replaced by Motion Compensation Net+MV Encoder/Decoder Net+Optical Flow Net, and Entropy Coding block was replaced by Bit Rate Estimation Net [45]. Another approach similar to [45] presented in work [46] (MOVI-Codec) is to use LSTM-Unet neural network which allow to skip Motion Compensation block in process of compression video. Another possibility for applying DNN is improving the speed of encoding HEVC by reducing complexity of some of HEVC algorithms. The papers [47,48] (IPCNN, ImageNet) shows that by replacing algorithms for intra prediction mode decision by a Convolutional Neural Networks (CNNs) it possible to have less computational complexity but similar or better performance. Some methods aim to improve certain stages of HEVC processing, like in work [49] where researchers by using DNN models like DAIN [50] and SloMo [51] improve effectiveness of codec in interpolating frames. In the work of Çetinkaya et al. [52] the use of CNNs for Coding Tree Unit process can be also seen.

To measure quality of processed frames/images by compression methods, image quality assessment (IQA) metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), Visual Information Fidelity (VIF), Mean Square Error (MSE), Mean Opinion Score (MOS), BD-PSNR and BD-Rate are used.

PSNR provides an indication of the overall quality of an image, but it does not take into account human perception [53,54,55]. SSIM takes into account both the luminance and chrominance channels of an image, making it a more robust measure than PSNR alone [53,55,56]. Visual Information Fidelity (VIF) measures how well a compression algorithm preserves the visual information in an image. VIF is used extensively in video coding standards such as H.264 and H.265, where it is combined with other metrics to optimize image quality [53,55,56,57]. BD-PSNR and BD-Rate are measures of compression efficiency over a range of bitrate or quality values [58,59].

Mean Squared Error (MSE) refers to the average squared difference between two sets of data points or images. In the context of image quality assessment, it is a widely used metric that measures how closely the compressed image resembles its original counterpart. The lower the MSE value, the better the compression and the higher the perceived image quality. Mean Opinion Score (MOS) is a widely-used metric for evaluating the perceived quality of images, videos and other multimedia content by human perception. MOS is particularly useful for evaluating the effectiveness of image compression algorithms that aim to preserve visual quality while minimizing file size.

To evaluate efficiency of proposed in this paper method, metrics like PSNR and SSIM were calculated for the test video.

3. Concept of the Proposed Method

3.1. General Concept of the Proposed Method

The general idea of the proposed method of improving video quality is presented below in form of flowchart diagram (Figure 1). The elements of diagram are representing the subsequent steps taken to improve quality of compressed video frames.

In the proposed method the main component is the Adjusting DNN (ADNN). ADNN input is fed with a video frame which is to be subjected to AVC or HEVC compression. As a result of the operation of the neural network, subsequent frames are created which are video components assumed to have better visual quality compared to video subjected to AVC or HEVC compression without prior preparation with use of ADNN. ADNN input layer accepts a single frame of video image and returns a single frame (adjusted to HEVC codec) in the output.

Due to the characteristics of HEVC, it should be noted that ADNN works on the YUV444p colour model. In addition, due to hardware limitations, the presented neural network works natively with an image resolution of 128 × 128. Native work on the presented resolution for higher image resolutions is solved by using the Sliding window algorithm.

3.2. HEVC Compression Adjustment Learning Process

In order to present the learning process, the following objective function relationships will be discussed:

$${MSE}_{HevcAdjusting}=\mathrm{M}\mathrm{S}\mathrm{E}\left(\left(floor\left({\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}_{Hevc}\left(QP\right)\cdot 255\right)\right),\left({\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}_{Adjusted}\left({\displaystyle \frac{\left(52-QP\right)}{52}}\right)\cdot 255\right)\right)QPϵ〈0;51〉$$

(1)

$$L_{AdjustingLoss}=\left(2.2246\cdot \left({\displaystyle \frac{{MSE}_{HevcAdjusting}}{255}}\right)\right)$$

(2)

$$\nabla L_{Adjusting}={\displaystyle \frac{{\partial L}_{AdjustingLoss}}{\partial {Var}_{Adjusting}}}$$

(3)

where $MSE$ is the mean square error; $floor$ is the floor function; ${\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}_{Hevc}$ is the image obtained after HEVC compression; $QP$ (Quantization Parameter) is the parameter determining the degree of HEVC compression; ${\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}_{Adjusted}$ is the image created by the trained ADNN mimicking the image after HEVC compression; ${MSE}_{HevcAdjusting}$ is the mean square error of the image produced by the ADNN in relation to the image compressed with HEVC, both parameterized by the $QP$ value; $L_{AdjustingLoss}$ is the ADNN error; ${Var}_{\mathrm{A}\mathrm{d}\mathrm{j}\mathrm{u}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g}}$ is the tensor of the ADNN model weights; $\nabla L_{Adjusting}$ is the gradient of the ADNN objective function.

In the equations for ${MSE}_{HevcAdjusting}$ and $L_{AdjustingLoss}$ there are multiplication and division operations by the numerical value 255, respectively, this is due to the fact that the neural network performs better when input data is normalized (scaled) in the range $〈0;1〉$. After all the computations are done on the processed frame to convert output of neural network to image form the output is multiplied by the value of 255 and then subjected to the $floor$ function as each pixel value is represented in the form of integer values in the range $〈0;255〉$.

The learning process of neural network consisted of 300 epochs of the learning algorithm. In this process, the Adam Optimizer algorithm was used. During learning process the learning rate value was changed to improve and speed up achieving convergence by neural network for a given problem. For the first 100 epochs, the learning rate value was set to 10⁻⁴. Then, the ANN giving the best results from 100 learning epochs was selected and for this ANN in the next 200 epochs, the learning rate was set 10⁻⁶. The batch size used was 16.

The training set consisted of 4,000 random images for each epoch from ImageNet dataset [60], which were transformed to a resolution of 128 × 128, then these images were subjected to HEVC compression with a value of 0. The training set prepared in this way was used in the learning process, which is equivalent to the entry ${\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}_{\mathrm{H}\mathrm{e}\mathrm{v}\mathrm{c}}\left(0\right)$ in equation number 1. On the other hand, the entry ${\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}_{\mathrm{H}\mathrm{e}\mathrm{v}\mathrm{c}}\left(QP\right)$ in the same equation is equivalent to processing the image from the training set by performing the appropriate HEVC compression on the image, the degree of which is parameterized by the QP value in the range of integer values $〈0;51〉$.

Considering the similarities in image processing by H.264/AVC (Advanced Video Coding) and its successor H.265/HEVC (High Efficiency Video Coding) video codecs, only one DNN was trained on the HEVC characteristics.

3.3. Neural Network Architecture

The ADNN structure presented in this manuscript uses layers such as: BatchNormalization, Concatenate, Conv2D, Conv2DTranspose, DepthwiseConv2D and multiplication operation. The activation functions used in the ADNN structure are GELU and LeakyReLU.

Table 1, Table 2 and Table 3 show the ADNN structure. Table 1 and Table 2 show the Convolution Basic Layer (CBL) and Transposed Convolution Basic Layer (TCBL), respectively. The logical layers are composed of a sequence of the previously mentioned base layers offered by the TensorFlow machine learning library. Table 3 shows the general model of applied ANN. In Table 1, Table 2 and Table 3, individual numerical values in layer type descriptions refer to previous layer numbers.

Table 2 and Table 3 show that there is only one difference between the presented logical layers, which is the use of a different basic layer for the third layer in the sequential logical models.

In Table 3, the Input image value is the image stored in the range of numeric values $〈0;1〉$ to be subjected to ADNN processing. $InputQPvalue$ is the compression value from the range $〈0;51〉$ unified to the range of number values $\left(0;\right.1⟩$ on which DNN operates in accordance with the following relationship:

$$InputQPvalue={\displaystyle \frac{\left(52-QP\right)}{52}}QPϵ〈0;51〉$$

(4)

where $QP$ (Quantization Parameter) is the parameter determining the degree of HEVC compression; $InputQPvalue$ is the calculated value of $QP$ converted for ANN computations purpose in the value range $〈{\displaystyle \frac{1}{52}};1〉$.

Parameters of ADNN relating to hardware requirements are shown in Table 4. Speed of ANN processing was measured on GeForce 1080Ti GTX 11GB graphics processing unit (GPU).

4. Results

In this section, the results of the described method will be presented and discussed. First AVC and then HEVC results of improving the quality of video compressed with video codec in the context of prior preparation of video frames by ADNN will be presented and discussed.

The research was conducted on a Linux operating system (Ubuntu 22.04) using the TensorFlow version 2.13.0 machine learning library. To process video the FFmpeg video handling library (version 4.4.2-0ubuntu0.22.04.1) was used. The source codes were written in Python version 3.10.12.

In the further part of the section, the performance results of presented method obtained for individual frames of the test video [61] and for selected DAVIS [62], GoPro [63], REDS [64], Vimeo-90K-T [65] test datasets of videos are presented. For DAVIS [62], the selected tested videos are: bear, blackswan, boat, bus and dog. For GoPro [63], the selected tested videos are: GOPRO0384, GOPRO0385, GOPRO0396, GOPRO0410. For REDS [64], the selected tested videos are: 001, 002, 003, 004 and 005. For Vimeo-90K-T [65], the selected tested videos are: 00001 and 00002. Video enhancement tests were conducted by default at 640 × 640 video resolution. Due to the fact that ADNN was trained for images with a resolution of 128 × 128, the moving window method was used to process the entire image.

The effectiveness of the method for selected values of QP are presented through coefficients (PSNR and SSIM) that mathematically determine the similarity between images. These coefficients represent the calculated similarity value however, it is not an absolute determinant of image quality. The potentially human-visible modification of the image as a result of the coefficient calculation may be considered insignificant. There is also a possibility of the reverse situation, consisting in finding an improvement in image quality by a human, while the calculation of the coefficient shows otherwise.

4.1. AVC Compression Enhancement Research Results

The results of ADNN processing in the context of improving the visual quality of video subjected to AVC compression are presented in Table 5, Table 6 and Table 7, which show the averaged results of processing on sample test video frames [61] with the image resolution changed to 640 × 640.

A graphical preview of ADNN processing in the context of improving video quality is shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 and in Appendix A.

From the analysis of the results in Table 5, Table 6 and Table 7 and Figure 2, it is noticeable that there is no direct improvement and a slight deterioration of the PSNR value for ADNN compression in comparison to AVC compression. This is due to the fact that ADNN makes more changes to the image than native AVC compression.

Based on the results of the study, it was found that the proposed method works effectively when there is no direct translation of the QP compression factor value fed to the ADNN input to the target QP compression value for AVC compression. Starting from QP value 25 there is noticeable tendency to improve the quality of the video.

In order to properly present the use of ADNN capabilities, a conversion table (Table 8) was compiled on the basis of the conducted research, which presents suggested compression values for ADNN in relation to the target AVC compression value. It should be emphasized that conversion values in Table 8 are used further in this manuscript to present further results in the context of the AVC compression. Meaning that for a compression value, e.g. QP = 45, the value of the compression ratio at the ADNN input was given in accordance with conversion table (Table 8) as 36, and then processed image from ADNN was given for the AVC video codec, for which the value of compression is QP = 36. Thus, keeping for the indicated value a larger number details with a similar output file size.

In order to utilize the conversion table (Table 8), it is needed to first determine the target compression ratio QP for AVC, then read the corresponding QP value for ADNN from the table and use it in the calculations for the image preprocessing by ADNN. Using the values from the conversion table when determining the QP value for ADNN provides a greater visual quality improvement value for the AVC compression target QP value, than using directly.

The average gain of the proposed method in numerical form for specified QP value can be obtained by reading the values from Table 8. Average of ∇PSNR is 2.03 dB and the average value of file size reduction is 1.16 KB for QP value greater than 24. The graphical results of the ADNN processing are shown in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8.

Figure 3. Preview of compression effects for frame 200 and 400 of the test video [61] with QP=41 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 3. Preview of compression effects for frame 200 and 400 of the test video [61] with QP=41 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 4. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=45 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 4. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=45 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 5. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=51 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 5. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=51 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 6. Preview of compression effects for frame 200 with QP=43 of the test video [61] (viewed from the left: the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 6. Preview of compression effects for frame 200 with QP=43 of the test video [61] (viewed from the left: the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 7. Preview of compression effects for frame 600 with QP=47 of the test video [61] (viewed from the left: the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 7. Preview of compression effects for frame 600 with QP=47 of the test video [61] (viewed from the left: the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 8. Preview of compression effects for frame 800 with QP=51 of the test video [61] (viewed from the left: the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

Figure 8. Preview of compression effects for frame 800 with QP=51 of the test video [61] (viewed from the left: the image compressed by pure AVC, and the image processed by ADNN and compressed by AVC).

4.2. HEVC Compression Enhancement Research Results

The results of ADNN processing in the context of improving the visual quality of video subjected to HEVC compression are presented in Table 9, Table 10 and Table 11, which show the averaged results of processing on sample test video frames [61] with the image resolution changed to 640 × 640. A graphical preview of ADNN processing in the context of improving video quality is shown in Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 and in Appendix A.

From the analysis of the results in Table 9, Table 10 and Table 11 and Figure 9, it is noticeable that there is no direct improvement and a slight deterioration of the PSNR value for ADNN compression in comparison to HEVC compression. This is due to the fact that ADNN makes more changes to the image than native HEVC compression. Similar to the previously discussed results for native AVC compression.

Based on the results of the study, it was found that the proposed method works effectively when there is no direct translation of the QP compression factor value fed to the ADNN input to the target QP compression value for HEVC compression. Starting from QP value 23 there is noticeable tendency to improve the quality of the video.

In order to properly present the use of ADNN capabilities, a conversion table (Table 12) was compiled on the basis of the conducted research, which presents suggested compression values for ADNN in relation to the target HEVC compression value. Similar to the previously discussed results for native AVC compression.

It should be emphasized that conversion values in Table 12 are used further in this manuscript to present further results in the context of the HEVC compression. Meaning that for a compression value, e.g. QP = 45, the value of the compression ratio at the ADNN input was given in accordance with conversion table (Table 12) as 38, and then processed image from ADNN was given for the HEVC video codec, for which the value of compression is QP = 38. Thus, keeping for the indicated value a larger number details with a similar output file size.

In order to utilize the conversion table (Table 12), it is needed to first determine the target compression ratio QP for HEVC, then read the corresponding QP value for ADNN from the table and use it in the calculations for the image preprocessing by ADNN.

The average gain of the proposed method in numerical form for specified QP value can be obtained by reading the values from Table 12. Average of ∇PSNR is 1.27 dB and the average value of file size reduction is 0.76 KB for QP value greater than 22. The graphical results of the ADNN processing are shown in Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15.

Figure 10. Preview of compression effects for frame 200 and 400 of the test video [61] with QP=41 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 10. Preview of compression effects for frame 200 and 400 of the test video [61] with QP=41 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 11. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=45 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 11. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=45 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 12. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=51 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 12. Preview of compression effects for frame 600 and 800 of the test video [61] with QP=51 (viewed from the top, the rows represent consecutive video frames, viewed from the left, the consecutive columns represent the original image, the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 13. Preview of compression effects for frame 200 with QP=43 of the test video [61] (viewed from the left: the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 13. Preview of compression effects for frame 200 with QP=43 of the test video [61] (viewed from the left: the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 14. Preview of compression effects for frame 600 with QP=47 of the test video [61] (viewed from the left: the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 14. Preview of compression effects for frame 600 with QP=47 of the test video [61] (viewed from the left: the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 15. Preview of compression effects for frame 800 with QP=51 of the test video [61] (viewed from the left: the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

Figure 15. Preview of compression effects for frame 800 with QP=51 of the test video [61] (viewed from the left: the image compressed by pure HEVC, and the image processed by ADNN and compressed by HEVC).

5. Conclusions

Improving the quality of video material after compression is a challenging issue. Improvement in image quality can be determined using various methods such as MSE, PSNR, or SSIM. However, it should be noted that mathematical calculations confirming or denying improvement in image quality may not take into account visible or invisible defects in the image that can be perceived by some viewers.

During the studies, it was found that there is an increase in the mathematical average PSNR value of approximately 2.03 dB for AVC and 1.27 dB for HEVC codec for each frame of video subjected to compression using AVC or HEVC starting from compression value (QP) value of 25 for AVC and 23 for HEVC. General tendency towards visual improvement in image quality with the use of the proposed method was confirmed from the compression values mentioned in the previous sentence up to its highest value of compression in its range of up to 51. The higher average gain values for AVC than HEVC are due to the fact that ADNN was trained using the compression characteristics of HEVC codec, which is a successor to AVC codec with stronger compression than AVC.

The proposed method results in a more accurate representation of color and details in the image. ADNN makes preliminary changes to the image preparing it for compression by taking into account the degradation characteristics caused by HEVC codec. In comparison to the popular trend of improving the quality of an already degraded image, the proposed method aims to improve the quality by trying to preserve it before compression algorithm is applied. The proposed method can be also used alongside with methods that corrects quality of images after decompression in order to obtain further quality improvement of decompressed video frames.

Further research on the proposed method should consider increasing the mathematical similarity of compressed image to original image for a wider range of compression offered by AVC and HEVC. To achieve this goal, further optimization of training process of neural network can be introduced. It is considered to use learning method based on use of Generative Adversarial Neural Networks (GAN) or Diffusion networks. GAN method will allow to pretrain ADNN in learning process of generator and discriminator to find in unsupervised manner dependencies (neural representations of codec characteristics) of AVC and HEVC processing, that can be later utilized to improve supervised mode of improvement image training.

Modification to the architecture of DNN model may allow for further optimizations. The use of models like IPCNN [47], DAIN [50] and SloMo [51] as part of ADNN structure may allow to capture or predict how frames will be modified after AVC and HEVC compression, and in this way more effectively adjust characteristics of video for coding process.

Next important thing to optimize will be objective function. Proper construction of objective function will allow to highlight for DNN (in training process) the key elements that impact perception of images quality. The approach with use only MSE and PSNR will not allow to further improvements of proposed method, because these quality indicators focus mainly on pixel-wise image differences and do not capture perceptually relevant differences. The objective function should be based on perceptual loss functions like SSIM, VIF, MOS which will be the nearest to human perception performance indicators.