DUVR-Net: A Dual-Path Underwater Video Restoration Network Based on Middle-Frame Residual and Transformation Convolution Kernel

1. Introduction

Water is a crucial medium for human exploration and research, covering approximately 71% of Earth’s surface. Underwater video recording plays a vital role in marine biology research, underwater archaeology, and ecological surveys [1,2,3,4]. However, due to the physical properties of water, such as light absorption, scattering, suspended particles, and other factors,underwater videos typically suffer from poor visibility, color distortion, and low contrast [5], which significantly hinder their practical applications. Traditional underwater video enhancement methods primarily rely on image processing algorithms to enhance each frame independently. While these methods have achieved some success in improving video quality, they face several key challenges. First, processing each frame independently ignores the inherent temporal continuity and style consistency between frames in a video, which may lead to temporal artifacts and inconsistent enhancement effects. The emergence of UVEB [6] provides a large-scale (4K) high-resolution underwater video enhancement dataset and introduces UVE-Net, which converts current frame information into convolution kernels and passes them to adjacent frames to achieve effective inter-frame information exchange. However, the UVE-Net model is massive, requiring substantial computational resources and parameters, resulting in slow training processes and limited practical applications.

To address these limitations, we propose a novel underwater video enhancement framework that efficiently processes video sequences while maintaining temporal coherence. Our method introduces a dual-channel mechanism that first divides long video sequences into multiple shorter image sequences for input. For each image sequence, we extract the middle frame and enhance it using multi-scale and multi-channel attention mechanisms. The multi-channel processing better extracts image color features for enhancement, while multi-scale processing captures image details at different scales to enhance image detail enhancement. After enhancing the middle frame of the image sequence, the enhancement effect is propagated to adjacent frames through two complementary pathways. The effectiveness of our dual-channel mechanism is demonstrated in Figure 1, where we compare our method against existing approaches (UVE-Net [1] and PRWNet [7]) on a challenging underwater scene. As shown, our method better preserves the natural color and detail of underwater structures while effectively removing blur and color distortion present in the original frames.

In DUVE-Net,the first pathway is the residual-based repair pathway. Within the same short image sequence, frames exhibit temporal continuity and style consistency, so the enhancement effects needed for other frames share similarities with the enhancement effects obtained from the middle frame. Therefore, the enhancement effects needed for other frames can be derived from the middle frame’s enhancement effects through inter-frame relationships. By calculating the difference between the enhanced and original middle frame, we obtain the enhancement effect residual of the middle frame. Considering the differences between frames, we calculate the inter-frame residual by comparing the middle frame with remaining adjacent frames. By applying convolution to the inter-frame residual as weight parameters for the middle frame’s enhancement effect residual, we can obtain independent enhancement effects for each frame, which can be added to the corresponding original frames to achieve enhanced results. The residual-based repair pathway effectively transmits the enhancement effects of the middle frame.

The second pathway is the convolution-based repair pathway, which generates adaptive convolution kernels from the enhanced middle frame to process the original adjacent frames. Each original frame independently uses the enhanced middle frame to generate convolution kernels for convolution, allowing the middle frame’s detail information to be transmitted to each frame. This design achieves effective inter-frame information transfer. The convolution-based repair pathway focuses more on the details in the enhancement effect transmission, reducing the impact of inter-frame differences.

The main contributions of our work can be summarized as follows:

1. We propose a novel underwater video enhancement framework that explicitly considers inter-frame relationships, ensuring temporal consistency in enhancement results.

2. We introduce a dual-channel mechanism combining residual-based repair pathway and convolution-based repair pathways, which effectively propagates enhancement information from middle frames to adjacent frames.

3. Our method achieves remarkable parameter efficiency, requiring only $\sim 90k$ parameters while maintaining high-quality enhancement effects for 4K underwater videos.As shown in Figure 2, our DUVR-Net achieves superior PSNR performance with significantly fewer parameters compared to existing methods.

4. Extensive experiments on real underwater videos demonstrate the effectiveness of our method in terms of both visual quality and computational efficiency.

2. Related Work

2.1. Underwater Image Enhancement Methods

Underwater Image Enhancement methods can be broadly categorized into physical model-based methods, physical model-free methods, and deep learning-based methods. Physical model-based methods construct models by considering image degradation factors in specific environments, thereby reversing or compensating for degradation effects and improving image quality [8,9,10]. Physical model-free methods directly adjust image pixel values through histogram equalization [11] and color balance, bypassing the complex optical propagation models to enhance underwater images through direct manipulation of pixel intensity distributions and color characteristics. With the successful application of deep learning in computer vision tasks, various innovative architectures have emerged. Notable networks include WaterNet [12], which employs adaptive filters [13] and CNNs [14] for quality improvement; Ucolor [15], which utilizes medium transmission-guided multi-color space embedding; and FA+Net [16], which incorporates color enhancement and pixel attention modules. Transformer-based [17] solutions such as U-shape [18] have also demonstrated excellence in capturing global features and handling long-range dependencies. Recent lightweight networks developed based on generative adversarial networks, such as FspiralGAN10 [19], are more suitable for resource-constrained environments. Although these networks have made significant progress, challenges still persist in color distortion correction, texture preservation, and computational efficiency. In the field of UIE, balancing quality and efficiency remains a more critical issue.

2.2. Underwater Video Enhancement Methods

Early underwater video enhancement methods primarily extended single-frame image enhancement algorithms to video processing through frame-by-frame processing. Although some methods improved efficiency through fast filtering algorithms or lightweight deep networks, this independent frame processing often led to temporal artifacts and inter-frame flickering. To address these issues, researchers began exploring temporal relationships between frames. For instance, Ancuti et al. [20] introduced temporal bilateral filtering to achieve inter-frame smoothing, while Li et al. [21] extended video dehazing algorithms to underwater scenarios. Qing et al. [22] proposed a spatiotemporal information fusion method to estimate consistent transmission maps and background light.Meanwhile, recent advances in video restoration have provided new insights for underwater video enhancement. Researchers have proposed various models to capture inter-frame relationships and aggregate temporal information, including optical flow estimation, deformable convolutions, and cross-frame attention layers. Notable examples include Shift-Net14 [23], which introduces grouped spatiotemporal displacement to implicitly capture inter-frame correspondences, and UVE-Net [2], which innovatively transforms frame information into convolution kernels for efficient inter-frame information exchange. Inspired by these approaches, we propose DUVR-Net, a dual-path method that simultaneously utilizes residual compensation and transformative convolution kernels for underwater video enhancement.

3. Methods

3.1. Overview of DUVR-Net

The input to DUVR-Net consists of consecutive frames ${\{x_i\in {\mathbb{X}}^{3\times H\times W}\}}_{i=t-k}^{t+k}$ from an underwater video, where t denotes the index of the middle frame, and $T=2k+1$ represents the total number of input frames. The output of DUVR-Net comprises enhanced consecutive underwater video frames ${\{y_i\in {\mathbb{Y}}^{3\times H\times W}\}}_{i=t-k}^{t+k}$. The operation of DUVR-Net can be formulated as shown in Equation (1):

$${\{y_i=DUVR\_Net\left(x_i\right)\}}_{i=t-k}^{t+k}$$

(1)

$\mathbb{X}$ and $\mathbb{Y}$ denote the original low-quality underwater videos and their corresponding enhanced versions, respectively.The overall architecture of DUVR-Net is illustrated in Figure 3.

3.2. Middle Frame Enhancement Module

The Middle Frame Enhancement Module primarily focuses on enhancing individual middle frames, with its network architecture specifically designed for image-mode enhancement. To address the challenges of contrast deficiency and color distortion during the enhancement process, we adopt the enhancement strategy from FA+Net [16], implementing a dual-branch enhancement mode comprising multi-channel and multi-scale pathways during middle frame enhancement. The multi-channel branch transforms the input into features with varying channel dimensions to better capture color feature distributions across RGB channels. In the multi-scale branch, the input is converted into features with different heights and widths, utilizing $1\times 1$ convolutions to perform consecutive processing of individual image pixels, which not only reduces the parameter burden but also better captures color information for each pixel. The information fusion of the dual branches is accomplished through Inverse Fourier Transform [24], with a hyperparameter $\mathbf{\alpha }$ controlling the fusion ratio between spatial and frequency domain information.Subsequently, the fused features undergo fine-grained enhancement through the serial application of multi-scale modules and spatial attention mechanisms, ultimately producing the enhanced middle frame.

3.3. Video Enhancement Module

The video enhancement module takes as input a sequence of consecutive frames from a short underwater video, along with their corresponding enhanced high-quality intermediate frame. The output is the enhanced consecutive underwater video frames corresponding to this input sequence.The enhancement is achieved by propagating information from the enhanced middle frame to the remaining frames through residual and Transformation convolution kernels.

3.3.1. Residual Repair Model

The Residual Modification Model (RMM) aims to extract the enhancement effects obtained from the middle frame and modify these effects according to the differences between each frame and the middle frame, subsequently applying the modified results to the input frame sequence. The RMM takes as input both a consecutive frame sequence requiring enhancement and its enhanced middle frame. By extracting, customizing, and propagating the enhancement effects instead of enhancing each frame individually, this approach significantly reduces the model’s computational complexity and parameter count.

The key operations of RMM are as follows, given the feature ${\{x_i\in {\mathbb{X}}^{3\times H\times W}\}}_{i=t-k}^{t+k}$ and the enhanced middle frame ${\widehat{x}}_t$ derived from $x_t$.

Middle frame enhancement residual($R_1$) extraction:

$$R_1={\widehat{x}}_t-x_t$$

(2)

Inter-frame residual($R_2$) calculation:

$$R_2={\{x_i-x_t\}}_{i=t-k}^{t+k}$$

(3)

Residual modulation($R_3$),⊙ represents Element-wise multiplication:

$$R_3=\mathrm{expand}\left(R_1\right)\odot R_2$$

(4)

Enhancement effect integration:

$${\mathrm{Out}}_{RMM}=\mathbb{X}+R_3$$

(5)

In summary, the complete processing of RMM can be expressed as:

$${\mathrm{Out}}_{RMM}=\mathbb{X}+{\{\mathrm{expand}({\widehat{x}}_t-x_t)\odot (x_i-x_t)\}}_{i=t-k}^{t+k}$$

(6)

3.3.2. Transformation Convolution Kernel Model

The transformation convolution kernel converts the enhanced intermediate frame into a 3×3 convolution kernel through Dynamic Convolution Kernels Generation (DCLG) and applies convolution operation to the current frame, thereby obtaining information from the enhanced intermediate frame and achieving inter-frame information transfer.The process of TCKM can be expressed as:

$${\mathrm{Out}}_{TCKM}=\mathrm{DCLG}\left({\widehat{x}}_t\right)*{\left\{x_i\right\}}_{i=t-k}^{t+k}$$

(7)

where ${\left\{x_i\right\}}_{i=t-k}^{t+k}$ represents the features of the input consecutive frames. ${\widehat{x}}_t$ represents the enhanced middle frame derived from $x_t$. $\left(*\right)$ represents convolution.

3.4. Loss Function

We calculate the loss of the middle frames and video as the total loss function $\mathcal{L}$:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{Char}}(y_t,\widehat{x_t})+{\mathcal{L}}_{\mathrm{Percep}}(y_t,\widehat{x_t})+\lambda \left[{\mathcal{L}}_{\mathrm{Char}}{\left\{(y_i,\widehat{x_i})\right\}}_{i=t-k}^{t+k}+{\mathcal{L}}_1{\left\{(y_i,\widehat{x_i})\right\}}_{i=t-k}^{t+k}\right]$$

(8)

where ${\mathcal{L}}_{\mathrm{Char}}$ refers to Charbonnier Loss [25], ${\mathcal{L}}_{\mathrm{Percep}}$refers to Perceptual Loss [26], ${\mathcal{L}}_1$refers to $L_1$ Loss [27]. ${\left\{y_i\right\}}_{i=t-k}^{t+k}$is the GT of sample${\left\{x_i\right\}}_{i=t-k}^{t+k}$ , and $y_t$ is the GT of $x_t$.$\lambda$ is a weighting parameter that controls the contribution of video-related losses, and its value increases as the number of epochs grows.

4. Results

4.1. Settings

4.1.1. Dataset

We used the large-scale high-resolution Underwater Video Enhancement Baseline (UVEB) [2] released on July 7, 2024, as our data source. The UVEB dataset comprises videos from multiple countries, encompassing various scenes and video degradation types to accommodate diverse and complex underwater environments. Due to the extensive size of the UVEB dataset, we utilized the first 150 pairs of video sequences in our experiment, consisting of 150 source videos and 150 target videos. By extracting one frame every six frames, we obtained a total of 16,836 images, including 8,418 source image sequences and 8,418 target image sequences. The ratio of training set to test set was set at 9:1, resulting in 7,563 images for training and 855 images for testing.

4.1.2. Implementation Details

Our experiments are implemented using the PyTorch [28] framework with two NVIDIA Tesla V100-SXM2-16GB GPUs. We use an ADAM optimizer for network optimization. The initial learning rate is set to ${4\times 10}^{-4}$. The total number of epochs is 100. The batch size is 4, and the patch size of input video frames is 512×512, with each sequence containing either 3 or 5 frames.

4.2. Comparisons with State-of-the-Art Methods

4.2.1. Quantitative Comparison

In quantitative measurement, we use peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and mean square error (MSE) [29] as metrics for image quality. Floating point operations (FLOPs) [30] and parameter count (Params) [31] serve as indicators of model efficiency and performance. From these quantitative results, our method outperforms others in terms of image quality. As shown in Table 1, for 4K video, our model uses the minimal computational and parameter resources, in the same order of magnitude as FA+Net, which has the smallest model size in image processing tasks. DUVRNet has only 1/350 of the parameter count of UVE-Net while improving the PSNR metric by 3.19dB, demonstrating the superiority and feasibility of this algorithm

4.2.2. Qualitative Comparison

As shown in Figure 4, we visually compared our network enhancement results with other methods. Due to its simple network structure, PRW-Net cannot adequately restore underwater images, resulting in enhanced images with low saturation. The enhancement results presented by PRW-Net appear unnatural in color and generally too dark. In contrast, our method demonstrates superior ability in removing color distortion while restoring appropriate brightness levels.

4.3. Ablation Studies

4.3.1. Effectiveness of Middle Frame Enhancement Module

To explore the impact of the middle frame enhancement module on the entire network, we replaced it with a residual neural network and examined the effects of this substitution. As shown in Table 2, MFEM represents the Middle Frame Enhancement Module in the model. Comparing columns (a) and (d) in the Table 2, MFEM enables the network to achieve better quality improvement with fewer parameters. Additionally, as demonstrated in Figure 5, our implemented MFEM exhibits superior capability in recovering color and brightness information in the images.

4.3.2. Effectiveness of RRM and TCKM

The results in columns (b), (c) and (d) of Table 2 verify the effectiveness of RRM and TCMK. From columns (b) and (d), we observe that the introduction of the RRM module not only reduces the MSE from 1.935 to 0.882 but also accelerates the inference time. The visual results in Figure 5 demonstrate that the absence of RRM and TCKM modules may lead to imprecise enhancement effects, resulting in artifacts such as ghosting and color distortion. The incorporation of both RRM and TCKM modules contributes to faster and better enhancement performance.

5. Discussion

Compared to traditional frame-by-frame processing methods, DUVR-Net not only has a smaller model parameter count but also demonstrates the capability to process 4K videos. The model shows considerable potential in practical applications such as marine research and underwater archaeology. It can provide high-quality input for downstream tasks like underwater object detection.Although DUVR-Net is specifically tailored for underwater videos, its network architecture is still based on image enhancement networks. The dual-channel mechanism primarily functions to propagate image enhancement effects to adjacent frames. Therefore, in specific scenarios (such as scenes with intense motion), there might be certain limitations. Future work will focus on incorporating other advanced technologies (such as adaptive parameter adjustment) to make the model more adaptable to multi-environmental tasks. Additionally, attention will be paid to optimizing the enhancement strategy for intermediate frames.

6. Conclusions

In this paper, we present DUVR-Net, a lightweight model for 4K underwater video enhancement that aims to overcome the limitations of previous frame-by-frame enhancement methods. Our approach employs a dual-channel strategy that effectively reduces computational complexity by separately transmitting enhanced color content with attention mechanisms and restored details. DUVR-Net currently has the smallest parameter count among underwater video enhancement models, while experiments validate its superior performance in underwater video enhancement tasks. We believe that DUVR-Net provides new design concepts and directions for underwater video enhancement, with practical applications in marine research and underwater archaeology.