Automatic Unwanted Foreground Object Detection in Images Using Deep Learning

1. Introduction

A digital image is a numerical representation of a real image that a computer can store and process. Digital images are electronically saved on storage devices in different file formats, including JPEG, GIF, GIF89a, PNG, SGV (Scalabe Vector Vector), TIFF, etc. Figure 1 shows October 2025 reports that more than 5 billion active social media user identities are now recorded, with the global total reaching 5.04 billion at the start of 2024. Indeed, the journey of social media was initiated more than 50 years ago.

The trend of images [2] in social media increases daily. People take pictures as their passion, and some take pictures to show off their photography skills to others and post them on social networking sites such as Facebook, Instagram, Twitter, etc. Figure 2 shows the increasing trend in the use of social media.

The worldwide increasing trend of photography makes image quality maintenance more important. The reflections, fences, fog, shadow, and raindrops appear as unwanted foregrounds in images that decrease the visual quality of an image, as illustrated in Figure 3.

Detection and removal of unwanted foregrounds without significant degradation in visual quality from images is challenging. Several researchers have proposed techniques for unwanted foreground detection and removal from a single image and a set of frames. Some techniques targeted single image unwanted foreground detection and removal, while others performed on a set of input frames on unwanted foreground detection and removal [3,4,5,6,7,8].

There is a need to develop a comprehensive system that detects unwanted foreground reflection, shadow, fog, fence, and rain from images without significantly degrading the image quality that may help in the removal of unwanted foreground. This study aims to design a system based on deep learning that detects unwanted foreground-like reflection, raindrops, fog, shadow, and fences from images to improve their visual quality without significantly losing essential details. This study proposes a deep learning-based technique for automated detection of four types of unwanted foreground: fence, rain, fog, and shadow. Using a dataset sourced from online repositories and self-captured images, including specific datasets for each foreground type, the model, utilizing the VGG-16 deep learning architecture, achieves a 97.13% detection accuracy. This approach significantly enhances image quality by effectively separating unwanted foreground from the background, thereby addressing a crucial aspect of image processing.

The rest of the paper is organized as follows: Section 2 summarizes the related literature on automatic unwanted foreground detection and removal from images. The proposed approach is described in detail in Section 3. Experimental evaluation and analysis of results are presented in Section 4, while conclusions, along with future research scopes, are highlighted in Section 5 and Section 6.

2. Literature Review

Many studies have focused on removing unwanted foreground like reflections, haze, shadows, fences, and raindrops from images using various algorithms and techniques. However, the detection of these unwanted foreground elements is often overlooked. While removal methods have advanced, the critical step of accurately identifying and isolating these elements before removal remains a missing link in the research. This gap highlights the need for more work on detection techniques to improve overall image quality. Based on the type of unwanted foreground, the existing foreground removing techniques can be divided into the following five categories:

• Fence-removing techniques
• Reflection-removing techniques
• Raindrops removing techniques
• Shadow-removing techniques
• Haze-removing techniques

2.1. Fence Removing Techniques

Gupta et al. [3] proposed an algorithm for image de-fencing. Image de-fencing without losing image details is a challenging task. They proposed that Condition Generative Adversarial Networks (cGANs) remove the fence from the image and recover the details of the image that can be lost due to fence removal. The proposed methodology contains two stages: In the first stage, the author generates a mask for the input fence image, and the second stage generates a de-fence image recovery network. In this paper, a public fence dataset contains the regular fence pattern. They also use the Pascal VOC dataset with irregular fence patterns. For training purposes, they also use the COCO dataset. The length of the training dataset is 29413. They use three quantitative metrics: MOS, SSIM, and LPIPS; the results of these metrics are 4.34, 0.87, and 0.17, respectively.

Fences are sometimes used for security purposes, such as in zoos or redistricted areas. However, some important information is lost when we take pictures of such areas and image quality decreases. Matsui et al. [4] use CNN for fence removal from a single image. The author proposes two CNN networks, U-Net and Res-Net. On DetectNet, you will find the code and test data. The PSNR and SSIM matrices results are 27.11± 2.58 and 0.906 ± 0.034, respectively.

The study, authored by Sanakaraganesh Jonna, Moushumi Medhi, and Rajiv Ranjan Sahay from IIT Kharagpur, focuses on the automatic removal of occlusions, such as fences, from single images using deep learning. The proposed methodology consists of a two-stage convolutional neural network: the first stage performs occlusion segmentation using a UNet-based fully convolutional network (OccNet), while the second stage employs a Generative Adversarial Network (GAN) for realistic image inpainting. The model is trained and evaluated on multiple datasets, including the UCSD Fence and IITKGP Fence datasets for segmentation and the Places2 and CelebA datasets for inpainting. The results demonstrate superior performance compared to existing methods, achieving high precision, recall, F-measure, and MAE while producing high-quality, artifact-free inpainted images. However, the approach has some limitations, such as occasional false detections when background structures resemble fences, high computational costs, and the need for a larger training dataset.

The study by Stavros Tsogkas, Fengjia Zhang, Allan Jepson, and Alex Levinshtein from Samsung AI Center, Toronto, introduces a fast and efficient multi-frame de-fencing method using optical flow and deep learning. The approach follows a three-step process: (1) fence segmentation using a U-Net model, (2) segmentation-aware optical flow estimation, and (3) flow-guided inpainting using a CNN to reconstruct a keyframe with the fence removed. The method is trained on synthetic data from Vimeo-90k and the De-fencing dataset and evaluated on a collected dataset of 185 real-world bursts with ground truth backgrounds. It outperforms existing methods in SSIM, PSNR, and LPIPS while running at 7 FPS, making it significantly faster. However, its performance depends on accurate fence segmentation and struggles with unusual fence patterns or low-contrast scenes.

The study by Divyanshu Gupta, Shorya Jain, Utkarsh Tripathi, Pratik Chattopadhyay, and Lipo Wang introduces a robust image de-fencing method using a two-stage conditional generative adversarial network (cGAN). The first cGAN detects the fence mask from a fenced image, while the second reconstructs a visually appealing de-fenced image. The model is trained on a combination of a public fence segmentation dataset and a synthetic dataset, which includes artificially generated fence structures on images from Pascal VOC and COCO, totaling around 29,000 images. The approach outperforms existing de-fencing techniques, achieving a Mean Opinion Score (MOS) of 4.34, Structural Similarity Index (SSIM) of 0.87, and LPIPS of 0.17, with an average processing time of 51 milliseconds per image. Despite its effectiveness, the method has some limitations, particularly in handling fence shadows, highly textured backgrounds, and extreme occlusions, which may slightly degrade performance in complex scenarios.

2.2. Reflection Removing Techniques

A method is suggested by Tasi et al. [5] to eliminate reflection from a single image. Layer separation, pixel compensation, detail reconstruction, and normalizing are the four steps of the purpose methodology: layer separation, pixel compensation, detail reconstruction, and normalization. The proposed framework is iterative. The dataset images are collected from the search engine and [8,9]. Results on seven images are 0.426, 1.142, 4.832, 7.637, 9.096, 0.246, and 0.430 compared with [8,9,19].

Due to reflection in images, images lose much of their details. Abiko et al. [6] suggest a generative adversarial network to eliminate reflections from a single image. This approach comprises the Max pooling layer, batch normalization layer, leaky ReLU layer, conventional layer, and bilinear interpolation layer. The dataset used for that work is the PASCAL VOC 2012 dataset, and that set is 17000 images. The author discarded pale or grayscale images because such images can affect the training process. They use bicubic intercalate to convert the image to 256 × 256 sizes. After that, they use some images for the background layer and some for the reflection layer. Another dataset, the SIR2 benchmark dataset, was also used for evaluation purposes. For the evaluation of achievement, they used SSIM and PSNR matrices. SSIM helps in finding background image quality. Restoration results in PSNR /SSIM on the SIR2 benchmark dataset containing 199 postcard images, 200 solid images, and 55 wild images are 19.64/0.918, 23.87/0.928, and 24.97/0.932, respectively.

Image information is lost when we take an image through a glass or window due to reflection. Using deep learning, Fu et al. [7] suggested removing reflections from photos. For that objective, a significant number of more than 10,000 synthetic photographs are used. They Pick a dataset of more than 10000 synthetic images and 110 pairs of real images. Additionally, 110 genuine image pairs were tested. PSNR and SSIM results for synthetic images are 24.31 and 0.86, respectively. The PSNR and SSIM results for actual images are also 24.73 and 0.82, respectively. The suggested solution performs well and automatically recognizes and eliminates reflection from photos.

Fan et al. [8] suggest a deep neural network for single-picture reflection removal and image smoothing. The author suggested the Cascaded Edge and Image Learning Network (CEILNet), which can be tailored to tackle various image processing tasks like layer separation and image filtering. The authors used real photographs with backdrop layers that reflect the real world, which might be challenging to find by combining two photos with 0.8 and 0.2 different coefficients (0.8 for background and 0.2 for reflection). In the background, the relative PSNR, SSIM, NCC, and LMSE scores are 20.35, 0.7429, 0.8547, and 0.0277.

Diffuse reflection occurs when light is reflected on a rough surface to create a blurry image or when the image is not displayed at all. According to the law of reflection, the angle at which a ray reflects off a surface equals the angle at which the incident ray hits the surface. Jin et al. [9] proposed a CNbasedse approach for removing reflection from a single image. The model is evaluated on quantitative data as well as qualitative data. The results of the proposed model for PSNR, SSIM, NCC, and LMSE are 18.65, 0.7597, 0.7872, and 0.0218 on the background, respectively. The PSNR, SSIM, NCC, and LMSE results are 11.44, 0.3607, 0.4606, and 0.1150, respectively, on reflection.

A visual computing system must remove undesirable reflections from a single image via a glass window. Yang et al. [10] proposed a CNN-based approach for removing single-image reflection. The proposed algorithm estimates the background image as well as the reflection image. They use PASCAL VOC dataset images to generate synthetic data. They select two images, one for reflection and the other for background and crop them on 256 × 256 patches. From the 5717 photos in the PASCAL VOC dataset’s training set, they generate 50K training images. The PSNR, SSIM, NCC, and LMSE results are 17.08, 0.7163, 0.7669, and 0.0288 in the background, respectively.

Heo et al. [11] presented a Generative Adversal Network condition for removing image reflection. The author used the PASCAL VOC [14] dataset for the model training and a dataset size of 17,000 real-world images. After collecting the dataset, they cropped images to 256 × 256 size and then used one image for the background layer and another for the reflection layer. They used 6,000 images for training, 2000 for model validation, and 500 for testing. Two matrices, PSNR and SSIM, are used for evaluation. The values of these metrics depend on the weights of the parameters. When the values of parameters 1 = 100 and 2 = 100 then the values of PSNR and SSIM are 20.2003 and 0.7257, respectively.

Zhang et al. [12] showed how to separate reflection from a single image. The technique used a fully convolutional network with a complete training process with losses considering low-level and high-level visual input. The loss function included a feature loss from a visual perception network and an adversarial loss in the transmission layers that encode image properties. Additionally, they proposed a brand-new exclusion loss that guarantees pixel-level layer separation. They created a collection of real-world photos with reflection and matching ground-truth transmission layers for quantitative evaluation and model training. They gathered a dataset of 110 real images taken with ground truth in real-scene circumstances. According to the background, the PSNR, SSIM, NCC, and LMSE findings are 19.53, 0.7584, 0.8526, and 0.0207, respectively. On reflection, PSNR, SSIM, NCC, and LMSE yield findings of 18.69, 0.4945, 0.6283, and 0.1108, respectively.

Reflections in colour photos taken through a semi-transparent material are problematic for photographers and entertainment practitioners. This elimination is required to improve human cognition, and numerous computer vision methods will be used subsequently. In this article, Li et al. [13] showed you an automated way to remove reflective artefacts when imaging a scene behind a glass surface. They use an Optimization-based e SIFT-Flow implementation approach to remove reflection. The Input sequences range from three to five images. The input is a tiny set of k photos of the scene taken from slightly different perspectives (Indoor and outdoor). The PSNR, SSIM, NCC, and LMSE results are 17.12, 0.6367, 0.6673, and 0.0604 in the background, respectively. PSNR, SSIM, NCC, and LMSE results are 7.68, 0.2670, 0.3490, and 0.1214, respectively, on reflection removal.

Removing reflections from the video will give you a clearer, higher-quality video. This is important for both professional photographers and everyday users. Guo et al. [14] proposed an Augmented Lagrangian Multiplier-based algorithm for removing reflection from input frames. They use 15 synthesized images of 215 × 162 resolution. The PSNR, SSIM, NCC, and LMSE results were 14.58, 0.5077, 0.5802, and 0.0694 on the background. PSNR, SSIM, NCC, and LMSE results are 14.12, 0.3150, 0.3516, and 0.1774 on reflection removal.

Self-driving car cameras are often mounted behind glass windshields, resulting in reflections in the scene being shot. To solve this problem, Alayrac et al. [15] proposed a CNN-based approach. The author suggested brand-new 3D convolutional architectures for capturing uncertainty and training them to distinguish blended videos. They use real-life videos to present quantitative and qualitative results. The PSNR, SSIM, NCC, and LMSE results were 23.62, 0.7867, 0.9023, and 0.0200 on the background, respectively. PSNR, SSIM, NCC, and LMSE results are 21.18, 0.6320, 0.7535, and 0.1517, respectively, on reflection removal.

The H. Lan et al., propose a face reflection removal network (FRRN) using multispectral fusion of RGB and near-infrared (NIR) images [32]. Their objective is to improve reflection removal in face images by leveraging the different spectral properties of RGB and NIR images. The methodology involves using a deep learning framework consisting of an encoder (Contextual Encoder Module - CEM) and two decoder branches (NIR Inference Decoder Module - NIDM and Image Inference Decoder Module - IIDM). Additionally, they introduce a Reflection Confidence Generation Module (RCGM) based on Laplacian convolution and a channel attention-based residual block to estimate the reflection confidence in an image. The dataset for training and evaluation is constructed using the JAI AD-130 GE camera, capturing 10,000 synthetic image pairs and 300 test images, along with 40 real reflection images. The results demonstrate that FRRN outperforms existing reflection removal methods, achieving higher PSNR and SSIM scores while preserving facial details more effectively. However, the model’s limitations include its dependence on high-quality NIR images and the potential difficult The study titled “Single-Image Reflection Removal Using Deep Learning: A Systematic Review” by A. Amanlou et al. provides a comprehensive analysis of deep learning techniques used for reflection removal [33]. The authors systematically reviewed research papers published between 2015 and 2021, selecting 25 relevant studies from an initial pool of 1600 papers. Their methodology involved searching multiple digital libraries, applying inclusion/exclusion criteria, and conducting a quality assessment of the selected papers. The study analyzes datasets used in training and evaluation, identifying that synthetic datasets are commonly used due to the scarcity of real-world reflection removal datasets. The review highlights significant advancements in convolutional neural networks (CNNs) and generative adversarial networks (GANs) for reflection removal, achieving improved accuracy in separating reflections from background scenes. However, limitations remain, including difficulties in handling strong reflections, dependency on high-quality datasets, and challenges in real-world applications. The authors emphasize the need for more comprehensive datasets and improved algorithms to enhance the effectiveness of deep learning models in reflection removal. The study serves as a valuable resource for researchers, offering insights into current trends, challenges, and future directions in this field.y in handling extremely complex real-world reflections.

2.3. Raindrops Removing Techniques

Raindrop removal is a difficult task. Shao et al. [16] proposed a network based on Selective Skip connection GAN (SSCGAN) for raindrop removal from a single image. For better results, the author added SSCM and SAB in SSCGAN. A Dataset of 861 images is used for training purposes. They used a dataset of 861 images. They created the dataset by themselves and used Canon EOS 60 and Sonny A6000 for image acquisition. Two tests were conducted: Test A and Test B. Test A and Test B each have 58 and 249 photos, respectively. PSNR and SSIM results for Test A are 30.99 and 0.9450, respectively. Additionally, Test B yielded PSNR and SSIM readings of 25.75 and 0.8434, respectively.

Raindrops on glass windows, windshields, or lenses can obscure the visibility of background scenes and degrade the image. Eigen et al. [17] showed how to remove localized rain and soil artefacts from a single image using post-capture image processing. They proposed a convolutional neural network. They gathered a dataset of clean/corrupted image pairs and then trained a customized convolutional neural network. The pairs in the dataset are 64-pixel subregions of training images with and without corruption that are generated randomly. Using an nVidia GTX 580, their MATLAB GPU implementation can recover a 3888 × 2592 colour image in the 60s and a 1280 × 720 colour image in the 7s. In outside tests, proposed models showed that it can effectively remove dirt and rain. The results of evaluation matrices PSNR and SSIM were 28.59 and 0.6726, respectively.

The paper “MARR-GAN: Memristive Attention Recurrent Residual Generative Adversarial Network for Raindrop Removal” by Q. Chai et al. [34] introduces MARR-GAN, a model combining attention mechanisms and recurrent residual convolutional blocks to improve raindrop removal while preserving image details. It also proposes a memristor-based hardware implementation for real-time processing. The model was trained on Qian et al.’s dataset (861 image pairs) and Quan et al.’s real-world dataset (250 images). Results show MARR-GAN outperforms existing methods in PSNR and SSIM scores. Limitations include high memory requirements and dependency on high-quality training data.

The study by Y. Hamzeh et al. [35] presents a comprehensive review of techniques for detecting and removing raindrops in automotive vision systems. The authors analyze both classical image-processing methods and deep-learning approaches. Various datasets, including real-world raindrop images and synthetic data, were considered in evaluating different techniques. The study highlights the strengths and weaknesses of existing solutions, emphasizing the challenges posed by adherent raindrops on windshields. While several methods show promise, the paper notes that real-time performance and computational efficiency remain major limitations. Additionally, the need for more diverse and representative datasets is emphasized for improving algorithm robustness in real-world scenarios.

K. L. V. Satyanarayana et al. [36] proposed a raindrop removal method using a Generative Adversarial Network (GAN) combined with an Attentive Generative Adversarial Network (AGAN). Their approach enhances object detection in rainy conditions by first removing raindrops and then applying YOLO V7 for accurate detection. The dataset consisted of paired images with and without raindrops for effective model training. The method achieved a Peak Signal-to-Noise Ratio (PSNR) of 24.986 dB and a Structural Similarity Index (SSIM) of 84.56%, with detection accuracies of 96% for bicycles and 76% for persons. Limitations include its current restriction to static images and untested performance in diverse weather conditions.

The study by G. Wang et al. [37] proposes an Attentive Feature Refinement Network (AFR-Net) for single-image raindrop removal. The method incorporates a feature refinement block (FRB) and an attentive guidance block (AGB) to enhance rain removal by progressively refining features while maintaining image details. The approach is evaluated on the RD-Data for raindrop removal and the RS-Data for rainstreak removal, including both synthetic and real-world rainy images. AFR-Net achieves state-of-the-art results, significantly improving PSNR and SSIM metrics while maintaining faster inference speed through knowledge distillation. However, some challenges remain, such as occasional over- or under-deraining in complex scenarios, and further optimization may be required for real-world applications.

2.4. Shadow Removing Techniques

To restore the contents of the image in a photo-realistic manner, shadow removal, a crucial computer vision problem, must first be detected and correctly removed from an image. Jung et al. [18] suggested an efficient water-filling algorithm for digitized document shadow removal. To create a topographic surface of an input digitized document, the brightness value of each pixel was utilized first. Then, an immersion simulation calculated the shading artefact on the paper. The scanned page is reconstructed using the Lambertian surface model after the estimated shading artifact. They conducted comprehensive testing on several digitized documents produced by smartphones in poor illumination to assess the performance of the suggested algorithm. The outcomes of the PSNR and SSIM evaluation matrices are 17.04 and 0.7990, respectively. Visit https://github.com/seungjun45/Water-Filling to get the codes.

The visual quality and readability of a digital copy of a document are both improved by removing shadows from the image. The traditional document enhancement challenges are revisited in Kligler et al. [19] study. Rather than proposing a novel solution for a specific problem, Kligler et al. [19] presented a groundbreaking generic method. The fundamental idea is to feed new data into any state-of-the-art algorithm to improve it. They used 381 intentionally darkened photographs from four categories: handwritten documents, printed documents, posters, and fonts to evaluate the method. Each category contains seven to ten documents, each of which has been taken with eight to twelve distinct shadow intensities, forms, and locations. There were 81 photos derived from 11 periodicals, with small typefaces, limited colour change, and mild shadowing. The results of evaluation matrices PSNR and SSIM were 23.25 and 0.8081, respectively.

Shadows are, first and foremost, a local reduction in light reaching the surface. Second, there are local variations in the amount of light that is reflected from the surface and reflected by the viewer. The majority of shadow detection and segmentation techniques rely on picture analysis. The proposed algorithm of Bako et al. [20] automatically detected and removed annoying shadows from photos of documents and other text-based objects. The algorithm analyzed the image intensity of the local block and divided it into two clusters: background and text. It then calculated the average cluster of backgrounds and created a shadow map that would be used to produce the final result without shadows. Using a Canon 5D Mark II DSLR camera, tripod, photo lamp, and opaque objects, they captured 81 shadow images from 11 documents, each with 59 shadow intensity and shape variations. The results of evaluation matrices PSNR and SSIM were 30.01 and 0.9231, respectively.

A shadow is a dark shape that appears on a surface when a body blocks light from a source. Shadows hurt a variety of computer vision techniques, including segmentation, object detection, scene analysis, stereo, tracking, and others. Shadow reduction is a crucial pre-processing step to improve visual function. Wang et al. [21] presented Stacked Conditional Generative Adversarial Networks for jointly learning shadow detection and removal. The proposed network comprised two stacked CGANs, each with a generator and a discriminator. After receiving a shadow image, the first generator creates a shadow detection mask. The second generator receives the shadow image and its anticipated mask, producing a shadow-free image. They used the ISTD dataset for model evaluation, consisting of 1870 photographs with shadows, shadow masks, and shadow-free images. The PSNR and SSIM evaluation matrices results are 33.14 and 0.9408, respectively.

Lin et al. [22] recommended the Background Estimation Document Shadow Removal Network (BEDSR-Net) to remove shadows from single document images. Along with their shadow-free images and shadow masks, they also gave many artificial shadow images, trained their model using these images, and provided evidence that the artificial shadows continue to work effectively for real photos. The author splitted the dataset into 7,533 for training and 776 for testing. The collection contained 540 photos of 25 documents under various lighting, including paper, newspaper, and slides. This dataset is exclusively used for testing purposes. On the BEDSR-Net model, the PSNR and SSIM matrices yield 37.55 and 0.9534, respectively.

Le et al. [23] presented a fresh deep-learning technique for eliminating shadows from a single image. They use SP-Net and M-Net deep networks, respectively, to forecast shadow parameters and shadow matte. The proposed model is trained and tested using the shadow removal dataset (ISTD). Regarding RMSE (root mean square error), the model achieves a 40 percent reduction in the shadow region, dropping RMSE from 13.3 to 7.9. To alter the shadow parameters, they produced a new augmented ISTD dataset based on fake shadow photos. The RMSE on the shadow region is decreased to 7.4 when they train the model on a fresh dataset.

The presence of shadow in an image degrades its visual quality. Nowadays, shadow detection and removal are important problems that must be solved. Fan et al. [24] proposed an end-to-end deep CNN network to remove shadows from images. The proposed network consists of an encoder-decoder and a refinement network. To analyze, the proposed network author constructed a new shadow image database RSDB and used UIUC, UCF, and RSDB datasets for qualitative and quantitative evaluations. The RSDB dataset contained 2685 shadow and shadow-free image pairs. Quantitative results of SSIM on the RSDB dataset are 0.9826, and PSNR results are 25.7078 on shadow images. The proposed network provides better results for removing shadows from images than other algorithms.

When something blocks the light source, a shadow is cast. It is a problem that lowers the image quality and could make some text or other information difficult to see or read. To resolve the difficulties, the research of Fu et al. [25] provides a novel solution by rewriting the assignment as an exposure fusion problem. FusionNet with shadow support is the solution suggested by Fu et al. [25] to this problem. The author inputs a shadow image to generate a fusion weight map over all overexposed photos. To further reduce the remaining shadow trails, they also suggest a limit-aware RefineNet. The ISTD, ISTD+, and SRD datasets are used to train and assess the suggested approach in this study. On ISTD, ISTD+, and SRD, the proposed network by RMSE achieved shadow removal results of 5.92, 4.2, and 6.51, respectively. Visit https://github.com/tsingqguo/exposurefusion-shadow-removal to get the project’s source code.

The NTIRE 2024 Image Shadow Removal Challenge Report was authored by multiple researchers, including Florin-Alexandru Vasluianu, Tim Seizinger, Zhuyun Zhou, and many others [28]. The challenge aimed to advance shadow removal techniques through two tracks—one focusing on fidelity and the other on perceptual quality. The methodology involved evaluating shadow removal models based on structured similarity (SSIM), peak signal-to-noise ratio (PSNR), and perceptual metrics such as LPIPS and Mean Opinion Score (MOS). Participants used the WSRD+ dataset, which includes complex self-shadow and cast shadow interactions, with improved image alignment techniques to enhance training consistency. The results indicated that the top-performing models produced high-quality shadow-free images, with leading methods excelling in texture and color reconstruction. However, limitations included pixel misalignment challenges, sensitivity to lighting variations, and the need for extensive computational resources for deep learning models. The study emphasized the importance of robust datasets and advanced architectures to improve real-world shadow removal applications.

The paper titled “Shadow detection using a cross-attentional dual-decoder network with self-supervised image reconstruction features” was authored by R. Fernandez-Beltran et al. [29]. The objective of the study was to enhance shadow detection using a novel Cross-Attentional Dual Decoder Network (CADDN), which integrates fine-grained image reconstruction features. Unlike conventional methods, this model employs two decoders: one for reconstructing the original input image and another for generating shadow masks, ensuring a better understanding of shadowed and illuminated regions. Additionally, a cross-attention mechanism was incorporated to prioritize relevant features, and skip connections with noise were used to improve feature transfer. The model was trained and tested on several benchmark datasets, including AISD, CUHKMAP, and SBU, which contain diverse and complex shadowed scenes. Experimental results demonstrated that CADDN achieved superior performance compared to state-of-the-art methods, consistently ranking highest in Intersection over Union (IoU), F1-score, and Overall Accuracy (OA) across datasets. Despite its success, the model has certain limitations, such as its reliance on high-quality training data and computational complexity, which may affect real-time performance in practical applications. Future work aims to refine loss formulations, explore additional decoder architectures, and integrate semi-supervised learning to enhance adaptability.

The paper “ShadowRefiner: Towards Mask-free Shadow Removal via Fast Fourier Transformer” by Wei Dong et al. presents a mask-free shadow removal approach using deep learning. The proposed ShadowRefiner model integrates spatial and frequency domain representations to enhance image quality while eliminating shadows. It consists of two modules: (1) Shadow Removal Module, which employs a ConvNext-based U-Net to extract features and remove shadows, and (2) Refinement Module, utilizing a Fast-Fourier Attention Transformer (FFAT) to improve texture details and color consistency. The model was evaluated on ISTD, ISTD+, and WSRD+ datasets, significantly outperforming existing mask-free methods and achieving top rankings in the NTIRE 2024 Image Shadow Removal Challenge. However, the method relies on deep learning architectures, requiring substantial computational resources.

The study “Shadow Segmentation With Image Thresholding for Describing the Harshness of Light Sources” by V. Štampfl et al. aims to develop a novel method for quantifying light source harshness using shadow analysis [31]. Utilizing a test environment, the researchers captured images under controlled lighting conditions and applied image thresholding to segment shadows into umbra and penumbra regions. The results indicated that the method effectively described the quality of light sources, such as halogen, LED, and xenon, while remaining unaffected by light intensity. Despite its efficiency, the method is computationally intensive and may require enhancements in shadow isolation and noise reduction for broader applicability.

2.5. Haze Removing Techniques

Dust, smoke, fog, and other dry particles that obscure the sky’s transparency cause haze, which is often atmospheric. S. Kuanar et al. [26] address the single-image haze reduction issue in low-light conditions. A network that combines glow, airlight, and direct transmission attenuation was proposed by S. Kuanar et al. [26]. The author gathered photographs of nighttime haze from internal and external sources to train the model, created a transparency map using the NYU depth dataset, and then restored the haze-free image. The natural hazy image findings for the PSNR and SSIM evaluation parameters are 19.25 and 0.9958, respectively.

The study by A. R. Rani et al. introduces DLHR-TSDR, a deep learning-based model for haze removal and traffic sign detection in autonomous vehicle navigation [38]. The methodology consists of HRU-Net, which removes haze and enhances image clarity, and TSDR-CNN, which detects and classifies traffic signs in real-time. The model is trained on the CURE-TSD dataset, which contains 9,904 images covering 33 types of traffic signs captured under varying conditions. Experimental results show superior performance, with 99.01% accuracy, 99.21% precision, 99.46% recall, and 99.89% F1-score, outperforming existing methods. While effective, the model requires further real-time optimizations and could benefit from transfer learning to improve multi-class sign classification in challenging environments.

The study by Y. Liu et al. presents C2AIR, a novel, lightweight, and consolidated approach for aerial image dehazing. The methodology introduces Density Aware Query Modulation (DAQM) to handle different haze densities, Cross Collaborative Feed-Forward (CCF) Block for extracting multi-scale structural features, and Adaptive Gated Feature Fusion (AGFF) for effective feature integration. The model is trained on Sate-1K and RICE datasets, ensuring robust performance across various haze levels. Experimental results show superior SSIM and PSNR values compared to state-of-the-art methods: Sate-1K (Thin haze: SSIM = 0.9723, PSNR = 30.32; Thick haze: SSIM = 0.9413, PSNR = 27.97) and RICE (SSIM = 0.9815, PSNR = 35.21). The model offers high performance with low computational cost but requires further optimization for real-time applications and more diverse training data.

The study, conducted by Yi Liu et al., introduces HcaNet, a haze-concentration-aware network designed for real-scene image dehazing [40]. The methodology incorporates high-quality codebook priors and two novel components: HcaM, which reduces information loss in domain transformation, and FSFM, which enhances texture details. The model is trained using synthetic datasets (DIV2K, Flickr2K, RESIDE OTS) and evaluated on real-world datasets (O-Haze, Dense-Haze, NH-Haze, RTTS). Experimental results demonstrate that HcaNet surpasses state-of-the-art methods in real-scene dehazing, achieving higher SSIM and PSNR values: O-Haze (SSIM = 0.733, PSNR = 16.778), Dense-Haze (SSIM = 0.497, PSNR = 11.410), and NH-Haze (SSIM = 0.564, PSNR = 13.156). However, the method is computationally expensive, and future work aims to optimize efficiency and incorporate additional priors for better real-world adaptability.

2.6. Hybrid Techniques

Li et al. [27] proposed a method that removed rain, fog, snow, and raindrops using a single Neural Architecture Search (NAS). They trained their model on outdoor rain, snow100k, and raindrops.The dataset contains Outdoor rain contained 9K training and 1.5K validation samples Snow100 K synthetic snow images. Raindrop consisted of 1119 pairs of real raindrop images. The PSNR and SSIM matrices results on the raindrop dataset are 31.12 and 0.9268, respectively. The results of PSNR and SSIM matrices on the Snow100K dataset were 28.33 and 0.8820, respectively. The results of the entire architecture of PSNR and SSIM on the test dataset were 24.71 and 0.898, respectively.

Table 6. Summarized Techniques for Unwanted Foreground Removal.

Author	Objective and Methodology	Dataset	Results	Limitations
Gupta et al. [3]	Proposed cGANs algorithm for the removal of the fence from the image	Pascal VOC data and COCO (29413)	MOS 4.34 SSIM 0.87 LPIPS 0.17	Only remove the fence, not reflection and raindrops
Matsui et al. [4]	CNN and for fence removal from a single image	DetectNet 545 real-world fence images	PSNR 27.11± 2.58 SSIM 0.906 ± 0.034	Only remove fence
Sanakaraganesh Jonna, Moushumi Medhi, Rajiv Ranjan Sahay (IIT Kharagpur)	Automatic occlusion (fence) removal using deep learning for better image quality. Two-stage CNN: (1) Occlusion segmentation (UNet-based), (2) Image inpainting (GAN-based).	UCSD Fence, IITKGP Fence (segmentation), Places2, CelebA (inpainting).	Outperforms existing methods in precision, recall, F-measure, and MAE. Produces high-quality, artifact-free images.	Occasional false detections, high computational cost, limited training data.
Stavros Tsogkas, Fengjia Zhang, Allan Jepson, Alex Levinshtein (Samsung AI Center, Toronto)	Three-step process: (1) Fence segmentation (U-Net), (2) Optical flow estimation, (3) Flow-guided inpainting (CNN) to reconstruct a keyframe.	Synthetic Data (Vimeo-90k, De-fencing dataset) and 185 real-world bursts with ground truth backgrounds	Outperforms existing methods in quality (SSIM, PSNR, LPIPS) and runs at 7 FPS, making it much faster.	Performance depends on accurate fence segmentation and struggles with unusual fences or low contrast scenes.
Divyanshu Gupta, Shorya Jain, Utkarsh Tripathi, Pratik Chattopadhyay, Lipo Wang	Develop an image de-fencing approach using a two-stage cGAN model: one for fence mask detection and another for image reconstruction.	Public fence segmentation dataset + synthetic dataset (Pascal VOC & COCO), totaling ~29K images.	High-quality de-fenced images, MOS: 4.34, SSIM: 0.87, LPIPS: 0.17, with 51ms processing time.	Struggles with shadows, highly textured backgrounds, and extreme occlusions.
Tasi et al. [5]	The proposed methodology contains four steps: layer separation, pixel compensation, detail reconstruction, and normalization to remove reflection from a single image.	Images from Google search engine	Results on seven images are 0.426, 1.142, 4.832, 7.637, 9.096, 0.246, and 0.430 compared with [10,11,12].	Remove reflection from the image
Abiko et al. [6]	Proposed a generative adversarial network for single-image reflection removal with gradient constraints	PASCAL VOC 2012 dataset and SIR2 benchmark dataset	The results of restoration in PSNR /SSIM on the SIR2 benchmark dataset are 199 postcard images19.64/0.918, 200 solid images23.87/0.928, 55wild images 24.97/0.932	Only remove reflection not fence and raindrops
Fu et al. [7]	Proposed a deep learning method for the removal of reflection from images	Pick a dataset of more than 10000 synthetic images and 110 pairs of real images	Results of synthetic images PSNR 24.31 SSIM 0.86 Results of real images PSNR 24.73 SSIM 0.82	Only remove the reflection from the image
Fan et al. [8]	Cascaded Edge and Image Learning Network	Use Real photos with ground-truth background layers	PSNR 20.35 SSIM 0.7429 NCC 0.8547 LMSE 0.0277	The proposed network is only used for the single image reflection removal and image smoothing
Jin et al. [9]	CNN	The model is evaluated on quantitative data as well as qualitative data.	Background PSNR 18.65 SSIM 0.7597 NCC 0.7872 LMSE 0.0218 Reflection PSNR 11.44 SSIM 0.3607 NCC 0.4606 LMSE 0.1150	Only remove reflection from a single image
Yang et al. [10]	CNN	PASCAL VOC and they generated 50K training images	PSNR 17.08 SSIM 0.7163 NCC 0.7669 LMSE 0.0288	Only removing unwanted reflections from a single image
Heo et al. [11]	Generative Adversal Network	PASCAL VOC and dataset size is 17,000 images of the real world	PSNR 20.2003 SSIM 0.7257	Only removing unwanted reflections from a single image
Zhang et al. [12]	CNN	They gathered a dataset of 110 genuine photographs in natural scene situations with ground truth.	Background PSNR 19.53 SSIM 0.7584 NCC 0.8526 LMSE 0.0207 Reflection PSNR 18.69 SSIM 0.4945 NCC 0.6283 LMSE 0.1108	Only separate reflection from a single image and for the training of the model they use a small dataset
Li et al. [13]	Optimization-based e SIFT-Flow	The input is a tiny set of k photos of the scene taken from slightly different perspectives	Background PSNR 17.12 SSIM 0.6367 NCC 0.6673 LMSE 0.0604 Reflection PSNR 7.68 SSIM 0.2670 NCC 0.3490 LMSE 0.1214	Only remove reflective artifacts when imaging a scene behind a glass surface
Guo et al. [14]	Augmented Lagrangian Multiplier-based algorithm	They use 15 synthesize images of 215 × 162 resolution	Background PSNR 14.58 SSIM 0.5077 NCC 0.5802 LMSE 0.0694 Reflection PSNR 14.12 SSIM 0.3150 NCC 0.3516 LMSE 0.1774	Only removing unwanted reflection from the image and the size of the dataset is very small
Alayrac et al. [15]	CNN	They use real-life videos to present quantitative and qualitative results.	Background PSNR 23.62 SSIM 0.7867 NCC 0.9023 LMSE 0.0200 Reflection PSNR 21.18 SSIM 0.6320 NCC 0.7535 LMSE 0.1517	Only removing unwanted reflections from a set of images
H. Lan et al. [32]	Develop a face reflection removal network (FRRN) using multispectral fusion of RGB and NIR images. The network consists of an encoder (CEM) and two decoders (NIDM & IIDM) to suppress reflections. Uses a reflection confidence generation module (RCGM) to guide the process.	A custom dataset captured using JAI AD-130 GE camera, including 10,000 training images and 300 test images, plus 40 real images for evaluation.	Outperformed state-of-the-art methods in reflection removal. Improved PSNR and SSIM scores, achieving better facial detail recovery.	Performance may vary with different NIR camera sensors. Some cases of extremely strong reflections still pose challenges.
A. Amanlou et al. [33]	Conduct a systematic review of single-image reflection removal using deep learning from 2015-2021. Analyzed 1600 papers, selecting 25 based on relevance. Explores dataset availability, deep learning architectures, and evaluation metrics.	Various datasets including SIR2, synthetic datasets, and real-world datasets used for evaluation.	Identified deep learning as a dominant approach. SIR2 dataset widely used. SSIM and PSNR are common evaluation metrics. Challenges in handling strong reflections and real-world images persist.	Limited real-world datasets. Struggles with extreme reflections, overlapping layers, and outdoor scenes. Need for more automated methods and improved dataset diversity.
Shao et al. [16]	Selective Skip connection GAN for the raindrop removal from a single image	Dataset of 861 images	Results of Test_a PSNR 30.99 SSIM 0.9450 Test_b PSNR 25.75 SSIM 0.8434	Only remove raindrops
Eigen et al. .[17]	Convolutional Neural Network	They gathered a dataset of clean/corrupted image pairs	PSNR 28.59 SSIM 0.6726	Only show how to remove localized rain and soil artefacts from a single image using post-capture image processing.
Q. Chai et al. [34]	Propose MARR-GAN, a memristive attention recurrent residual generative adversarial network for raindrop removal. Uses attention gate connections, recurrent residual convolutional blocks, and a memristor-based hardware implementation.	Raindrop dataset by Qian et al. (861 training pairs, 58 Testa, 249 Testb samples) and RainDS dataset by Quan et al. (1000 high-resolution images, 250 raindrop pairs).	Outperformed state-of-the-art methods in raindrop removal. Improved PSNR and SSIM on multiple datasets, achieving superior image restoration.	Requires specialized hardware for optimal performance. Performance may vary based on raindrop complexity and lighting conditions.
Y. Hamzeh et al. [35]	Review detection and removal techniques for raindrops in automotive vision systems. Discusses classical image-processing methods and deep-learning approaches.	Various datasets including real-world automotive images affected by rain.	Identified challenges and best-performing techniques. Showed that deep learning improves raindrop removal compared to classical methods.	Real-world applications still face challenges in extreme weather conditions. Some methods struggle with small or overlapping raindrops.
K. L. V. Satyanarayana et al. [36]	Develop an object detection system under rainy conditions using YOLO V7. Utilizes a Generative Adversarial Network (GAN) to remove raindrops and enhance visibility before applying object detection.	Custom dataset with images captured under rainy conditions. Uses a raindrop dataset for training the deraining model.	Improved object detection accuracy in rainy images. Achieved PSNR of 24.986 dB and SSIM of 84.56%. Successfully detected objects like bicycles (96%) and persons (76%).	May not perform well under extreme rain conditions. Limited dataset diversity. Future work needed for video-based applications and multiple weather conditions.
G. Wang et al. [37]	Propose an Attentive Feature Refinement Network (AFR-Net) for single-image raindrop removal. Uses a feature refinement block (FRB) and an attentive guidance block (AGB) to enhance rain removal.	RD-Data (raindrop removal), RS-Data (rainstreak removal). Evaluated on synthetic and real-world rainy images.	Achieved state-of-the-art results on raindrop and rainstreak removal. Improved PSNR and SSIM. Faster inference speed with knowledge distillation.	Some over- or under-deraining issues in challenging cases. May require further optimization for real-world scenarios.
Bako et al. [20]	Shadow removal algorithm	They capture 81 shadow images from 11 documents	PSNR 30.01 SSIM 0.9231	Only removing unwanted shadows from a single image
Wang et al. [21]	Stacked Conditional Generative Adversarial Networks	They use the ISTD dataset that contained 1870 images that are shadow, shadow mask, and shadow-free	PSNR 33.14 SSIM 0.9408	Only Shadow Detection and Shadow Removal
Lin et al. [22]	Proposed the BEDSR-Net for the removal of shadows from the single document image	25 documents in 540 pictures, including slides, papers, and newspapers	PSNR 37.55 SSIM 0.9534	Only remove the shadow from the single document image
Le et al. [23]	SP-Net and M-Net deep networks	ISTD	RMSE 7.4	Only removing unwanted shadow from a single image
Fan et al. [24]	End-to-end deep CNN network	RSDB which contained 2685 shadow and shadow-free image pairs.	PSNR 25.7078 SSIM 0.9826	Only remove the shadow from a single image
Fu et al. [25]	FusionNet with shadow support	ISTD, ISTD+, and SRD datasets	RMSE results on ISTD 5.92 ISTD+ 4.2 SRD 6.51	Only remove the shadow from the image
NTIRE 2024 Challenge Organizers (F.-A. Vasluianu et al.) [28]	Evaluated state-of-the-art shadow removal models in NTIRE 2024 Challenge. Organized in two tracks: fidelity and perceptual quality. Used WSRD+ dataset with improved image alignment.	WSRD+ dataset (extended version of WSRD). Includes diverse shadow interactions and object textures.	Top solutions achieved near ground-truth quality. Improved PSNR, SSIM, and LPIPS scores. User study ranked perceptual quality.	Challenge limited to dataset scenarios. Real-world variability and extreme lighting conditions remain challenging.
R. Fernandez-Beltran et al. [29]	Propose a deep learning model (CADDN) for shadow detection using a cross-attentional dual-decoder network. Utilizes image reconstruction features and cross-attention to improve accuracy.	AISD, CUHKMAP, SBU datasets	Outperformed state-of-the-art methods in shadow detection. Achieved highest IoU, F1-score, and overall accuracy in multiple datasets.	Performance may decrease with excessive noise in encoder features. May require further tuning for highly complex shadow cases.
Wei Dong, Han Zhou, Yuqiong Tian, Jingke Sun, Xiaohong Liu, Guangtao Zhai, Jun Chen	Develop a mask-free shadow removal model (ShadowRefiner) using Fast Fourier Transformer (FFT) and a ConvNext-based U-Net. The method refines images through a Fast-Fourier Attention Transformer (FFAT) module.	ISTD, ISTD+, WSRD+ datasets. Tested on NTIRE 2024 Image Shadow Removal Challenge.	Achieved state-of-the-art performance in mask-free shadow removal. Ranked 1st in the Perceptual Track and 2nd in the Fidelity Track of NTIRE 2024. Improved texture and color consistency.	Dependent on high-quality training data. Some pixel misalignment issues. May struggle with highly complex real-world shadow variations.
V. Štampfl et al. [31]	Develop a method to describe light source harshness using shadow segmentation. Utilizes image thresholding to isolate shadows, segment them into umbral and penumbral regions, and calculate a harshness metric.	84 photographs (42 pairs) with variations in light source type, quality, and intensity. Images captured under controlled conditions.	Successfully quantified light harshness, showing different values for LED, halogen, and xenon light sources with and without light-shaping attachments. Method provided comparable numerical results.	Sensitive to image noise, requires controlled conditions, and effectiveness depends on light intensity. Further refinement needed for automation and noise reduction.
Kuanar et al. [26]	Multi-path dilated convolution network	NYU depth dataset	PSNR 19.25 SSIM 0.9958	Only discuss the single-image haze reduction problem.
A. R. Rani et al. [38]	Proposes DLHR-TSDR, a deep learning-based model for haze removal and traffic sign detection and recognition (TSDR) for autonomous vehicle navigation. The model consists of two main components: HRU-Net for haze removal and TSDR-CNN for traffic sign recognition. HRU-Net enhances image clarity, while TSDR-CNN detects and classifies traffic signs in real-time. The system aims to improve autonomous vehicle safety by providing accurate sign detection under adverse conditions.	CURE-TSD Dataset (Carleton University Retinal Eye-Traffic Sign Dataset) with 9,904 images of traffic signs captured in various lighting and weather conditions. The dataset includes 33 traffic sign types, such as regulatory, cautionary, and informatory signs.	DLHR-TSDR achieves 99.01% accuracy, outperforming traditional methods like SSD, JAAD, and HSV-based models. Performance metrics: Precision = 99.21%, Recall = 99.46%, F1-score = 99.89%, Specificity = 99.34%. The method significantly improves detection accuracy and robustness.	Requires further optimization for real-time applications. Performance may be affected by extreme weather conditions. Future work includes integrating transfer learning for enhanced multi-class traffic sign classification.
Ashutosh Kulkarni [39]	Proposes C2AIR, a lightweight and consolidated model for aerial image dehazing. Introduces Density Aware Query Modulation (DAQM) to handle haze density variations, Cross Collaborative Feed-Forward (CCF) Block for learning multi-scale features, and Adaptive Gated Feature Fusion (AGFF) for effective feature integration. The model requires only a single checkpoint for different haze densities and is computationally efficient.	Sate-1K Dataset (hazy-clean aerial image pairs with thin, moderate, and thick haze levels), RICE Dataset (real-world aerial images affected by haze). Combined training on both datasets to improve generalization.	Outperforms state-of-the-art methods. Sate-1K Dataset: Thin haze (SSIM = 0.9723, PSNR = 30.32), Moderate haze (SSIM = 0.9677, PSNR = 29.65), Thick haze (SSIM = 0.9413, PSNR = 27.97). RICE Dataset: SSIM = 0.9815, PSNR = 35.21. Achieves better dehazing quality with lower computational cost.	Although computationally efficient, further optimization is needed for real-time applications. The model could benefit from additional training on more diverse real-world datasets.
Yi Liu et al. [40]	Proposes HcaNet, a haze-concentration-aware network for real-scene image dehazing using high-quality codebook priors. Introduces HcaM to reduce information loss and FSFM to refine texture details. Uses synthetic hazy-clean image pairs for training and generalizes to real-world data.	Uses synthetic datasets (DIV2K, Flickr2K, RESIDE OTS) for training and evaluates on real-world datasets (O-Haze, Dense-Haze, NH-Haze, RTTS).	HcaNet outperforms state-of-the-art dehazing methods. O-Haze: SSIM = 0.733, PSNR = 16.778. Dense-Haze: SSIM = 0.497, PSNR = 11.410. NH-Haze: SSIM = 0.564, PSNR = 13.156. The model achieves significant improvements over previous methods in real-scene dehazing.	Computationally intensive; requires optimization for real-time applications. Future work aims to integrate more prior knowledge for better adaptation to complex environments.
Li et al. [27]	Neural Architecture Search	Outdoor rain contained 9K training samples and 1.5K validation samples. Snow100K synthetic snow images. Raindrop consists of 1119 pairs of real raindrop images.	PSNR 24.71 SSIM 0.898	Only removes rain, fog, snow, and raindrops not a fence, reflection, and shadow

Table 6. Summarized Techniques for Unwanted Foreground Removal.

Author	Objective and Methodology	Dataset	Results	Limitations
Gupta et al. [3]	Proposed cGANs algorithm for the removal of the fence from the image	Pascal VOC data and COCO (29413)	MOS 4.34 SSIM 0.87 LPIPS 0.17	Only remove the fence, not reflection and raindrops
Matsui et al. [4]	CNN and for fence removal from a single image	DetectNet 545 real-world fence images	PSNR 27.11± 2.58 SSIM 0.906 ± 0.034	Only remove fence
Sanakaraganesh Jonna, Moushumi Medhi, Rajiv Ranjan Sahay (IIT Kharagpur)	Automatic occlusion (fence) removal using deep learning for better image quality. Two-stage CNN: (1) Occlusion segmentation (UNet-based), (2) Image inpainting (GAN-based).	UCSD Fence, IITKGP Fence (segmentation), Places2, CelebA (inpainting).	Outperforms existing methods in precision, recall, F-measure, and MAE. Produces high-quality, artifact-free images.	Occasional false detections, high computational cost, limited training data.
Stavros Tsogkas, Fengjia Zhang, Allan Jepson, Alex Levinshtein (Samsung AI Center, Toronto)	Three-step process: (1) Fence segmentation (U-Net), (2) Optical flow estimation, (3) Flow-guided inpainting (CNN) to reconstruct a keyframe.	Synthetic Data (Vimeo-90k, De-fencing dataset) and 185 real-world bursts with ground truth backgrounds	Outperforms existing methods in quality (SSIM, PSNR, LPIPS) and runs at 7 FPS, making it much faster.	Performance depends on accurate fence segmentation and struggles with unusual fences or low contrast scenes.
Divyanshu Gupta, Shorya Jain, Utkarsh Tripathi, Pratik Chattopadhyay, Lipo Wang	Develop an image de-fencing approach using a two-stage cGAN model: one for fence mask detection and another for image reconstruction.	Public fence segmentation dataset + synthetic dataset (Pascal VOC & COCO), totaling ~29K images.	High-quality de-fenced images, MOS: 4.34, SSIM: 0.87, LPIPS: 0.17, with 51ms processing time.	Struggles with shadows, highly textured backgrounds, and extreme occlusions.
Tasi et al. [5]	The proposed methodology contains four steps: layer separation, pixel compensation, detail reconstruction, and normalization to remove reflection from a single image.	Images from Google search engine	Results on seven images are 0.426, 1.142, 4.832, 7.637, 9.096, 0.246, and 0.430 compared with [10,11,12].	Remove reflection from the image
Abiko et al. [6]	Proposed a generative adversarial network for single-image reflection removal with gradient constraints	PASCAL VOC 2012 dataset and SIR2 benchmark dataset	The results of restoration in PSNR /SSIM on the SIR2 benchmark dataset are 199 postcard images19.64/0.918, 200 solid images23.87/0.928, 55wild images 24.97/0.932	Only remove reflection not fence and raindrops
Fu et al. [7]	Proposed a deep learning method for the removal of reflection from images	Pick a dataset of more than 10000 synthetic images and 110 pairs of real images	Results of synthetic images PSNR 24.31 SSIM 0.86 Results of real images PSNR 24.73 SSIM 0.82	Only remove the reflection from the image
Fan et al. [8]	Cascaded Edge and Image Learning Network	Use Real photos with ground-truth background layers	PSNR 20.35 SSIM 0.7429 NCC 0.8547 LMSE 0.0277	The proposed network is only used for the single image reflection removal and image smoothing
Jin et al. [9]	CNN	The model is evaluated on quantitative data as well as qualitative data.	Background PSNR 18.65 SSIM 0.7597 NCC 0.7872 LMSE 0.0218 Reflection PSNR 11.44 SSIM 0.3607 NCC 0.4606 LMSE 0.1150	Only remove reflection from a single image
Yang et al. [10]	CNN	PASCAL VOC and they generated 50K training images	PSNR 17.08 SSIM 0.7163 NCC 0.7669 LMSE 0.0288	Only removing unwanted reflections from a single image
Heo et al. [11]	Generative Adversal Network	PASCAL VOC and dataset size is 17,000 images of the real world	PSNR 20.2003 SSIM 0.7257	Only removing unwanted reflections from a single image
Zhang et al. [12]	CNN	They gathered a dataset of 110 genuine photographs in natural scene situations with ground truth.	Background PSNR 19.53 SSIM 0.7584 NCC 0.8526 LMSE 0.0207 Reflection PSNR 18.69 SSIM 0.4945 NCC 0.6283 LMSE 0.1108	Only separate reflection from a single image and for the training of the model they use a small dataset
Li et al. [13]	Optimization-based e SIFT-Flow	The input is a tiny set of k photos of the scene taken from slightly different perspectives	Background PSNR 17.12 SSIM 0.6367 NCC 0.6673 LMSE 0.0604 Reflection PSNR 7.68 SSIM 0.2670 NCC 0.3490 LMSE 0.1214	Only remove reflective artifacts when imaging a scene behind a glass surface
Guo et al. [14]	Augmented Lagrangian Multiplier-based algorithm	They use 15 synthesize images of 215 × 162 resolution	Background PSNR 14.58 SSIM 0.5077 NCC 0.5802 LMSE 0.0694 Reflection PSNR 14.12 SSIM 0.3150 NCC 0.3516 LMSE 0.1774	Only removing unwanted reflection from the image and the size of the dataset is very small
Alayrac et al. [15]	CNN	They use real-life videos to present quantitative and qualitative results.	Background PSNR 23.62 SSIM 0.7867 NCC 0.9023 LMSE 0.0200 Reflection PSNR 21.18 SSIM 0.6320 NCC 0.7535 LMSE 0.1517	Only removing unwanted reflections from a set of images
H. Lan et al. [32]	Develop a face reflection removal network (FRRN) using multispectral fusion of RGB and NIR images. The network consists of an encoder (CEM) and two decoders (NIDM & IIDM) to suppress reflections. Uses a reflection confidence generation module (RCGM) to guide the process.	A custom dataset captured using JAI AD-130 GE camera, including 10,000 training images and 300 test images, plus 40 real images for evaluation.	Outperformed state-of-the-art methods in reflection removal. Improved PSNR and SSIM scores, achieving better facial detail recovery.	Performance may vary with different NIR camera sensors. Some cases of extremely strong reflections still pose challenges.
A. Amanlou et al. [33]	Conduct a systematic review of single-image reflection removal using deep learning from 2015-2021. Analyzed 1600 papers, selecting 25 based on relevance. Explores dataset availability, deep learning architectures, and evaluation metrics.	Various datasets including SIR2, synthetic datasets, and real-world datasets used for evaluation.	Identified deep learning as a dominant approach. SIR2 dataset widely used. SSIM and PSNR are common evaluation metrics. Challenges in handling strong reflections and real-world images persist.	Limited real-world datasets. Struggles with extreme reflections, overlapping layers, and outdoor scenes. Need for more automated methods and improved dataset diversity.
Shao et al. [16]	Selective Skip connection GAN for the raindrop removal from a single image	Dataset of 861 images	Results of Test_a PSNR 30.99 SSIM 0.9450 Test_b PSNR 25.75 SSIM 0.8434	Only remove raindrops
Eigen et al. .[17]	Convolutional Neural Network	They gathered a dataset of clean/corrupted image pairs	PSNR 28.59 SSIM 0.6726	Only show how to remove localized rain and soil artefacts from a single image using post-capture image processing.
Q. Chai et al. [34]	Propose MARR-GAN, a memristive attention recurrent residual generative adversarial network for raindrop removal. Uses attention gate connections, recurrent residual convolutional blocks, and a memristor-based hardware implementation.	Raindrop dataset by Qian et al. (861 training pairs, 58 Testa, 249 Testb samples) and RainDS dataset by Quan et al. (1000 high-resolution images, 250 raindrop pairs).	Outperformed state-of-the-art methods in raindrop removal. Improved PSNR and SSIM on multiple datasets, achieving superior image restoration.	Requires specialized hardware for optimal performance. Performance may vary based on raindrop complexity and lighting conditions.
Y. Hamzeh et al. [35]	Review detection and removal techniques for raindrops in automotive vision systems. Discusses classical image-processing methods and deep-learning approaches.	Various datasets including real-world automotive images affected by rain.	Identified challenges and best-performing techniques. Showed that deep learning improves raindrop removal compared to classical methods.	Real-world applications still face challenges in extreme weather conditions. Some methods struggle with small or overlapping raindrops.
K. L. V. Satyanarayana et al. [36]	Develop an object detection system under rainy conditions using YOLO V7. Utilizes a Generative Adversarial Network (GAN) to remove raindrops and enhance visibility before applying object detection.	Custom dataset with images captured under rainy conditions. Uses a raindrop dataset for training the deraining model.	Improved object detection accuracy in rainy images. Achieved PSNR of 24.986 dB and SSIM of 84.56%. Successfully detected objects like bicycles (96%) and persons (76%).	May not perform well under extreme rain conditions. Limited dataset diversity. Future work needed for video-based applications and multiple weather conditions.
G. Wang et al. [37]	Propose an Attentive Feature Refinement Network (AFR-Net) for single-image raindrop removal. Uses a feature refinement block (FRB) and an attentive guidance block (AGB) to enhance rain removal.	RD-Data (raindrop removal), RS-Data (rainstreak removal). Evaluated on synthetic and real-world rainy images.	Achieved state-of-the-art results on raindrop and rainstreak removal. Improved PSNR and SSIM. Faster inference speed with knowledge distillation.	Some over- or under-deraining issues in challenging cases. May require further optimization for real-world scenarios.
Bako et al. [20]	Shadow removal algorithm	They capture 81 shadow images from 11 documents	PSNR 30.01 SSIM 0.9231	Only removing unwanted shadows from a single image
Wang et al. [21]	Stacked Conditional Generative Adversarial Networks	They use the ISTD dataset that contained 1870 images that are shadow, shadow mask, and shadow-free	PSNR 33.14 SSIM 0.9408	Only Shadow Detection and Shadow Removal
Lin et al. [22]	Proposed the BEDSR-Net for the removal of shadows from the single document image	25 documents in 540 pictures, including slides, papers, and newspapers	PSNR 37.55 SSIM 0.9534	Only remove the shadow from the single document image
Le et al. [23]	SP-Net and M-Net deep networks	ISTD	RMSE 7.4	Only removing unwanted shadow from a single image
Fan et al. [24]	End-to-end deep CNN network	RSDB which contained 2685 shadow and shadow-free image pairs.	PSNR 25.7078 SSIM 0.9826	Only remove the shadow from a single image
Fu et al. [25]	FusionNet with shadow support	ISTD, ISTD+, and SRD datasets	RMSE results on ISTD 5.92 ISTD+ 4.2 SRD 6.51	Only remove the shadow from the image
NTIRE 2024 Challenge Organizers (F.-A. Vasluianu et al.) [28]	Evaluated state-of-the-art shadow removal models in NTIRE 2024 Challenge. Organized in two tracks: fidelity and perceptual quality. Used WSRD+ dataset with improved image alignment.	WSRD+ dataset (extended version of WSRD). Includes diverse shadow interactions and object textures.	Top solutions achieved near ground-truth quality. Improved PSNR, SSIM, and LPIPS scores. User study ranked perceptual quality.	Challenge limited to dataset scenarios. Real-world variability and extreme lighting conditions remain challenging.
R. Fernandez-Beltran et al. [29]	Propose a deep learning model (CADDN) for shadow detection using a cross-attentional dual-decoder network. Utilizes image reconstruction features and cross-attention to improve accuracy.	AISD, CUHKMAP, SBU datasets	Outperformed state-of-the-art methods in shadow detection. Achieved highest IoU, F1-score, and overall accuracy in multiple datasets.	Performance may decrease with excessive noise in encoder features. May require further tuning for highly complex shadow cases.
Wei Dong, Han Zhou, Yuqiong Tian, Jingke Sun, Xiaohong Liu, Guangtao Zhai, Jun Chen	Develop a mask-free shadow removal model (ShadowRefiner) using Fast Fourier Transformer (FFT) and a ConvNext-based U-Net. The method refines images through a Fast-Fourier Attention Transformer (FFAT) module.	ISTD, ISTD+, WSRD+ datasets. Tested on NTIRE 2024 Image Shadow Removal Challenge.	Achieved state-of-the-art performance in mask-free shadow removal. Ranked 1st in the Perceptual Track and 2nd in the Fidelity Track of NTIRE 2024. Improved texture and color consistency.	Dependent on high-quality training data. Some pixel misalignment issues. May struggle with highly complex real-world shadow variations.
V. Štampfl et al. [31]	Develop a method to describe light source harshness using shadow segmentation. Utilizes image thresholding to isolate shadows, segment them into umbral and penumbral regions, and calculate a harshness metric.	84 photographs (42 pairs) with variations in light source type, quality, and intensity. Images captured under controlled conditions.	Successfully quantified light harshness, showing different values for LED, halogen, and xenon light sources with and without light-shaping attachments. Method provided comparable numerical results.	Sensitive to image noise, requires controlled conditions, and effectiveness depends on light intensity. Further refinement needed for automation and noise reduction.
Kuanar et al. [26]	Multi-path dilated convolution network	NYU depth dataset	PSNR 19.25 SSIM 0.9958	Only discuss the single-image haze reduction problem.
A. R. Rani et al. [38]	Proposes DLHR-TSDR, a deep learning-based model for haze removal and traffic sign detection and recognition (TSDR) for autonomous vehicle navigation. The model consists of two main components: HRU-Net for haze removal and TSDR-CNN for traffic sign recognition. HRU-Net enhances image clarity, while TSDR-CNN detects and classifies traffic signs in real-time. The system aims to improve autonomous vehicle safety by providing accurate sign detection under adverse conditions.	CURE-TSD Dataset (Carleton University Retinal Eye-Traffic Sign Dataset) with 9,904 images of traffic signs captured in various lighting and weather conditions. The dataset includes 33 traffic sign types, such as regulatory, cautionary, and informatory signs.	DLHR-TSDR achieves 99.01% accuracy, outperforming traditional methods like SSD, JAAD, and HSV-based models. Performance metrics: Precision = 99.21%, Recall = 99.46%, F1-score = 99.89%, Specificity = 99.34%. The method significantly improves detection accuracy and robustness.	Requires further optimization for real-time applications. Performance may be affected by extreme weather conditions. Future work includes integrating transfer learning for enhanced multi-class traffic sign classification.
Ashutosh Kulkarni [39]	Proposes C2AIR, a lightweight and consolidated model for aerial image dehazing. Introduces Density Aware Query Modulation (DAQM) to handle haze density variations, Cross Collaborative Feed-Forward (CCF) Block for learning multi-scale features, and Adaptive Gated Feature Fusion (AGFF) for effective feature integration. The model requires only a single checkpoint for different haze densities and is computationally efficient.	Sate-1K Dataset (hazy-clean aerial image pairs with thin, moderate, and thick haze levels), RICE Dataset (real-world aerial images affected by haze). Combined training on both datasets to improve generalization.	Outperforms state-of-the-art methods. Sate-1K Dataset: Thin haze (SSIM = 0.9723, PSNR = 30.32), Moderate haze (SSIM = 0.9677, PSNR = 29.65), Thick haze (SSIM = 0.9413, PSNR = 27.97). RICE Dataset: SSIM = 0.9815, PSNR = 35.21. Achieves better dehazing quality with lower computational cost.	Although computationally efficient, further optimization is needed for real-time applications. The model could benefit from additional training on more diverse real-world datasets.
Yi Liu et al. [40]	Proposes HcaNet, a haze-concentration-aware network for real-scene image dehazing using high-quality codebook priors. Introduces HcaM to reduce information loss and FSFM to refine texture details. Uses synthetic hazy-clean image pairs for training and generalizes to real-world data.	Uses synthetic datasets (DIV2K, Flickr2K, RESIDE OTS) for training and evaluates on real-world datasets (O-Haze, Dense-Haze, NH-Haze, RTTS).	HcaNet outperforms state-of-the-art dehazing methods. O-Haze: SSIM = 0.733, PSNR = 16.778. Dense-Haze: SSIM = 0.497, PSNR = 11.410. NH-Haze: SSIM = 0.564, PSNR = 13.156. The model achieves significant improvements over previous methods in real-scene dehazing.	Computationally intensive; requires optimization for real-time applications. Future work aims to integrate more prior knowledge for better adaptation to complex environments.
Li et al. [27]	Neural Architecture Search	Outdoor rain contained 9K training samples and 1.5K validation samples. Snow100K synthetic snow images. Raindrop consists of 1119 pairs of real raindrop images.	PSNR 24.71 SSIM 0.898	Only removes rain, fog, snow, and raindrops not a fence, reflection, and shadow

Detecting unwanted foreground like fences, fog, shadows, haze, and reflections in images is a tough task that hasn’t been explored much. Most methods focus on removing these elements rather than detecting them and usually handle only one type at a time. Detecting things like reflections or semi-transparent fences is hard because they blend with the background. Also, there aren’t enough good datasets to train detection models, and many algorithms are too slow for real-time use. Research needs to focus on detecting multiple unwanted elements at once, working well in different conditions, and creating faster, simpler models for devices like phones. Better tools and evaluation methods are also needed to improve image quality in areas like photography and surveillance.

3. Proposed Unwanted Foreground Detection Technique

Separating an image into the background and unwanted foreground layers is challenging because it requires resolving two intimately coupled problems: optical flow decomposition and layer reconstruction. There is no starting point because we lack data for both flows and layers due to the nature of the chicken-and-egg scenario. To address the problems, we suggest a deep learning-based solution in this study that targets four main types of foregrounds: a fence, raindrops, shadow, fog, and smog. The block diagram of the proposed system is shown in Figure 4.

3.1. Dataset Acquisition

The data is taken from online repositories and self-captured images to develop a system. The online repositories include GitHub and Kaggle. The ISTD Dataset contains 1870 images with shadow as the foreground. The sample images from the ISTD dataset, shown in Figure 5, are used for shadow.

The Weather Image Recognition Dataset, which contains 526 images as depicted in Figure 6, is used for rain. This dataset is downloaded from Kaggle, comprises eleven folders, including fog smog, rain, dew, forest, glaze, hail, lightning, rainbow, rime, Sandstrom, and snow. Only two folders from this dataset, rain and fog smog, are utilized. Their snapshot is shown below. figures.

The Weather Image Recognition Dataset, containing 851 images as shown in Figure 7, is used for fog. This dataset has been downloaded from Kaggle.

A de-fencing dataset containing 645 images, as shown in Figure 8, is used for the fence. This dataset has been downloaded from GitHub.

The total number of images of fog smog, rain, shadow, and fence dataset is 3,892.

We also created our dataset with ground-truth foreground and background layers, as shown in Figure 9. There are 465 images of the real dataset with ground-truth foreground and background layers. The additional material contains more information regarding the development of synthetic data.

Table 7 summarizes the number of images acquired against each type of unwanted foreground. The dataset comprises 4357 images, of which 3892 are taken from online repositories, and self captures 465.

3.2. Pre-Processing

3.2.1. Dataset Resizing

In this work, images are resized to 100 x 100 to reduce the time needed to train the model without significantly affecting model performance. It reduces computational load and memory usage, accelerates training, and avoids overfitting for image resizing. Uniform size aids in efficient batch processing and enhances model performance by providing a consistent input scale for feature extraction and learning.

3.2.2. Data Augmentation

Data augmentation artificially increases the training set by creating modified copies of a dataset using existing data. This work uses different techniques including rotation, shear, zoom, horizontal flip, and brightness. Several libraries, such as torch and torchvision, are used for data augmentation. Torch is a machine learning library, scripting language, and open-source scientific computing system based on the Lua computer language. The torch-vision package also includes widely used datasets, model architectures, common picture changes for computer vision, and tools for improving and modifying visuals and creating compositions that can be transformed programmatically.

Figure 10. Data augmentation techniques.

Figure 10. Data augmentation techniques.

Figure 11. Images after augmentation.

Figure 11. Images after augmentation.

After augmentation, the data size is increased from 4357 to 7027. Table 8 illustrates the number of images against each class before and after data augmentation.

3.3. VGG16

In this work, the VGG16 CNN model detects unwanted foreground. VGG16 is an object identification and classification algorithm that can classify 1000 pictures into 1000 different categories with an accuracy rating of 92.7%. It is a popular method for categorizing photographs and is easy to use with transfer learning. Sometimes referred to as ConvNets, convolutional neural networks are a subtype of artificial neural networks. An input layer, an output layer, and multiple hidden layers make up a convolutional neural network. One of the finest computer vision models currently available is the Convolutional Neural Network (CNN) variant known as VGG16. By analyzing the networks and improving the depth with a design that uses incredibly tiny (3 × 3) convolution filters, our model demonstrated a considerable improvement over the state-of-the-art setups. 138 trainable parameters were generated when the depth was increased to 16–19 weight layers. The block diagram of VGG16 is shown in Figure [11].

Figure 11. VGG-16 Architecture.

Figure 11. VGG-16 Architecture.

3.4. Detection

Different libraries like TensorFlow, matplotlib, and TensorFlow are used to detect unwanted foregrounds. Google developed and made TensorFlow, a Python library for quick numerical calculations, accessible. Because of Pyplot’s capabilities, Matplotlib can perform tasks that MATLAB can. Every pyplot function modifies a figure in some way, whether through the construction of a figure, including a plotting region or lines inside a plotting area, naming the plot, etc. Because of Pyplot’s capabilities, Matplotlib can perform tasks that MATLAB can. Every pyplot function modifies a figure in some way, whether through the construction of a figure, including a plotting region or lines inside a plotting area, naming the plot, etc.

3.5. Evaluation Parameters

To access the performance several evaluation parameters are used:

3.5.1. Accuracy

The percentage of correctly categorized data instances overall data instances is known as accuracy.

$$\mathrm{Accuracy}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}},$$

(7)

3.5.2. F1 Score

The F1 score is a weighted average of recall and accuracy. Accuracy and recall can both provide false positive and false negative findings; as a result, both are taken into consideration. Most of the time, the F1 score is more valuable than accuracy, especially if your class has an unequal distribution of students.

$$\mathrm{F}1 \mathrm{score}={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}},$$

(8)

3.5.3. Precision

How often accurate positive predictions are produced is a measure of precision (true positives).

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}},$$

(9)

3.5.4. Recall

The recall is the proportion of outcomes properly predicted to all hypotheses. It is sometimes referred to as specificity or sensitivity.

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}.$$

(10)

4. Experiments Analysis

Training and testing are performed on Google Colab. Google Colab is a Google product (free of cost), and its primary purpose is research. Colab allows researchers to write and execute their Python code through the browser. Colab is well suited to machine learning, deep learning, data analysis, and education.

The dataset is split into 80% for training and 20% for testing. Table 9 lists the sample of each class used for training and testing.

The number of epochs is set to 15. Figure 12 shows the outcomes of several epochs of the training process.

Figure 13. shows the training and validation accuracy graphs. The graph shows an accuracy above 97% with 0 loss.

Figure 14 shows the confusion matrix of the proposed system. It indicates that out of 248 images of rain, the model detects 237 accurately. One image of rain was detected as a fog image, one as a fence image, and four images of rain were detected as shadow images. It also shows that out of 357 fog images, the proposed model accurately detects 346 images. Seven images are classified as rain images, one as fence images, and three as shadow images. The number of fence images is shown in row 3 of the confusion matrix. The confusion matrix accurately detects 280 images as fence images. Out of 267 fence images, four images of fences were detected as rain images, 7 fence images were detected as fog images and two images of rain were detected as shadow images. The last row of the confusion matrix shows a total number of shadow images, 518. Our model detects 508 shadow images as shadow images, six as rain images, two as fog images, and two shadow images as fence images.

Table 10 shows the proposed system’s class-wise experimental results by depicting our experiment’s accuracy, precision, recall, and F1 score values. Rain, fog, fence, and shadow accuracy values were 96.08%, 97.48%, 98.04%, and 96.92% respectively. Overall accuracy is 94.26%.

Figure 16 shows the classification of the dataset. The dataset is classified into four classes: Rain, Fog, Shadow, and Fence

5. Conclusions

Detection of the unwanted foreground is a challenging task in image processing. This research paper presents an automatic unwanted foreground detection system based on deep learning to automate this process. The proposed technique is based on the VGG16 model. We have collected datasets from online repositories and real-world images for experimental analysis and model training. The experimental results indicate that the proposed system performs unwanted foreground detection with 97.13%. The unwanted foreground detection accuracy for rain, fog, fence, and shadow is 96.08%, 97.48%, 98.04%, and 96.92%, respectively. This work can be extended for more types of unwanted types, such as detection and removal of fences and reflection. Moreover, it can be integrated into security surveillance cameras for better image/video capturing in different weather conditions.