ISLS: An Illumination-aware Sauce-packet Leakage Segmentation Method

1. Introduction

With the advancement of computer vision technology, leakage segmentation has become crucial for intelligent industrial production [1,2,3]. Sauce-packet leakage segmentation is to determine whether a sauce-packet has leakage [4]. Unlike other segmentation scenarios, sauce-packet leakage segmentation faces issues of overexposure or insufficient illumination, resulting in blurred images. Moreover, traditional and most deep learning algorithms have insufficient performance in solving illumination imbalance [5,6,7]. The sauce-packet’s leakage results in many adverse outcomes, encompassing compromised product quality and increased material wastage. Hence, to facilitate the industrialization process, researching leakage segmentation for sauce-packet leakage is particularly important.

The leakage segmentation of sauce-packet can be divided into traditional methods and Convolutional Neural Network (CNN) methods [8]. Traditional methods rely on expert experience. Songming et al. [9] developed an improved detector using the Canny operator, which improve the computational efficiency and increase the precision of fibre identification. Sharma et al. [10] used a Histogram of Oriented Gradient (HOG) and Support Vector Machine (SVM) segmentation method, which integrate with a modified ResNet50 model for brain tumor detection to help clinicians. Similarly, Hongbin et al. [11] proposed a segmentation method by HOG and Local Binary Pattern (LBP), which combines both HOG and LBP features to accurately identify crack anomalies. Binwu et al. [12] developed a secondary template matching method, which extracted the Region of Interest (ROI) by using the four-threshold algorithm. However, traditional methods relied on hand-crafted features, and the generalization performance in real scenes is insufficient.

The CNN methods automatically extract discriminative features, and do not rely on expert experience. Wang et al. [13] proposed HRNet, which connected the high-to-low resolution convolution streams in parallel and repeatedly exchanges the information across resolutions. Yu et al. [14] proposed BiseNetv2, which involved a detail branch and a semantic branch. Xie et al. [15] proposed SegFormer, which unified Transformers with lightweight multilayer perception decoders. Cao et al. [16] proposed Swin-Unet, which designed a novel pure transformer-based U-shaped encoder-decoder for medical image segmentation. However, the above methods have not been applied to the field of sauce-packet leakage segmentation.

In this paper, we propose the two stage Illumination-aware Sauce-packet Leakage Segmentation (ISLS) method. Firstly, in the illumination-aware region enhancement stage, an efficient localization algorithm [17] is introduced to reduce the calculation of invalid areas. And we design the Uneven-Light Image Enhancement (ULIE) method to alleviate the problems of blurred images under uneven illumination conditions. Specifically, the ULIE method is built upon the retinex model to enhance image clarity under insufficient illumination condition. And the ULIE method utilizes the contrast limited adaptive histogram equalization to alleviate the leakage details in overexposure image. Secondly, in the leakage region segmentation stage, to effectively relieve the problem of information missing, we propose Multi-scale Feature Fusion Module (MFFM) for capturing multi-scale discriminative representation. Our MFFM fuses a variety of feature maps from top to bottom. And the resulting fused feature map serves as the input to our proposed network decoder, thereby enhancing the decoder’s semantic recovery performance. Finally, the Sequential Self-Attention Mechanism (SSAM) utilizes a sequential structure, which combines the channel and spatial attention mechanisms, thereby achieving effective mining of salience information. In summary, our method effectively relieves the impact of uneven illumination and improves the performance of feature extraction for blurred images.

Our main contributions can be summarized as follows:

1) To alleviate blurred image issues caused by uneven illumination, we propose the ULIE method via illumination-aware mechanism to enhance the texture details of leakage within the ROI.

2) The MFFM is proposed to fuse multi-level features with a small number of parameters, capturing multi-scale features to effectively relieve the issue of missing information.

3) To alleviate the interference of invalid information, we introduce the SSAM by combining spatial and channel attention mechanisms to enhance the discriminability of valid features in the ROI.

4) We generate a sauce-packet dataset to facilitate research. Furthermore, our method Mean Intersection over Union (mIoU) achieves 80.8% and Mean Pixel Accuracy (mPA) reaches 90.1% on the self-built dataset, which are +0.9% and +0.9% higher than the previous CNN method [18].

The rest of this paper is organized as follows. Section 2 describes the proposed method in detail. Section 3 conducts the experiments based on the self-built dataset to verify the effectiveness of the proposed method. Section 4 summarizes this paper.

2. Materials and Methods

We propose Illumination-aware Sauce-packet Leakage Segmentation (ISLS) for the industrial production line, as illustrated in Figure 1. The ISLS method includes a NVIDIA GPU and an intelligent sensor with a hardware accelerated EdgeAI development environment. We train the model on NVIDIA GPU and perform inference on intelligent sensors.

The cameras capture images of sauce-packets and transfer the images to NVIDIA GPU and intelligent sensors, where the images are stored as a raw dataset. The sauce-packet dataset is pre-processed using YOLO-Fastestv2 [17] detection algorithm to focus on the Region of Interest (ROI) and is augmented by our proposed Uneven-Light Image Enhancement (ULIE) method. We split the raw dataset into training and validation sets in a 7:3 ratio.

We train our proposed leakage segmentation network on the NVIDIA GPU. Subsequently, we assess the robustness and generalization of our trained model by cross-validation. The trained model is deployed on the NVIDIA Jetson TX2 intelligent sensors for inference, where it is utilized to identify real-time leakage in images (i.e., highlight the areas of leakage with bounding boxes).

We first introduce our uneven-light image enhancement in our designed ISLS, including the localization and ULIE method. Subsequently, we present the segmentation network, proposing the Multi-scale Feature Fusion Module (MFFM) and the Sequential Self-Attention Mechanisms (SSAM). Detailed explanations of these components will be presented in subsequent sections.

2.1. Uneven-Light Image Enhancement for Illumination-Aware Region Enhancement

The actual leakage segmentation of sauce-packet is often influenced by uneven-light sources, which consist of insufficient illumination and overexposure. To relieve the problem of image blurring caused by uneven illumination, we propose the Uneven-Light Image Enhancement (ULIE) method, employed in the illumination-aware region enhancement stage of ISLS. The ULIE method is inspired by the relevant image enhancement algorithms [19,20,21]. Our ULIE method can enhance the illumination of sauce-packet images under insufficient illumination conditions and improve the image contrast and texture details under overexposure conditions.

The input of ISLS is in a three-channel RGB format, where R, G, and B represent the color space values of red, green, and blue, respectively. We utilize the mean function in OpenCV to calculate the mean value of the RGB three channels in the ROI. Through extensive experimental analysis, we define 115 and 180 as the thresholds for insufficient illumination and overexposure, respectively. The implementation details of our ULIE method are as follows:

In the case of insufficient illumination, the ULIE method is built upon the retinex model [22,23]. The retinex model theory posits that a color image can be decomposed into two primary components: the illumination component (lighting) and the reflection component, as shown in Equation (1).

Li(x) = Tr(x) ◦ Re(x)

(1)

where Li(x) and Re(x) represent the input image and the image to be recovered, respectively. Tr(x) represents the illumination mapping image, and the ◦ operator represents the element-wise multiplication.

Firstly, to simplify the computation of ULIE method, it is commonly assumed that the three channels of images share the same illumination map [24]. The ULIE method calculates the maximum value among the RGB channels of the image to independently estimate the illumination of each pixel x, obtaining the initial estimation:

$$\mathrm{Tr}\left(\mathrm{x}\right)=\underset{\mathrm{c}\in \text{{}\mathrm{R},\mathrm{G},\mathrm{B}\text{}}}{\max }{\mathrm{Li}}^{\mathrm{c}}\left(\mathrm{x}\right)$$

(2)

where x represents individual pixel, c represents channels, and $L{i}^c\left(x\right)$ is the input image of the maximum channel in the RGB.

Secondly, to ensure that the illumination map does not cause the enhanced image to become overly saturated, the ULIE method modifies Re(x):

$$\mathrm{Re}\left(\mathrm{x}\right)=\frac{\mathrm{Li}\left(\mathrm{x}\right)}{\mathrm{Tr}\left(\mathrm{x}\right)+ϵ}$$

(3)

where $ϵ$ is a very small constant, to avoid denominating to zero.

Thirdly, the ULIE method employs the augmented lagrangian multiplier optimization method [25] to preserve the structural information and smooth texture details of sauce-packet images. The ULIE method introduces the following optimization problem to accelerate the processing speed of sauce-packet images:

$$\underset{Tr}{\min }{\left|\left|\widehat{Tr}-Tr\right|\right|}_F^2+\alpha {\left|\left|W\circ ▽\mathrm{Tr}\right|\right|}_1$$

(4)

where $\left|\right|$·$|{|}_F$ and $\left|\right|$·$|{|}_1$ represent the F norm and L1 regularization, $\alpha$ is the coefficient balancing the F norm and L1 regularization, respectively. Additionally, W is the weight matrix, and $▽\mathrm{Tr}$ represents a first-order derivative filter, encompassing both horizontal and vertical directions.

Finally, the ULIE method iteratively updates according to the retinex model, solving to obtain the result image Re(x) in Equation (1). The ULIE method applies BM3D [26] for denoising optimization of the result image Re(x). To reduce the computation of the denoising process in ULIE, the method transforms the RGB three channels of the result image Re(x) into YUV three channels [27] and performs denoising only on the Y channel:

Y = 0.299R + 0.587G + 0.114B
U = -0.169R - 0.331G + 0.5B + 128
V = 0.5R - 0.419G - 0.081B + 128

(5)

where Y represents luminance, U and V represent blue chrominance and red chrominance, respectively.

In the case of overexposure, the ULIE method divides the image into blocks to obtain overexposure regions (i.e., locally overexposed areas). Firstly, to obtain the illumination information of ROI, we convert the RGB color space into the YUV color space, as shown in Equation (5). The ULIE method divides the input image into several small blocks, and performs Contrast Limited Adaptive Histogram Equalization (CLAHE) [28] on each block, to enhance the clarity of the image. CLAHE clips and redistributes the histograms of each sub-image, thereby limiting the degree of contrast enhancement. CLAHE prevents the amplification of noise and excessive enhancement [29]. The ULIE method initially divides the original image into several non-overlapping sub-images, each denoted as s. We compute the the frequency of pixel values ${\mathrm{p}}_{\mathrm{s}}\left(\mathrm{i}\right)$, representing the data distribution of pixel values i within each sub-image. The definition of the ${\mathrm{p}}_{\mathrm{s}}\left(\mathrm{i}\right)$ is given by Equation (6):

$${\mathrm{p}}_{\mathrm{s}}\left(\mathrm{i}\right)=\frac{{\mathrm{n}}_{\mathrm{i}}}{\mathrm{N}}$$

(6)

where ${\mathrm{p}}_{\mathrm{s}}\left(\mathrm{i}\right)$ represents the frequency of pixel values equal to i, ${\mathrm{n}}_{\mathrm{i}}$ represents the number of pixels with a pixel value of i, and N represents the total number of pixels in the sub-image.

Secondly, the ULIE method computes the Cumulative Distribution Function (CDF) ${\mathrm{c}}_{\mathrm{s}}\left(\mathrm{i}\right)$ for each sub-image s in Equation (7), representing the cumulative frequency of pixel values less than or equal to i:

$${\mathrm{c}}_{\mathrm{s}}\left(\mathrm{i}\right)={\displaystyle \sum }_{\mathrm{j}=0}^{\mathrm{i}}{\mathrm{p}}_{\mathrm{s}}\left(\mathrm{j}\right)$$

(7)

where ${\mathrm{c}}_{\mathrm{s}}\left(\mathrm{i}\right)$ represents the CDF for the pixel value i, ${\mathrm{p}}_{\mathrm{s}}\left(\mathrm{j}\right)$ represents the frequency of pixel values equal to j.

Thirdly, the ULIE method utilizes Equation (8) to compute the transformation function ${\mathrm{T}}_{\mathrm{s}}\left(\mathrm{i}\right)$ for each sub-image s, representing the function that maps the original pixel value i to a new pixel value:

$${\mathrm{T}}_{\mathrm{s}}\left(\mathrm{i}\right)=\lfloor {\mathrm{c}}_{\mathrm{s}}\left(\mathrm{i}\right)\rfloor$$

(8)

where ${\mathrm{T}}_{\mathrm{s}}\left(\mathrm{i}\right)$ represents the transformed pixel value for the original pixel value i, L represents the maximum range of pixel values, and $\lfloor \text{·}\rfloor$ represents the floor function.

The ULIE method clips and redistributes ${\mathrm{p}}_{\mathrm{s}}\left(\mathrm{i}\right)$ for each sub-image s, limiting the degree of contrast enhancement, which prevents the amplification of noise and excessive enhancement [28]. Finally, the ULIE method consolidates all transformed sub-images ${\mathrm{T}}_{\mathrm{s}}\left(\mathrm{i}\right)$ into the final image and converts the image from YUV format back to RGB format.

The results of ULIE images are shown in Figure 2, where the left image is the non-optimized image, and the right image is the optimized image. Figure 2a shows that the image has improved overall illumination, with a clearer boundary between the leakage and the background. The ULIE method effectively enhances the image contrast and clarity. Figure 2b reveals that the illumination of the optimized image is more balanced. The ULEIE method alleviates the phenomenon of local overexposure, which further proves that our method effectively avoids gray jump [30]. We perform convolution and downsampling operations on the yellow and red box regions, obtaining the corresponding feature maps between non-optimized and optimized images. It is shown that the details of the optimized feature map are more obvious. In summary, through the above process, our method can effectively enhance the details and textures of sauce-packet images under insufficient illumination and overexposure.

2.2. ISLS Network Details for Leakage Segmentation

In the leakage region segmentation stage of the ISLS method, we propose our network with the EdgeNext backbone, which comprises only 1.3M parameters [31]. The EdgeNext integrates the advantages of Convolutional Neural Network (CNN) and Vision Transformer (ViT). The CNN extracts local features of images using convolution operation [32], and the ViT [33] captures global contextual information of images. The network is the end-to-end network, where the input channel dimension is 3 (i.e., RGB), and the input image size is 128×512. Our overall network structure is shown in Figure 3, which includes the encoder and decoder.

The encoder fuses the local and global representation. Firstly, the n×n Conv encoder consists of three modules. The n×n Conv encoder utilizes adaptive kernels to adjust the size of convolutional kernels based on distinct network layers, which aims to decrease computational complexity and enhance the receptive field [34]. Secondly, the SDTA encoder combines spatial and channel information. The SDTA encoder utilizes deep transposed convolution and adaptive attention mechanisms, which improve the performance of capturing local and global representation. Thirdly, the information of deep and shallow layer feature maps is fused by our MFFM, which improves the performance of encoder feature extraction. Our MFFM structure is as shown in Figure 4.

Specifically, we extract four feature maps of different sizes from the encoder, denoted as ${\mathrm{x}}_1$, ${\mathrm{x}}_2$, ${\mathrm{x}}_3$, ${\mathrm{x}}_4$. Firstly, the MFFM adjusts the channel number of the feature map ${\mathrm{x}}_1$ to 512 through a 1×1 convolution and 4× downsampling. Next, the MFFM applies similar operations with ${\mathrm{x}}_1$ to the feature map ${\mathrm{x}}_2$, with the 2× downsampling, as illustrated in Equation (9):

$${\mathrm{x}}_{\mathrm{i}}^{\text{’}}=\mathrm{MaxPooling}\left(\mathrm{Conv}\left({\mathrm{x}}_{\mathrm{i}}\right)\right),\mathrm{i}\in \text{{}1,2\text{}}$$

(9)

where MaxPooling represents the downsampling process through maximum-pooling operation, Conv represents the convolution operator.

Secondly, ${\mathrm{x}}_3$ has the same size as the output. Therefore, the MFFM only needs to utilize a 1×1 convolution, to adjust the channel number of the feature map ${\mathrm{x}}_3$ to 512, as shown in Equation (10):

$${\mathrm{x}}_3^{\text{’}}=\mathrm{Conv}\left({\mathrm{x}}_3\right)$$

(10)

Thirdly, the channel number of ${\mathrm{x}}_4$ is same with the output, therefore the MFFM performs only an ${\mathrm{Upsampling}}_2$ operation on the feature map ${\mathrm{x}}_4$ The ${\mathrm{Upsampling}}_2$ is achieved using nearest-neighbor interpolation, as depicted in Equation (11):

$${\mathrm{x}}_4^{\text{’}}{=\mathrm{Upsampling}}_2\left(x_4\right)$$

(11)

where ${\mathrm{Upsampling}}_2$ represents 2× upsampling operation.

Through the above operations, the feature maps ${\mathrm{x}}_1^{\text{’}}$, ${\mathrm{x}}_2^{\text{’}}$, ${\mathrm{x}}_3^{\text{’}}$ and ${\mathrm{x}}_4^{\text{’}}$ are obtained. Finally, we fuse the feature information of ${\mathrm{x}}_1^{\text{’}}$, ${\mathrm{x}}_2^{\text{’}}$, ${\mathrm{x}}_3^{\text{’}}$ and ${\mathrm{x}}_4^{\text{’}}$ to output the feature map ${\mathrm{x}}_5$, as shown in Equation (12):

$${\mathrm{x}}_5={\displaystyle \sum }_{\mathrm{i}=1}^4{\mathrm{x}}_{\mathrm{i}}$$

(12)

The reasons of MFFM small parameter number is that the Conv operator employs a 1×1 convolutional, the ${\mathrm{Upsampling}}_2$ operation uses nearest-neighbor interpolation, and downsampling is achieved through Maxpooling. 1×1 convolution operation only increases a small number of parameters, upsampling and downsampling operations do not increase the number of parameters. Compared to Feature Pyramid Network (FPN) [35] and AF-FPN [36], the parameter number of our MFFM is relatively small. That is, our MFFM has a parameter number of only 0.23M, with only 3.05% of the FPN and AF-FPN parameter number.

The decoder includes skip-connection and SSAM. The SSAM in the stage 1 of decoder aims to improve the identification of salient features. The SSAM keeps high-resolution in both channel and spatial branches, which enhances the salient features of sauce-packet ROI. Specifically, the SSAM contains two modules, consisting of channel-only module and the spatial-only module, as shown in Figure 5. For the channel-only module of SSAM, the output ${\mathrm{y}}_{\mathrm{in}}$ is generated by fusing the feature map ${\mathrm{x}}_{\mathrm{in}}$, obtained from both the skip connection and the channel attention mechanism. The specific computational process for channel attention mechanism is shown in Equation (13).

$$\begin{gathered} {\mathrm{y}}_{\mathrm{ch}}=\mathrm{S}\left(\mathrm{Re}\left(\mathrm{Conv}\left({\mathrm{x}}_{\mathrm{in}}\right)\right)\right)\otimes \mathrm{Re}\left(\mathrm{Conv}\left({\mathrm{x}}_{\mathrm{in}}\right)\right) \\ {\mathrm{y}}_{\mathrm{ch}}^{\text{’}}=\mathrm{Sig}\left(\mathrm{LN}\left(\mathrm{Conv}\left({\mathrm{y}}_{\mathrm{ch}}\right)\right)\right) \end{gathered}$$

(13)

where ${\mathrm{x}}_{\mathrm{in}}$ and ${\mathrm{y}}_{\mathrm{ch}}$represent the input and output of SSAM channel-only module. Conv represents for the convolution operator, Re represents reshape operator, and S represents softmax operator. Additionally, LN represents layer normalization, Sig represents the sigmoid function.

The process of the SSAM spatial-only module is similar, with one part from skip-connection, and the other part from spatial attention mechanism, as shown in Equation (14) for specific operations:

$$\begin{gathered} {\mathrm{y}}_{\mathrm{sp}}=\mathrm{S}\left(\mathrm{Re}\left(\mathrm{GP}\left(\mathrm{Conv}\left({\mathrm{y}}_{\mathrm{in}}\right)\right)\right)\right)\otimes \mathrm{Re}\left(\mathrm{Conv}\left({\mathrm{y}}_{\mathrm{in}}\right)\right) \\ {\mathrm{y}}_{\mathrm{out}}{=\mathrm{y}}_{\mathrm{in}}\odot \mathrm{Sig}\left(\mathrm{LN}\left({\mathrm{y}}_{\mathrm{sp}}\right)\right) \end{gathered}$$

(14)

where ${\mathrm{y}}_{\mathrm{in}}$ and ${\mathrm{y}}_{\mathrm{out}}$ represent the input and output of SSAM spatial-only module. GP, S, and LN represent to the global pooling operator, softmax operator, and layer normalization, respectively. $\otimes$ and $\odot$ represent the tensor product and multiplication operations, respectively.

The stages 2 to 5 of our proposed network decoder contains SSAM and Concat operator. The Concat operator concatenates feature maps of two branches in the channel dimension, as shown in green box of Figure 3. Specifically, the one branch feature map comes from the SSAM output, which is upsampled 2×. The other branch feature map comes from skip connections, which can avoid gradient vanishing and improve the training speed of the network [37].

3. Experiments and Results

3.1. Dataset and Experiment Setting

Currently, there are almost no publicly available datasets for sauce-packet leakage segmentation. Therefore, we generate a dataset at Nanjing University of Posts and Telecommunications, captured by an industrial high-speed camera namely the MER2-134-90GC, and the Daheng Image industrial lens HN-P-1628-6M-C2/3. Our self-built dataset comprises images with varying degrees of leakage. Specifically, it includes 315 images under normal illumination conditions, 143 images under insufficient illumination, and 148 images with overexposure. Some examples from the self-built dataset are presented in Figure 6.

The backbone of our proposed network is EdgeNext. Our experimental environment is PyTorch 2.0.1. We divide the dataset into training and validation sets, containing 424 and 182 images, respectively. During the training phase, we employed the NVIDIA RTX GPU 4060, while for inference, we deployed it to the NVIDIA Jetson TX2. Meanwhile, we utilize the cross-validation strategy to verify the robustness of the model. During training, we use DiceLoss [38] to measure the degree of similarity between the predicted results and the ground truth. In addition, we employ FocalLoss [39] to relieve class imbalance by emphasizing hard-to-classify examples, which makes the model pay more attention to challenging pixels. During deployment, we utilize pruning technology to accelerate our model.

3.2. Evaluation Indexes

To evaluate the performance of the method, we selected 5 widely used evaluation indices: Mean Intersection over Union (mIoU), Mean Pixel Accuracy (mPA), F1-score, Params, and Frames per Second (FPS). Params is employed as the evaluation index for the model parameters. And FPS is the evaluation index for the inference speed of a model. The definitions are shown in Equation (15)–(17):

$$\mathrm{mIoU}=\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

(15)

$$\mathrm{mPA}=\frac{1}{\mathrm{n}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(16)

$$\mathrm{F}1\text{-score}=\frac{2\text{×}\mathrm{precision}\text{×}\mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}$$

(17)

where precision and recall represent TP/(TP+FP) and TP/(TP+FN) respectively.

3.3. Experiment Analysis of ULIE

We employ the controlled variable method to explore the optimal thresholds of ULIE for insufficient illumination and overexposure. In the case of insufficient illumination, to validate the optimal threshold as 115, we maintain the overexposure threshold at 180. We adjust the insufficient illumination threshold within the range of 75 to 145. The ISLS evaluation indices (i.e., mIoU, mPA, Accuracy, F1-score) for different thresholds are shown in Figure 7. We observe that the evaluation indices reach optimum when the threshold is set at 115.

Simultaneously, to verify the optimal threshold for overexposure as 180, we maintain the insufficient illumination threshold at 115. We adjust the overexposure threshold within the range of 150 to 220. The ISLS evaluation indices for various thresholds are presented in Figure 8. It is noted that the evaluation indices achieve their peak performance when the threshold is set at 180.

In summary, we define the thresholds of insufficient illumination and overexposure are 115 and 180, respectively.

3.4. Analysis of Ablation Study

In this paper, we propose the two stage Illumination-aware Sauce-packet Leakage Segmentation (ISLS) method. The ISLS method consists of the Uneven-Light Image Enhancement (ULIE) method, Multi-scale Feature Fusion Module (MFFM), and Sequential Self-Attention Mechanism (SSAM). To assess the performance of ISLS, we conducted ablation studies on each component.

Ablation for ULIE: We propose ULIE to alleviate the problem of blurring images in uneven illumination conditions, which improves the visibility of the images. In Table 1, compared to the baseline, mIoU improves by about 1.5% and mPA by about 3.9% after using ULIE. The ULIE method is not based on deep learning, which relies on the adjustment and optimization of image illumination and contrast. Hence, the ULIE method does not require training any parameters. Through Section 2.1 methods and formulas, the ULIE method adaptively enhances the illumination and contrast of images.

Ablation for MFFM: The mIoU and mPA of the MFFM reach 78.7% and 87.9%, as shown in Table 1. Compared with baseline, the mIoU and mPA of MFFM increase 3.1% and 2.3%. In comparison to other feature fusion modules, MFFM exhibits significantly fewer parameters than both FPN and AF-FPN, with only 3.05% of FPN and AF-FPN. Additionally, the mIoU and mPA of MFFM are 1.4% and 2.7% higher than FPN, and the mIoU and mPA of MFFM are 3.4% and 4.3% higher than AF-FPN. Significantly, FPN and AF-FPN structures are intricate, and the features of leakage are monotonous, rendering the extraction of salient features less effective.

Ablation for SSAM: We add attention mechanism to ISLS to improve the performance of feature extraction and compare with Global Attention Mechanism (GAM) [40] and Simple, Parameter-Free Attention Module (SimAM) [41]. In Table 2, it can be observed that utilizing SSAM results in an improvement of about 2.4% in mIoU compared to GAM and around 3.7% compared to SimAM. Additionally, mPA shows an increase of approximately 0.9% and 3.5%, respectively.

To further analyze the effectiveness of our method, we use the Grad-cam [43] to visualize the attention heatmap. As shown in Figure 9, the redder the heatmap, the more the attention mechanism focuses on the feature. Figure 9c exhibits strong capabilities for global feature extraction but shows insufficient performance in capturing local information. Figure 9b,d does not fully focus on the leakage regions and exists the problem of attention divergence. Our ISLS method with SSAM effectively alleviates interference from invalid information, which results in a more focused on leakage regions, as shown in Figure 9e.

The leakage segmentation results of sauce-packet with different attention mechanisms are shown in Figure 10. Our method with SSAM achieves more refined leakage boundary segmentation for sauce-packets by employing spatial and channel adaptive weighting.

Our proposed ISLS achieves 4 FPS on the NVIDIA Jetson TX2. Our ISLS meets the real-time requirements for industrial applications, which demand a minimum of 3 FPS. Figure 11a illustrates the CPU utilization of the ISLS method. The GPU utilization of the leakage segmentation stage is illustrated in Figure 11b.

Before optimization, the GPU utilization often approached 100% (green line), posing risks of system crashes or failures. To ensure system stability and device reliability, we optimized the ISLS method by integrating L1 unstructured pruning technology [44], with a reduction of 20% in the model’s parameter. Performing fine-tuning training while pruning can effectively mitigate the performance degradation caused by pruning, with almost no decrease in accuracy. The GPU utilization after optimization decreases to a certain extent (red line), accelerating model inference while simultaneously decreasing GPU load and enhancing system stability.

Figure 11c illustrates the comparison between the FPS before optimization (green line) and after optimization (red line). Experimental results demonstrate a significant improvement in the performance of our method after optimization. Compared with the unoptimized method, the optimized method improves FPS by 2.7 times.

Experimental results demonstrate that ISLS performs well under uneven illumination conditions. We deploy the ISLS on intelligent sensors, the system fulfills the real-time requirements of industrial applications.

3.5. Comparison with Other Segmentation Methods

In this section, we compare ISLS with several state-of-the-art (SOTA) semantic segmentation networks, including HRNet [13], BiseNetv2 [14], SegFormer [15], PSPNet [45], DeepLabv3 [46], and LIEPNet [18]. The evaluation results are shown in Table 3. The design advantage of our model alleviates the negative impact of uneven illumination and effectively captures semantic features of multi-level leakage. Consequently, we achieved the highest accuracy, with mIoU, mPA, and F1-score reaching 80.8%, 90.1%, and 88.8%, respectively.

Additionally, we compare ISLS with several classical traditional segmentation methods, including template matching [48], Canny edge segmentation [49], contour segmentation [50], PCA segmentation [51], and iForest segmentation [54]. The evaluation results of traditional leakage segmentation algorithms are shown in Table 4. The performance of traditional methods is insufficient in the sauce-packet leakage segmentation. On the one hand, some traditional algorithms mistakenly identify sealing imprints on the sauce-packet and the black blocks. On the other hand, these algorithms significantly affected by uneven-lighting conditions. These above problems of traditional algorithms result in lower accuracy in sauce-packet leakage segmentation.

4. Conclusions

In this paper, our objective is to address the issue of detecting leakage in blurred images under uneven illumination conditions. We propose the Illumination-aware Sauce-packet Leakage Segmentation (ISLS) method, consisting of illumination-aware region enhancement and leakage region segmentation stages. The first stage of ISLS reduces redundant computations of image enhancement processing and alleviates the image blurring caused by uneven illumination, which effectively enhances image details and textures. In the second stage of ISLS, we design a leakage segmentation network. In our proposed network, the Multi-scale Feature Fusion Module (MFFM) efficiently fuses the shallow and deep layer features with a small number of parameters, which improves the feature extraction performance. Additionally, the Sequential Self-Attention Mechanism (SSAM) achieves feature enhancement in both channel and spatial dimensions, improving the identification of salient features. Extensive experiments on our self-built dataset demonstrate that our method effectively alleviates the blurred sauce-packet imaging issue and outperforms existing algorithms. Furthermore, our method improves the stability and reliability of industrial systems and reduces the waste of production resources. The performance testing of the intelligent sensors also validates the suitability of our ISLS method for the current scenario.

In future work, we will further focus on model parameter compression and precision tuning. Meanwhile, we will contemplate the deployment of the state-of-the-art SAM large model on the edge to augment the quality inspection rate of industrial sauce-packet products, thereby enhancing the efficacy of quality control processes.