MATLAB Application to Userfriendly Design Fully Convolutional Data Description Models for Defect Detection of Industrial Products and Its Concurrent Visualization

1. Introduction

Recently, image data-based deep learning models such as convolutional neural network (CNN), support vector machine (SVM), convolutional auto encoder (CAE), variable auto encoder (VAE), fully convolution network (FCN) such as U-Net and so on have been applied to defect detection for various kinds of industrial products and materials. For example, after some defect is detected in inspection process using a CNN model, gradient-weighted class activation mapping (Grad-CAM) or Occlusion Sensitivity is applied to visualization process of the defect areas where the CNN is gazing. This means that defect detection process and visualization one have to be separately employed in the production line.

In this paper, fully convolutional data description (FCDD) approach [1] is applied to the defect detection and its concurrent visualization of two kinds of industrial products and an industrial material. As for FCDD, Jang and Bae proposed an anomaly detection system which was understood through FCDD trained only using OK samples to provide related users with details such as detection results of abnormal defect patterns, defect size, and location of defect patterns on wafer bin map [2]. The effectiveness is analyzed using open dataset, providing promising results of the proposed anomaly detection system. Also, Yasuno et al. reported obtained accurate and explainable results demonstrating experimental studies on concrete damage and steel corrosion in civil engineering [3]. In addition, to develop a more robust application, they applied their method to another outdoor domain that contains complex and noisy backgrounds using natural disaster datasets collected using various devices. Moreover, they proposed a valuable solution of deeper FCDDs focusing on other powerful backbones such as VGG16, ResNet101, and Inceptionv3 to improve the performance of damage detection and to implement ablation studies on disaster datasets.

Our developed MATLAB application for building defect detection models has already allowed users to efficiently design, train and test various kinds of models such as originally designed CNN, transfer learning-based CNN, SVM, CAE, VAE, FCN, and YOLO, however, FCDD has not been supported yet [4]. This paper includes the software development of MATLAB application extended to flexibly build FCDD models [5], in which one of desirable CNN models can be easily chosen for the backbone of an FCDD model to be designed. Also, in order to be able to quantitatively apply trained FCDD models to defect detection of industrial products or materials, threshold value, which is the distance from the center of hyper sphere, is statistically determined analyzing the distribution of anomaly scores obtained with training dataset.

Through the paper, the usefulness and effectiveness of FCDD models in terms of defect detection and its concurrent visualization of understanding are quantitatively and qualitatively evaluated by comparing conventional transfer learning-based CNN models.

2. FCDD (Fully Convolutional Data Description)

2.1. Deep One-Class Classification

Liznerski et al. proposed FCDD models, in which the concept of hyper sphere classifier (HSC) [6] was employed. In this subsection, the objective function of HSC is reviewed. The objective function of HSC is designed as

$$\begin{array}{c}\hfill \underset{\mathcal{W},c}{min}{\displaystyle \frac{1}{n}}\sum _{i=1}^n\left(1-y_i\right)h\left(\varphi (X_i;\mathcal{W})-c\right)-y_{i}log\left(1-exp\left(-h\left(\varphi (X_i;\mathcal{W})-c\right)\right)\right)\end{array}$$

(1)

where $X_1,X_2,\cdots ,X_n$ denote a collection of sample images, and n is the number of training images. $\mathcal{W}$$=\{W^1,\cdots ,W^L\}$ are the weights of the network $\varphi :{\mathbb{R}}^{c\times h\times w}\to {\mathbb{R}}^d$, in which $L\in \mathbb{N}$ is the number of hidden layers, c, h, and w are the channel, height, and width of the images, respectively. Also, $y_1,y_2,\cdots ,y_n$ are labels where $y_i=1$ denotes an anomaly image, and $y_i=0$ denotes a normal image. Furthermore, $c\in {\mathbb{R}}^d$ is the center of hypersphere, and $h\left(\varphi (X_i;\mathcal{W})-c\right)$ is the pseudo-Huber loss [7] given by

$$\begin{array}{c}\hfill h\left(a\right)=\sqrt{{\parallel a\parallel }^2+1}-1\end{array}$$

(2)

which approximates $a^2/2$ for small values of a and a straight line with slope 1 for large values of a, so that interpolation from quadratic to linear penalization can be performed. Due to HSC loss, normal samples and anomalous ones are mapped near c and away from the center c, respectively. The first term of Equation (1) becomes effective when the label of a training image is negative, i.e., $y_i=0$. This is the pseudo Huber loss $h(\varphi (X_i;\mathcal{W})-c)$ based on the distance $\varphi (X_i;\mathcal{W})-c$, which quantifies how far the output of the network is from the center of hyper sphere c. As can be guessed, the closer the output of the network and c are, the smaller the value of first term becomes. On the other hand, the second term becomes effective when the label of a training image is positive, i.e., $y_i=1$. The larger $\varphi (X_i;\mathcal{W})-c$ is, i.e., the farther the output of the network is from c, the closer $exp\left(-h\left(\varphi (X_i;\mathcal{W})-c\right)\right)$ approaches to 0. Consequently, $log\left(1-exp\left(-h\left(\varphi (X_i;\mathcal{W})-c\right)\right)\right)$ approaches to $log1$, i.e., 0. It is expected from the above validation results of the two terms that the network will be trained so that normal images without a defect are mapped near the center c, whereas anomalous images with a defect are located far from the center.

2.2. Fully Convolutional Data Description Model

Figure 1 illustrates the overall network structure of the authors’ designed FCDD model. In this subsection, the concept of FCDD [1] and its loss function in training are described in detail. In FCDD implementation, the center c corresponds to the bias term in the last layer of the networks, i.e., is included in the network, which is why c is omitted in the FCDD objective function. In Liznerski’s paper, an FCN model $\varphi$ employed in the former part performs $\varphi :$${\mathbb{R}}^{c\times h\times w}\to$${\mathbb{R}}^{u\times v}$, by which a feature map $\varphi (X;\mathcal{W})$ downsized into $u\times v$ is generated from an input image X. A heat map of defective regions can be produced based on the feature map. The pseudo-Huber loss $A\left(X\right)$ in terms of an output matrix from the FCN part, i.e., a feature map is given by

$$\begin{array}{c}\hfill A\left(X\right)=\sqrt{\varphi {\left(X;\mathcal{W}\right)}^2+1}-1\end{array}$$

(3)

where the calculation is done with element-wise operation, i.e., pixel-wisely to be able to form a heat map. The object function in training an FCDD model is given by

$$\begin{array}{c}\hfill \underset{\mathcal{W}}{min}{\displaystyle \frac{1}{n}}\sum _{i=1}^n\left(1-y_i\right){\displaystyle \frac{1}{u·v}}{\parallel A\left(X_i\right)\parallel }_1-y_{i}log\left(1-exp\left(-{\displaystyle \frac{1}{u·v}}{\parallel A\left(X_i\right)\parallel }_1\right)\right)\end{array}$$

(4)

With the same solution about Equation (1), the first term has a valid value in case that the label of a training image is negative, i.e., $y_i=0$, where L1 norm $\parallel A\left(X_i\right){\parallel }_1$ is divided by the total pixels $u·v$ of a feature map. The value can be considered as the average per one pixel. Therefore, when normal images are given to the network in training, the weights are adjusted so that each pixel forming a heat map can approach to 0.

On the other hand, the second term becomes effective when the label of a training image is anomaly ($y_i=1$), and $exp(·)$ has a value close to 0 with the increase of the average loss per one pixel, so that the value of log function $log(·)$ also approaches to 0 with the lapse of training time. It is confirmed from the above discussion that Equation (4) using Equation (3) enables both to minimize the sum of the averages of $\parallel A\left(X_i\right){\parallel }_1$ of non-defective images and to maximize those of defective images.

Figure 2. Application of FCDD modeling developed on MATLAB 2024b.

Figure 2. Application of FCDD modeling developed on MATLAB 2024b.

2.3. How to Determine the Threshold Value for Prediction by FCDD

Figure 3 shows examples of training results. (a) is completely trained case, in which normal and anomaly samples in training dataset are completely separated. (b) is not-completely trained case, in which overlaps of normal and anomaly bins are seen. We usually face two such cases of training result according to the difficulty of given defect detection tasks. Also, it is observed from the distributions that anomaly images tend to have more variety of features than normal ones.

If a test image $X_i$ is given to a trained FCDD model, then the FCDD outputs a distance $D\left(X_i\right)$ from the center of hypersphere. $D\left(X_i\right)$ is also anomaly score given by

$$\begin{array}{c}\hfill D\left(X_i\right)={\displaystyle \frac{1}{u·v}}{\parallel A\left(X_i\right)\parallel }_1\end{array}$$

(5)

In order to predict whether this image is normal or abnormal, a threshold value has to be determined on horizontal axes in Figure 3(a) or Figure 3(b). In this subsection, two threshold determination methods are proposed according to (a) completely trained case and (b) not completely trained case. In the case of (a), the threshold $Th$ is simply determined by

$$\begin{array}{c}\hfill Th={\displaystyle \frac{1}{2}}\left(D_{max}\left\{N_N\right\}+D_{min}\left\{N_A\right\}\right)\end{array}$$

(6)

where $N_N$ and $N_A$ are the numbers of normal images and anomaly ones, $D_{max}\left\{N_N\right\}$ and $D_{min}\left\{N_A\right\}$ are the maximum value of $D\left(X_i\right)(i=1,\cdots ,N_N)$ and the minimum value of $D\left(X_i\right)(i=1,\cdots ,N_A)$, respectively. On the other hand, in the case of (b), the threshold $Th$ is determined by each weighted mean value including overlapped bins, which is given by

$$\begin{array}{c}\hfill Th={\displaystyle \frac{1}{2}}\left(D_{\mathrm{mean}}\left\{N_N\right\}+D_{\mathrm{mean}}\left\{N_A\right\}\right)\end{array}$$

(7)

where $D_{\mathrm{mean}}\left\{N_N\right\}$ and $D_{\mathrm{mean}}\left\{N_A\right\}$ are the mean values of $D\left(X_i\right)(i=1,\cdots ,N_N)$ and $D\left(X_i\right)(i=1,\cdots ,N_A)$, respectively.

Main dialogue developed on MATLAB environment as shown in Figure 2 has been extended so that FCDD models can be user-friendly trained, tested and built. The threshold value discussed in this subsection can be automatically set through this dialogue. It is expected by applying trained FCDD models to quality control process of industrial products and materials that defect detection and its clearer visualization of understanding can be concurrently conducted.

3. Comparison of Transfer Learning-Based CNN and FCDD

In this section, comparison in terms of classification accuracy and its visualization of understanding is conducted between two type of models by applying them to three kinds of tasks, i.e., two industrial products and one industrial material. The first model is well-known transfer learning-based CNN models which have been applied to various kinds of defect inspection systems in industrial fields. The second model is the authors’ interesting FCDD model introduced in the previous section. As for the dataset for training and test, casting manufacturing product images for quality inspection has been downloaded from Kaggle database [8]. Figure 4 and Figure 5 show examples of brake rotors without and with defects, respectively. In the following experiments, a VGG19-based CNN model and a VGG19-based FCDD model are evaluated and compared in terms of defect detection and its visualization.

3.1. In case of Transfer Learning-Based CNN Model Based on VGG19

The authors have applied several kinds of transfer learning-based CNN models as shown in pull down menu in Figure 2 to defect inspection of industrial products and materials [9,10]. It has been empirically confirmed from classification experiments of test images that VGG19-based transferred CNN models could always perform high accurate detection ability, so that this time also the same design approach is employed to the defect detection for comparison.

The numbers of images used are 400 OKs and 400 NGs for training dataset, and 100 OKs and 100 NGs for test dataset. After the training through 10 epochs while applying stochastic gradient decent momentum (SGDM) optimizer [11] and cross entropy loss function, the VGG19-based CNN was evaluated using the test dataset. Table 1 shows the classification result of the test dataset. As can be seen, this binary classification task of images as shown in Figure 4 and Figure 5 seems to be not difficult for the CNN model, in which only one image named `cast_def_0_571.jpeg’ shown in Figure 6 is misclassified as normal (OK). However, this false negative actually looks as a normal product since almost no defect or damage can be observed in the image. There may have been a human labeling error.

Then, visualization of understanding, i.e., where the CNN is looking at in classification, is tried to be generated using Grad-CAM [13] and Occlusion Sensitivity [14]. Figure 7 and Figure 8 show examples of visualization results. Superiority of Occlusion Sensitivity can be confirmed by comparing these two results, i.e., defect regions are identified more clearly and properly in the heat maps. Note that in this experiment, two parameters such as mask size and stride required for Occlusion Sensitivity are set to 45 and 22, respectively.

3.2. In case of FCDD

Our designed FCDD model consists of 36 layers, in which layers from 1st to 28th in original VGG19 are extracted with their powerfully-tuned weights and deployed in the encoder part of the FCDD, i.e., FCN part, for a feature extractor to produce 28×28×512 sized feature maps. In training the FCDD using the same training dataset in the previous subsection, Adam (Adaptive moment estimation) optimizer [12] is applied to tuning the weight parameters mainly for the 29th, 32nd, 35th convolution layers and 30th, 33rd batch normalization layers in the FCDD, i.e., to minimize the object function given by Equation (4). After training the FCDD model through 10 epochs, its generalization ability is simply evaluated using the same test dataset in the previous subsection, so that almost the same result is obtained as shown in Table 2. One image misclassified as false negative is checked, so that, in this case also, `cast_def_0_571.jpeg’ is confirmed. It is interesting for us that both the VGG19-based CNN and the FCDD could properly identify the `cast_def_0_571.jpeg’ as normal in spite of being labeled as anomaly in the test dataset. Actually, since `cast_def_0_571.jpeg’ looks like a normal product without defect, it can be guessed that this image is mislabeled incorrectly by human error, i.e., as a labeling noise.

Then, visualization ability for understanding the basis of classification is evaluated using the same images as shown in Figure 7 and Figure 8. In the case of the FCDD model, superior and distinct heat maps as shown in Figure 9 can be obtained without separately and additionally applying Grad-CAM or Occlusion Sensitivity.

4. Further Comparisons of CNN and FCDD

Two kinds of further comparisons similar to the experiment in the previous section are conducted to deeply evaluate the superiority of FCDD model using different datasets of a fibrous industrial material and a wrap film product.

4.1. Defect Detection and Visualization of a Fibrous Industrial Material

Figure 10 and Figure 11 show examples of fibrous industrial materials without and with defects, respectively. There are seen undesirable shape deformation defects such as frayed tip and split end.

4.1.1. In case of Transfer Learning-Based CNN Model Based on VGG19

As evaluated in the previous section, classification experiments were conducted using a VGG19-based transfer learning CNN model. The numbers of images used are 187 OKs and 214 NGs for training dataset, and 46 OKs and 55 NGs for test dataset. After training 300 epochs, we evaluated the VGG19-based CNN using the test dataset. Table 3 shows the classification result of the test dataset, in which 10 images are misclassified. The defect detection accuracy is 90.1%. Note that the original images with the large resolution of 2590×1942 have to be downsized into 224×224 as shown in Figure 12 which is that of the input layer of VGG19, so that unfortunately minute features of defects seem to be lost in some degree.

Then, two visualization tools, Grad-CAM and Occlusion Sensitivity, are applied to identify the areas where the CNN is looking at in classification. Figure 13 and Figure 14 show examples of the visualization results. Similar to the results in the previous section, the superiority of Occlusion Sensitivity is observed, however, the thin split-hair type defect is not properly identified as shown in the left side photo.

4.1.2. In case of FCDD

Similarly, after training the FCDD model for 300 epochs, its generalization ability is evaluated using the same test dataset as in the previous subsection. Table 4 shows the classification result of the test dataset. As can be seen, in the case of FCDD model also, complete defect detection was difficult due to the same cause of the decay of minute defect features as in the previous subsection. The classification accuracy is 89.1%.

Then, visualization images to show the basis of each classification are shown in Figure 15. In the case of the FCDD model, superior and distinct heat maps as shown in Figure 15 can be obtained without separately and additionally applying Grad-CAM or Occlusion Sensitivity.

4.2. Defect Detection and Visualization of a Wrap Film Product

Up to now, working and effort on systematically detecting defective wrap film products as shown in Figure 16 have been devoted by inspectors and engineers. However, there is still a problem such that sufficient detection performance cannot be obtained even if commercially available image detectors are used. The undesirable dislocation of the transparent film and the light reflection from the film seem to make the defect detection more difficult. The authors researched SVM models for the defect detection designed based on VAE model or CNN model, and tried to apply them to the detecting defective wrap film products [15].

In the 3rd test trial, wrap film products are used for comparison. Figure 16 shows non-defective and defective images on the production line, in which typical defects such as long core, deviation of rolling, protrusion of film, deformation of core, taper-shaped rolling are seen. The wrap film products are constantly fed with the same orientation through the automation line, and the snapshots of target film areas are regularly captured by a fixed camera, so that the template matching technique to crop target film area is easy to be applied [15].

A dataset for training is composed of 118 OK and 158 NG images. Also, a dataset for tesing is 468 OK and 628 NG images. All the images are in advance cropped by using the template matching method.

4.2.1. In case of Transfer Learning-Based CNN Model Based on VGG19

After training a transfer learning-based CNN model based on VGG19, the generalization ability is evaluated using the test dataset. The result is tabulated in the confusion matrix shown in Table 5, in which the detection accuracy is 98.3%. Also, Figure 17 and Figure 18 show the visualization results using Grad-CAM and Occlusion Sensitivity, respectively, when each defective image is predicted by the CNN model. As can be seen, it seems that it is not easy for these two visualizers to validly produce heat maps catching the defective areas surrounded with red dotted lines.

4.2.2. In case of FCDD

Then, after training a VGG19-based FCDD model, the generalization ability is evaluated using the same test dataset. The result is tabulated in the confusion matrix shown in Table 6, in which the detection accuracy is 98.3%. Also, Figure 19 shows the visualization results which can be concurrently produced with defect detection. As can be seen, it is observed that the FCDD model can generate more valid and clear heat maps compared to those by Grad-CAM and Occlusion Sensitivity.

5. Conclusions

Our developed MATLAB application for building defect detection models has already allowed users to efficiently design, train and test various kinds of models such as an originally designed CNN, transfer learning-based CNN, SVM, CAE, VAE, FCN, and YOLO, however, FCDD models have not been supported yet. This paper includes the software development extended to flexibly build FCDD models, by which one of desirable CNN models is easily selected for the backbone of FCDD. In this paper, VGG19-based FCDD models have been built and applied to defect detection tasks of two industrial products and one industrial material, then their detection and visualization abilities are experimentally evaluated while comparing conventional transfer learning-based CNN models with Grad-CAM and Occlusion Sensitivity. As for the generalization ability for defect detection, both models perform almost the same classification ability although, for example, the visual features of several minute defects are lost due to the downsizing of images. On the other hand, as for visualization ability of understanding, the superiorities of three FCDD models are experimentally confirmed due to its concurrent and more distinct visualization performance. One of the attractive functions of our developed FCDD designer is the flexible selection of backbone CNN to be incorporated in the former part of FCDD model. In this paper, only VGG19 is selected for the backbone due to our past empirical knowledge, however, performances of other combinations, in which other powerful CNN models such as AlexNet, GoogleNet, NASNET-Large, ResNet50, DarkNet-53 and so on are used for the backbone, have not been evaluated. In the future, therefore, FCDD models incorporated with such CNN models will be compared each other in terms of both detection accuracy and visualization of understanding.