Design of Identification System Based on Machine Tools’ Sounds Using Neural Networks

1. Introduction

In order to solve the issues related to product quality control faced by the manufacturing industry, the authors is developing an application that can easily and effectively design and train a machine learning model that has the same or more ability to identify defective products as skilled inspectors. Figure 1 and Figure 2 show the main and sub dialogues developed on MATLAB. By using the proposed application, the authors are supporting engineers to built their desired machine learning models for defect detection of industrial products and materials included in images and movies. Available models are convolutional neural network (CNN), convolutional autoencoder (CAE), support vector machine (SVM), you look only once (YOLO) [1,2], U-Net, segmenting objects by locations (SOLO) [3,4], mask region-based CNN (Mask R-CNN) [5], fully convolutional data description (FCDD) and so on [6]. In fact, in manufacturing industries, there is a strong need for flexible anomaly detection systems that allow workers to easily cope with all the processes from setting up the environment to operating the system, so that we are evaluating and improving the application through trial use, and expanding functionality based on feedbacks from users.

Up to now, many relevant researches of anomaly detection system based on sound data have been proposed. Harada et al. proposed a baseline system for first-shot-compliant unsupervised anomaly detection for machine condition monitoring, in which a simple autoencoder-based implementation combined with a selective Mahalanobis metric is implemented as a baseline system. The performance is evaluated to set the target benchmark for the forthcoming Detection and Classification of Acoustic Scenes and Events (DCASE2023T2) [7]. Zhou et al. proposed an incremental learning-based anomaly sound detection model that enhances the model’s capacity to learn from continuous data streams, reduces knowledge forgetting, and improves the stability of the model in anomaly sound detection task. Experiments using Task 2 data from the DCASE2020 challenge shows that the proposed method effectively improves the average AUC and average pAUC by 7% to 10% when compared to the fine-tuning strategy [8]. Also, Dong et al. proposed a self-encoder model combining residual CNN and long and short-term memory (LSTM) network is used to extract features in both spatial and temporal dimensions, respectively, to make full use of the information of the audio signal [9]. In addition, Sekhar et al. proposed a texture analysis-based transfer learning CNN models so that they can be applied to a three classes (high/medium/low tool wear) classification task of tool wear based on the noise generated during mild steel machining. Machining acoustics were converted to spectrogram images so that they can be given to the input layer of each CNN, in which four pre-trained models as SqueezeNet, ResNet50, InceptionV3, GoogLeNet were used for backbone.

However, there seems to be almost no discussion about the optimal way to design sound data for training machine learning models. Also, it seems that concurrent visualization of understanding in time domain is not well realized when NN models are applied to anomaly detection of time series data such as acoustics data. In this paper, the authors have considered neural network systems that can be easily applied to classification, anomaly detection and prediction of NC machine tools as shown in Figure 3. In addition to images and videos used for training, in order that time series data such as mechanical sounds and vibrations can be used as multidimensional vector data, design functions for shallow NN, recurrent NN, 1D CNN, and AE are implemented in the application shown in Figure 2. For training machine learning models, many sound block (SB) data extracted with a designated sampling time are generated from nine categories of mechanical sounds collected from multiple machine tools. We report on the evaluation of the classification performance of each model on test data while changing the extraction time, which determines the length of the sound block, and the number of sound blocks used for training.

Finally, SB data-based FCDD is proposed for multi-dimensional vector data to realize anomaly detection of time-series data and its concurrent visualization of understanding, in which for example, time series sound data are converted to 1 line gray-scale images followed by BMP images for training FCDD. The effectiveness of the proposed model is evaluated by experiments.

2. Machining Tool Operating Sound and Sound Blocks

In the experiment, multiple machine tools installed at the university’s machine design and manufacturing center were operated, and operating sounds from nine categories were collected for 10 seconds at a sampling rate of 44100 [Hz] using a smartphone and saved in WAV files. Sound blocks extracted from WAV files are used for training neural network models. Figure 4 shows an example of extraction of time series sound block data from a WAV file which is recorded from a general purpose milling machine. In this experiment, sound block data are sampled with 0.005 [s], so that the length of a SB data file is 44100×0.005≒220.

According to the length, the number of input layer’s neurons is designed as 220. Also, in this case, the number of sound blocks extracted from a WAV file becomes 2000, whose 80%, 10%, and 10% are assigned for training, validation, and test, respectively

3. Training and Evaluation of a Classical Shallow Neural Network

3.1. Neural Networks for Classification

Table 1 shows the numbers of sound blocks prepared for training, evaluation, and test of nine categories. Also, Figure 5 and Figure 6 shows four layered normal neural networks and recurrent neural networks, respectively. The numbers of units in the first and second hidden layers 100 and 50, respectively. Moreover, Figure 7 shows the structure of 1D CNN consisting of 11 layers, in which the number N of labels from the output layer is 9 according to the classification task of 9 categories.

After trained by the adaptive moment estimation (ADAM) [10] and cross entropy loss, these neural network models were compared and evaluated through a classification task of test sound block data. As a result, mean values of classification accuracies are 87.6%, 88.4%, and 99.5%, respectively. Although there was no significant difference in classification performance between NN and RNN, higher classification performance was obtained with 1D CNN. As for training time, also, although approximately tens of thousands of epochs were needed for conversion of NN and RNN, 1D CNN could be training in shorter training time. Actually, the numbers of training weight parameters of NN, RNN, and 1D CNN were 27,609, 52,609, 11,273, respectively. Table 2 shows each layer’s output activation and the numbers of parameters. Note that the meanings of T and C are time series data and the number of channels, respectively. It is expected from the above results that, in the future, anomaly detection using 1D CNN will be possible by accumulating both of normal and anomaly machining sounds.

3.2. Autoencoder for Anomaly Detection

When the objective is to detect abnormalities in machine tools, etc., it seems effective to apply an autoencoder model that can be trained using only normal machining sounds. In this evaluation experiment, an autoencoder is designed as shown in Figure 8. A structural feature of the autoencoder is that the input and output layers have the same number N of units. The loss function is given by Eq. (1) which is composed of the mean suared error (MSE) loss, L2 regularization term of weights, and sparse regularization term based on Kullback Leibler Divergence (KLD) [11]. The autoencoder is trained so that sound blocks at the input layer can be equivalently generated from the output layer. The autoencoder is trained such that every sound block $x\in {\Re }^{N\times 1}$ in the training data given to the input layer can be neuron-level-equally reconstructed as $\widehat{x}\in {\Re }^{N\times 1}$ at the output layer.

$$\begin{array}{c}\hfill E={\displaystyle \frac{1}{M}}{\displaystyle \frac{1}{N}}\sum _{m=1}^M\sum _{n=1}^N{\left(x_{mn}-{\widehat{x}}_{mn}\right)}^2\\ \hfill +\alpha \left[\sum _{i=1}^N\sum _{j=1}^H{\left(w_{ij}^1\right)}^2+\sum _{j=1}^H\sum _{k=1}^N{\left(w_{jk}^2\right)}^2\right]\\ \hfill +\beta \left[\sum _{i=1}^H\left\{\rho log{\displaystyle \frac{\rho }{{\widehat{\rho }}_i}}+(1-\rho )log{\displaystyle \frac{1-\rho }{1-{\widehat{\rho }}_i}}\right\}\right]\end{array}$$

(1)

where N is the length of a sound block, which is the number of units both in input and output layers. H and M are the numbers of units in the hidden layer and that of all the sound blocks in training data. $w_{ij}^1$ are the weights between the input layer and hidden layer, $w_{jk}^2$ are those between the hidden layer and output layer. Also, $\alpha$ and $\beta$ are the coefficients to weight the degrees of penalties of L2 regularization and sparse regularization, respectively. Note that ${\widehat{\rho }}_i$ in the third term is the mean value of activation generated by the sigmoid function h at the ith unit in the hidden layer, which is given by

$$\begin{array}{c}\hfill {\widehat{\rho }}_i={\displaystyle \frac{1}{M}}\sum _{j=1}^{M}h\left(w_i^{\left(1\right)T}{x}_j\right)\end{array}$$

(2)

where $w_i^{\left(1\right)T}$ is the ith row data in the weight matrix $w^{\left(1\right)}$, $x_j$ is the jth training data. Note that $\rho$ in Eq. (1) is the desired value of $\widehat{\rho }$, to which 0.05 is set. As can be seen, the loss function to train the autoencoder is composed of the first term about MSE, the second term about L2 regularization, and third term about sparse regularization based on KLD.

In training the autoencoder, the reconstruction loss included in Eq. (1) according to an input of sound block $x\in {\Re }^{N\times 1}$ is given by

$$\begin{array}{c}\hfill E_{\mathrm{mse}}={\displaystyle \frac{1}{N}}\sum _{n=1}^N{\left(x_n-{\widehat{x}}_n\right)}^2\end{array}$$

(3)

A threshold value is defined as the maximum value of $E_{\mathrm{mse}}$ obtained during the training. After training, if $E_{\mathrm{mse}}$ obtained from a SB for test is under the threshold value, then the SB is predicted as the same machine with the autoencoder, otherwise a different machine. This means that if the autoencoder is trained using the SB data extracted from one machine tool, a test SB data can be predicted as the same machine or not.

In experiments, SB data of a band saw shown in Table 1 is used for training the autoencoder. At the first step, the extracted time of a sound block is set to 0.005 [s], i.e., data length N is set to 220 ($\fallingdotseq 44100\times 0.005$), however, a successful classification result could not be obtained. Accordingly, the extracted time is gradually increased to 0.05, i.e., N=2205, so that each 300 test SB data except the band saw could be well identified as different from those of the band saw as shown in Table 3. Note that the number of neurons H in the hidden layers is 1000, consequently, that of all the weight parameters is 4,413,205. It is expected from the results that detection of anomaly mechanical sound will be possible by training the autoencoder based on the sound in normal situation.

4. SB Data-Based FCDD Model for Anomaly Detection and Visualization of Time Series Data

It has been confirmed from the experiments up to the previous sections that 1D CNN and autoencoder are effective for classification and identification of SB data, respectively. As can be expected, 1D CNN is also applied to anomaly detection tasks by redesigning the output layer for binary classification, i.e., normal and anomaly.

4.1. The Proposed FCDD for Time Series Data such as SB Data

At this stage, one of serious problems in dealing with the time series data such as SB data is how clearly and concurrently anomaly areas should be visualized and understood. In this paper, to cope with need, a SB data-based FCDD model is further proposed as shown in Figure 9 to perform anomaly detection and its concurrent visualization without secondly using Grad-CAM [12] or Occlusion Sensitivity [13]. The proposed method allows us to construct an FCDD-based anomaly detection system for time series data such as SB data.

Objective function of FCDD [14] is briefly introduced. In Liznerski’s paper, an FCN model $\varphi$ employed in the former part performs $\varphi :$${\mathbb{R}}^{c\times h\times w}\to$${\mathbb{R}}^{u\times v}$, by which a feature map $\varphi (\mathit{X};\mathcal{W})$ downsized into $u\times v$ is generated from an input image X. A heat map of defective regions can be produced based on the feature map. The pseudo-Huber loss $\mathit{A}\left(\mathit{X}\right)$ [15] in terms of an output matrix from the FCN part, i.e., a feature map is given by

$$\begin{array}{c}\hfill \mathit{A}\left(\mathit{X}\right)=\sqrt{\varphi {\left(\mathit{X};\mathcal{W}\right)}^2+1}-1\end{array}$$

(4)

where the calculation is done with element-wise operation, i.e., pixel-wisely to be able to form a heat map. The object function in training an FCDD model is given by

$$\begin{array}{c}\hfill \underset{\mathcal{W}}{min}{\displaystyle \frac{1}{n}}\sum _{i=1}^n\left(1-y_i\right){\displaystyle \frac{1}{u·v}}{\parallel \mathit{A}\left({\mathit{X}}_i\right)\parallel }_1\\ \hfill -y_{i}log\left(1-exp\left(-{\displaystyle \frac{1}{u·v}}{\parallel \mathit{A}\left({\mathit{X}}_i\right)\parallel }_1\right)\right)\end{array}$$

(5)

The first term has a valid value in case that the label of a training image is negative, i.e., $y_i=0$, where L1 norm $\parallel \mathit{A}\left({\mathit{X}}_i\right){\parallel }_1$ is divided by the total pixels $u·v$ of a feature map. The value can be considered as the average per one pixel. Therefore, when normal images are given to the network in training, the weights are adjusted so that each pixel forming a heat map can approach to 0.

On the other hand, the second term becomes effective when the label of a training image is anomaly ($y_i=1$), and $exp(·)$ has a value close to 0 with the increase of the average loss per one pixel, so that the value of log function $log(·)$ also approaches to 0 with the lapse of training time. It is confirmed from the above discussion that Eq. (5) using Eq. (4) enables both to minimize the sum of the averages of $\parallel \mathit{A}\left({\mathit{X}}_i\right){\parallel }_1$ of non-defective images and to maximize those of defective images. The main dialogue shown in Figure 1 enables to user-friendly train, test and built FCDD models.

4.2. How to Generate Image Data from SB data

In this subsection, it is explained that how to generate input images $\mathit{X}{\in }^{N\times N}$ for FCDD from SB data in time-series domain. As shown in Figure 9, X has the same resolution with the input layer of FCDD. As already explained, A SB data $\mathit{s}{\in }^{1\times N}$ is directly extracted from a WAV file with a designated extraction time $\Delta t$ [s]. For example, if the sampling frequency of a WAV file is f [Hz] , then the length N of a SB data becomes $f\Delta t$. 1 line SB data $\mathit{s}=[s_1,s_2,\dots ,s_N]$ is transformed to 1 line gray-scale BMP image $\mathit{I}=[I_1,I_2,\dots ,I_N]$ as shown in Figure 10 through normalization by

$$\begin{array}{c}\hfill I_i={\displaystyle \frac{s_i-s_{\min }}{s_{\max }-s_{\min }}}(i=1,2,\dots ,N)\end{array}$$

(6)

where $s_{\max }$ and $s_{\min }$ are the maximum and minimum values of elements in s, respectively. Then, an expanded bitmap image $\mathit{X}{\in }^{N\times N}$ to be given to the input layer of FCDD can be constructed as

$$\begin{array}{c}\hfill \mathit{X}=\left[\begin{array}{c}\mathit{I}\\ \mathit{I}\\ ⋮\\ \mathit{I}\end{array}\right]\end{array}$$

(7)

4.3. Experiment of Identification of Machine Tools’ Sound Data and Its Concurrent Visualization Using an FCDD Model

In this subsection, an identification experiment is conducted using the nine kinds of SB data as shown in Table 4, in which it is assumed that the sound of Band-Saw is normal and other eight kinds of sounds are anomaly, so that 30 normal SBs and 30×8=240 anomaly SBs are used for training the FCDD model. After 200 epocks of training, all the training data were scored as shown in Figure 11. In order to use the trained FCDD as an anomaly detector, a threshold value for criteria has to be set. In this case, a threshold value 1 of anomaly score can be easily determined from the distribution in Figure 11.

After setting 1 to the threshold value, generalization ability was checked using test SB data. 300 normal SBs and 300×8=2400 anomaly SBs were used for testing, so that all the images could be accurately classified to normal (Band-Saw) or anomaly (Except for Band-Saw) as shown in Table 5. Figure 12 shows the histogram of the test SB data’s mean anomaly scores predicted by the FCDD model. Figure 13 also shows examples of predicted maps generated by the FCDD, in which it is observed that anomaly regions within the time series data are validly visualized.

5. Conclusions

The authors have been developing a design, training and building application with a user-friendly operation interface for CNN, CAE, SVM, YOLOX, SOLOv2, FCDD, and so on, which can be used for the defect detection of various kinds of industrial products even without deep skills and knowledge concerning information technology. In those models, images are basically used for training data. In this paper, an intelligent anomaly diagnosis system for NC machine tools is considered, i.e., what structures of neural networks should be applied to this task. Mechanical sound and vibration generated from machine tools themselves or machining sound and vibration generated from router bits, i.e., end mill cutters are recorded as wave files and used for training data. Extracted SB data from wave files are employed for training NN models. It is confirmed from experiments that a 1D CNN and an autoencoder are effective for a sound classification task and a sound identification one, respectively. In this paper, a SB data-based FCDD model is further proposed for anomaly sound detection of removal machining by NC machine tools and its concurrent visualization, in which time series data such as SB data can be directly applied to training and testing without converting them to other domain such as frequency. The effectiveness of the proposed method is shown through experiments.