An Improved Fault Diagnosis Approach for Pumps Based on Neural Networks with Improved Adaptive Activation Function

1. Introduction

The fault diagnosis of the pumping unit in the process of petroleum collection has been a critical research topic. Due to the complex underground environment, during the reciprocate movement of the sucker rod, there are many unknown factors, which are prone to result in the failure of the pumping machine and then form a safety hazard. Load (P) and displacement (S) are the parameters generated when the donkey head of the pumping unit moves up and down, and the closed curve formed by them is the indicator diagram. It can reflect the influence of gas, oil, water, sand, wax and other factors on pumping unit in real time [1]. If the pump is in the fault state for a long time, the wear of the pump will be aggravated and the service life of the equipment will be further reduced.

The traditional method for fault diagnosis of pumping unit is to measure the load change with displacement at the suspension point, draw the suspension point indicator diagram, and then diagnose the working condition of the pumping unit according to the shape of the indicator diagram. The disadvantages of the traditional method are as follows: first, the fault of the pumping unit is judged mainly by the method of manual identification of indicator diagram, which has great influence on human factors and low recognition accuracy. Secondly, due to the large number and wide distribution of pumped Wells, manual detection of Wells is time-consuming and laborious. However, as the complexity and high cost of pump units, there is less tolerance for performance degradation, productivity decrease, and safety hazards, therefore, it is necessary to detect and identify all potential faults rapidly [2,3]. It means that it is imperative to replace the manual fault diagnosis of the pumping unit with a computer.

With the continuous progress of deep learning technology, using deep learning technology for fault diagnosis has become a new trend. For example, Convolutional Neural Network [4] , Generation Adversarial Network [5], and Long Short Term Memory[6] et.al show superior performance in fault diagnosis.

At present, the deep learning technology used in fault diagnosis of pumping unit is to classify the indicator diagram of different kinds of faults. Automatic feature extracted from raw data is an outstanding advantage of deep learning technology, and it will not depend on the diagnostic knowledge of specialists [7].

In 2018, Y.Duan[8] proposed an improved Alexnet model to realize the automatic recognition of indicator graphs, and compared it with the current common neural network model. In 2019, J. Sang [9] proposed a PSO-BP neural network algorithm, aiming at the problems of slow convergence and unstable results of the traditional BP neural network algorithm, designed the adjustment rules of the inertia weight and learning factor of the PSO algorithm, and adjusted the weight coefficient of the output layer and the hidden layer of the BP neural network algorithm. In 2020, L.Zhang [10] used Freeman chain code and differential code to extract the characteristics of dynamometer card data of pumping unit group. Then a diagnosis model based on BP neural network was proposed, and the fault type of pump group can be automatically identified according to dynamometer card. In 2022, H.Hu[11] proposed a model based on the ResNet-34 residual network to identify the indicator diagrams, which added a residual block structure to the traditional convolutional neural network to establish a direct connection between the upper layer input and the lower layer output and achieved the recognition and classification of six power diagrams through parameter adjustment. In the same year, T.Bai[12] proposed a fault diagnosis method based on time series transformation generative adversation network (TSC-DCGAN).

Because of the complexity of pumps working conditions, there are different shapes of indicator diagrams in different working states. The indicator diagrams of different kinds of faults are similar in a certain degree thus indistinguishable samples are produced. This will lead to poor generalization ability of deep learning models and difficulty in between indistinguishable samples. The function of activation function is to carry out nonlinear transformation of data and solve the problem of insufficient expression and classification ability of linear model. If the network is all linear transformation, then the multi-layer network can be directly converted into a layer of neural network through matrix transformation. Therefore, the existence of activation function can make the deep learning model perform better with the increase of the number of layers. Therefore, we will propose a new activation function to imporove the generalization performance of the deep learning model, so that the faults of the pumping unit can be distinguished in a high dimensional space.

Rectifying linear unit(ReLU)[13], which has low computational complexity and fast convergence speed, can solve the problems of gradient disappearance and gradient saturation. In recent years, there have been many improved versions of ReLU (rectified linear unit). To solve the Dead ReLU phenomenon, the negative part of ReLU is substituted for a non-zero slope and Leaky ReLU [14] is proposed. Hence Leaky ReLU is more inclined to activate in the negative area.

In deep learning, the selection of activation function is generally determined according to the specific situation, and there is no fixed choice. As the adaptive activation function can be automatically adjusted to adapt to the network structure and practical problems, it has been widely developed. Parametric Rectified Linear Unit (PReLU) [15]is also used to solve the Dead ReLU phenomenon. The slope of the negative part can be obtained by learning from the data, rather than from defined fixed values. Therefore, PReLU has all the advantages of ReLU in theory and is more flexible than Leaky ReLU. In 2017, the Swish activation function was proposed. It has the characteristics of lower bound, no upper bound and non-monotonic. It is very smooth with its first derivative [16], and its performance is better than ReLU in many aspects. In 2021, H.Hu [17] proposed a new scheme to explore the optimal activation function with greater flexibility and adaptability by adding only a few parameters on the basis of traditional activation functions such as Sigmoid, Tanh and ReLU. This method avoids local minima by introducing a few parameters into a fixed activation function. In the same year, M.Zhao [18] used the specially designed subnetwork of Resnet-APReLU as an embedded module in order to adaptively generate the multiplicative coefficient in nonlinear transformation.

Based on the above discussions, an adaptive activation function combined with the gate-controlled channel transfer unit module (GCT)[19] is designed in this paper. The main contributions are as follows:

• We propose an improved adaptive activation function. Each layer of deep learning generates different activation functions, improves the generalization performance of deep learning models, and has strong adaptability to different deep learning models.
• We apply the proposed activation function to the fault diagnosis of pumping unit, so as to better extract features from the contours of the indicator diagram. The proposed activation function improves the accuracy of fault diagnosis and has a better search ability, which is verified and comprared with AlexNet[20], VGG-16[21], GoogleNet[22], ResNet[23] and DenseNet[24].
• The propose activation function is extended to the public datasets CIFAR10 which proves that the proposed activation function is suitable and universal.

The rest of this paper is organized as follows. In section 2, we introduce the pumping unit data set. In section 3, we introduce the common adaptive activation function, and propose the composition of our adaptive activation function. In section 4, the experimental analysis and the discussion on the pumping unit failure data set and the public dataset are presented. In section 5, we conclude the paper.

2. Experiment Design and Measurement

2.1. Introduction to pumping unit

At present, about 80 % of oil wells in most oil fields in China use rod pumping equipments, and the most widely used is the beam pumping unit[25]. The failure data set of pumping unit comes from the real data generated by the pumping unit operation in a certain oil field of Northeast China. The pumping unit is a part of a rod pumping unit. Rod pumping equipment is mainly composed of three parts: oil pumping unit, well pumping pump and sucker rod. The Rod pumping equipment is shown in Figure 1.

The pumping unit is driven by a motor and through the reducer transmission system and the execution system, the rod and the pump plunger are driven to move up and down, and finally the crude oil is lifted from the well to the surface. The operation of the pumping unit is shown in Figure 2.

2.2. Fault types of pumping unit

The fault data set of pumping unit consists of nine types of indicator diagrams: normal, insufficient fluid supply, contain sand, piston stuck, gas influence, pump up touch, pump down touch, double valve leakage, and pumping rod detachment. The following details will be introduced:

• Normal
  The pump work diagram made by normal operation refers to the position shift of the end suspension point relative to the lower dead point as the transverse setting mark, the self-weight force of the rod and the cumulative load received by the pump plug as the longitudinal setting mark. Drawn in parallel quadrilateral shape.
• Insufficient fluid supply
  The shortage of liquid supply is due to the insufficient amount of crude oil in the well, and the plunger pump inhales a large amount of air while drawing crude oil each time. As a result, a large amount of gas in the pump cannot be fully operated.
• Contain sand
  Because the well contains sand, the plunger creates an additional resistance in an area during movement. The additional resistance on the up stroke increases the load at the suspension point and on the down stroke at the same position. The increased resistance reduces the load at the suspension point. Because the distribution of sand particles in the pump barrel is not the same, its influence on the load varies greatly in various places, so it will lead to severe fluctuations in the load in a short time.
• Piston stuck
  When the pump plunger is stuck near the bottom dead point, the rod is in a stretched state during the up stroke and the down stroke since the whole stroke is actually the process of elastic deformation of the rod, the well work diagram at this time is approximately an oblique line.
• Gas interference
  The gas interference is the situation that the gas precent in the oil of the pumping well is high, while the crude oil precent is relatively low. This causes the pump barrel to extract most of the gas, resulting in a significant difference between the actual load and the theoretical load.
• Pump down touch
  When the anti-impact distance is too large, the piston running up is approaching the upper dead point, and the continuous upward movement of the piston collides with the moving val, which leads to the sudden loading of the piston and the bunching at the upper dead point.
• Pump up touch
  When the anti-impact distance is too small, it is attached to the lower dead point, and the piston moves down and collides with the fixed Val, resulting in sudden unloading of the piston and bunching at the lower dead point.
• Double valve leakage
  Double valve leakage refers to the situation where both the moving valve leakage and the fixed valve leakage happen at the same time, and the leakage may be caused by a combination of multiple faults.
• Pumping rod detachment
  The pumping unit’s power cannot be transmitted to the pump due to the detachment of the sucker rod, resulting in the inability to extract oil.

The failure of pumping unit will cause great economic losses and security risks. Therefore, rapid and accurate fault diagnosis of pumping unit is very necessary. The fault diagnosis process in this paper is as follows: Firstly, the displacement and load data of the pumping unit are collected by wireless dynamometer. Secondly, the indicator diagram of various faults is drawn by the collected data. Finally, the indicator diagram is preprocessed, and then the indicator diagram is input into the deep learning model to output the fault type. The fault diagnosis flow chart of the pumping unit in this paper is shown in Figure 3.

3. Theoretical Analysis

3.1. Common adaptive activation functions

PReLU activation function is a further improvement on the fixed predefined slope of LeakyReLU, which can be changed by backpropagation. It has better adapt ability [13] than LeakyReLU. The formula of activation function is as follows:

$$f\left(x\right)=Max(x,0)+\delta Min(x,0)$$

(1)

where x is the input, $\delta$ is the trainable multiplicative coefficient (i.e., slope). Each layer has its own $\delta$, which improves the nonlinear capability. In PReLU, $\delta$ in Eq.(1) is the learnable parameter during training, but it is $\delta$ constant and cannot be adjusted during testing.

The design of Swish activation function [16] is inspired by Long Short-term Memory (LSTM) neural network. The Swish activation function can prevent the gradient from gradually approaching zero and leading to saturation during training. It plays an important role in optimization and generalization. The formula of Swish activation function is as follows:

$$f\left(x\right)=xSigmoid\left(\zeta x\right)$$

(2)

where $\beta$ is the learnable parameter or constant. When $\zeta$ = 0, the Swish activation function becomes the linear function $f\left(x\right)=x/2$. When $\zeta$ =∞, the Swish activation function becomes 0 or x, which is equivalent to ReLU activation function. Therefore, the Swish activation function can be considered as a smooth function between linear function and ReLU activation function.

Compared with ReLU, Mish activation function is smoother at the origin [26]. The formula is as follows:

$$f\left(x\right)=xtanh\left(In(1+e^x)\right)$$

(3)

From Eq.(3), Mish Activation function has no upper limit, but only a lower limit, which can ensure no saturated region, thus there will be no vanishing gradient during the training. At the same time, it has a faster convergence speed.

3.2. The structure of adaptive activation functions

The structure of adaptive activation functions are shown in Figure 4. The input of the subnetwork is concatenated by the one-dimensional vector obtained from the two inputs. The two inputs are positive features after separation and negative features after separation. The separation of positive and negative features can highlight the key features. The following calculation paths are GCT→GAP→FC→ Batch Nomoalation(BN)→ ReLU→ FC→ BN→ Sigmoid →Scales. The function of each layer is described in following section.

GCT combines normalization methods and attention mechanisms, which makes it easy to analyze the relationships (competition orcooperation) between channels. As shown in Figure 5, The GCT module introduces three trainable parameters $\alpha$, $\beta$ and $\gamma$ to evaluate the communication channels. Among them, $\alpha$ helps embed the output adaptive ability, while $\beta$ and $\gamma$ are used to control the activation threshold, which determines the behavior of GCT in each channel. h and w are the dimensions of feature vectors, c is the number of channals, and L2-norm is the normalization of L2.

Global average pooling (GAP) can replace the fully connected (FC) layer to achieve dimensionality reduction. Eespically, it retains the spatial information extracted from the previous convolutional layers and pooling layers and can also strengthen the relationship between categories and feature maps [27].

ReLU is selected as the activation function of FC in the first layer to reduce the computational complexity and keep the gradient value within a reasonable range for feature extraction. The formula is as follows:

$$f\left(x\right)=\left\{\begin{array}{c}x,x\ge 0\\ 0,x<0\end{array}\right.$$

(4)

Then we add BN layer, which is a way to unify the scattered data and is similar to normal data standardization. It is also a way to optimize the neural network. The data with unified specifications can make it easier to learn the rules in the data for the deep learning model [28] and can also solve the problem of vanishing gradient. The normalization is described by the following formula:

$$\mu =\frac{1}{N_{batch}}\sum _{i=1}^{N_{batch}}{x}_i$$

(5)

$${\sigma }^2=\frac{1}{N_{batch}}\sum _{i=1}^{N_{batch}}(x_i-\mu {)}^2$$

(6)

$$\underset{i}{\overset{\wedge }{x}}=\frac{x_i-\mu }{\sqrt{{\sigma }^2+\epsilon }}$$

(7)

$$y_i=\psi \underset{i}{\overset{\wedge }{x}}+\theta$$

(8)

where $x_i$ and $y_i$ are the observed input and output of each Nbatch, $\mu$ represents the mean of the input, ${\sigma }^2$ represents the variance of the input, $\epsilon$ is a constant near zero, and $\theta$, $\psi$ are learnable parameters governing the scaling and shifting distributions. The activation function of the second FC layer is Sigmoid, which can limit the output value during the interval (0,1) and prevent excessive slope from affecting the performance of the activation function.

To summarize the above contents, the proposed adaptive activation function has the ability to automatically learn complex features. Different nonlinear transformation is applied to different inputs to improve the generalization performance of deep learning model, which will well solve the problems in extracting feature contour of indicator diagram and sparsity of indicator diagram in pumping unit fault diagnosis. The following experimental simulation will verify the effectiveness of the designed adaptive activation function.

4. Experimental simulation

This section mainly verified the performance of our activation function which was tested on AlexNet[20], VGG-16[21], GoogleNet [22], ResNet [23] and DenseNet [24]. The structure diagrams of these five networks are shown in Figure 6. Moreover, we compared our activation function with the traditional activation functions such as ReLU, Sigmoid, Tanh, LReLU and PReLU.

The experiment is mainly divided into two parts. The first part is the simulation on the fault diagnosis dataset of pumping unit. This will prove that the proposed adaptive activation function can extract the features of the indicator diagram and solve the sparsity problem of the indicator diagrams. The improvement in fault diagnosis accuracy indicates that indistinguishable samples are correctly classified. The second part is to verify the superiority of the designed adaptive activation function on the public dataset CIFAR10.

4.1. The dataset of pumping

Adaptive Moment Estimation (Adam) was used here, and the initial learning rate was 0.001. The epoch of training was no less than 200. The average accuracy of each model is shown in Table 1. In the fault diagnosis data set of pumping unit, the adaptive activation function proposed in this paper has the greatest accuracy improvement in the ResNet model. Compared to traditional activation function ReLU, Tanh, Sigmoid, LeakReLU, PReLU, Mish, Swish, the average accuracy of ResNet model with our activation function respectively increased by 1.7%, 5.09%, 5.72%, 2.54%, 1.46%, 1.75%, 1.56%.

Confusion matrix is a common index and visualization tool to evaluate the results of the classification model and it can judge the advantages and disadvantages of classifiers. The rows of the matrix represent the true value, and the columns of the matrix represent the predicted value. The confusion matrix can respectively count the number of the wrong classification and the right classification, and then display the results in a matrix. Figure 7 shows the confusion matrix of the five models for the pumping unit fault data set. It can be seen that the designed adaptive activation function can effectively represent the mapping relationship between the displacement and the load in the indicator diagram, and extract the features of the indicator diagram, thus those indistinguishable samples are correctly classified. Table 2 gives the accuracy of various types of faults for the five models, which proves that the proposed adaptive activation function can be well applied in the five models and has great adaptability to the models.

The loss function curve is shown in Figure 8. Among them, (a) AlexNet, (b) VGG-16 and (e) DenseNet have fast convergence speed. The loss functions of (d) ResNet and (e) GoogleNet decline relatively slowly, but eventually converge to the optimal value.

4.2. CIFAR10

We used CIFAR10 data set to conduct experiments and analysed AlexNet, VGG-16, GoogleNet, ResNet and DenseNet models with our activation function and the traditional activation functions. We augmented the data to reduce overfitting. The Adam was used with the initial learning rate 0.001. The epoch of training was no less than 200. The average accuracy of each model is shown in Table 3, where the designed activation function improves the performances of those. Among them, AlexNet, VGG-16 and DenseNet have the best performance. Compared with the traditional activation functions ReLU, Tanh, Sigmoid, LeakReLU, PReLU, Swish, and Mish, our activation function in AlexNet model is improved respectively by 1.84%, 4.11%, 5.45%, 0.79%, 2.74%, 1.91%, 2.04%; our activation function in VGG-16 model is improved respectively by 3.1%, 4.54%, 4.45%, 0.48%, 2.02%, 4.69%, 3.94%; our activation function in DenseNet model is improved respectively by 1.88%, 5.1%, 9.63%, 1.07%, 0.61%, 0.35%, 0.37%. The above data indicate the superiority of the proposed activation function.

5. Conclusions

In this paper, a new adaptive activation function is designed and applied to five models of neural networks. Specifically, the adaptive activation function improves the negative semi-axis slope of the ReLU activation function by combining the gated channel conversion unit to enhance the performance of the deep learning model. The activation function in each layer of neural network is unique, thus the input signal of each layer has a unique nonlinear transformation. Therefore, compared with the traditional fixed activation function, our activation function has a better nonlinear transformation ability and it can be well embedded in five models. Such as through the fault diagnosis data set of pumping unit, it is proved that our activation function can effectively display the mapping relationship between displacement and load in the indicator diagram, thus extract the features of the indicator diagram and solve the sparsity problem of the indicator diagrams. Indistinguishable samples are correctly classified. Through CIFIAR10 dataset, it verifies the superiority and universality of our adaptive activation function.

In short, the proposed adaptive activation function increases the accuracy of fault diagnosis and has a better generalization performance and search ability. Moreover, the proposed adaptive activation functions also can be well embedded in other models of neural networks.