Determining Water Pipe Leakage Using a RP-CNN Model – Identifying the Causes and Improving Poor Accuracy Cases

1. Introduction

1.1. Background

The deteriorating state of urban infrastructure in Japan, including maintenance and upgrades, is estimated to cost huge amounts of money to rectify. Therefore, to address these challenges, Infrastructure Maintenance, Renewal, and Management Technology designated a national project by the Council for Science, Technology, and Innovation called the Strategic Innovation Promotion Program (SIP) FY 2014–2018. During the SIP, the authors participated in Research and Development of Sensing Data Collection, Transmission, and Processing Technology for Social Infrastructure (Underground Structures). It was reported that the water supply pipeline network facility had a total length of 18 times the circumference of the equator, covering approximately 70% of water supply assets worth 40 trillion yen. As the proportion of old pipe that exceeds its service life of 40 years increases, it becomes necessary to switch from conventional methods to the use of advanced technologies. In order to address the issue of water supply pipeline infrastructure, SIP conducted research on the development and placement of high-sensitivity sensor terminals to construct a leakage detection model based on field test data acquired in the cities of Kawasaki and Kitakyushu.

In accordance with the first stage of SIP results, a discrimination method integrating visualization and image recognition of sound data was designed. Further research was conducted on leakage discrimination based on the test data obtained in the field. Geometric visualization was performed to analyze the characteristics of leakage sounds depending on the strength of deterministic properties. This included the use of water leakage sounds, background noise (data measured when there is no leakage), and tests to determine whether the data were probabilistic [1]. Second, there was a focus on the differences in the deterministic properties of leakage sound time series data. Additionally, the acquired data were visualized on a two-dimensional plane using a recurrence plot (RP). The visualized images served as input data for machine learning used to fabricate a leakage discrimination model with a convolutional neural network (CNN) [2]. Third, the RP-CNN model was applied to the actual data to verify accuracy of leakage discrimination. An average accuracy of more than 80% was obtained, confirming effectiveness of the proposed combined RP-CNN method [3]. Meanwhile, accuracy of model assessment was improved by applying filter processing focusing on the frequency components of the acoustic data [4].

The results of leakage detection did not show improvement in some points even after pre-processing. This also led to the inability of BA (balanced accuracy), an indicator of accuracy in machine learning, to exceed 80% [4]. In addition, points 3-B and 4-B had less accurate results, leading to the need for discrete research conducted by water leakage investigators (five engineers with more than 10 years of experience in a water leakage detection company) using human hearing. Identifying the causes of inaccuracy at these points is a challenge that should be overcome by developing new technologies exceeding the level of conventional methods used by leakage investigators.

In this research, Section 1 focuses on the background to improving accuracy of the water leakage detection model previously proposed [4]. It also aims to identify and improve the causes of low accuracy. Additionally, this section reviews the latest trends in water leakage detection, clarifying the originality of the present research. Section 2 describes the acoustic data used during the analysis, including items and observation equipment. This also provides an overview of a RP-CNN model applied to water leakage detection. In Section 3, dimension reduction of acoustic data (features based on frequency components) is performed using principal component analysis (PCA). This visualizes the uniqueness of the points showing inaccuracy by positioning samples based on principal component scores. This section examines two types of inaccuracies – inaccuracy of results to leakage (recall) and background noise (specificity) – including considering improvement measures for each result. However, to improve recall, an amplification is specifically proposed within the frequency range of 500–600 Hz estimated as the characteristic range of water leakage sound, combined with weak frequency reduction. In order to improve specificity, the use of a band-stop filter is proposed to remove the frequency component of 50 Hz and respective harmonics considered as AC noise (noise from the electric current source), to improve specificity. Additionally, in this section, the pre-processing proposed in the previous section is applied (centered on the characteristics of leakage sound and removal of electric current noise in acoustic data). Leakage detection testing using the RP-CNN model is carried out, verifying the usefulness of this research. Finally, in Section 4, the conclusions are reported, along with future work to explore more effective ways to apply the proposed pre-processing method.

1.2. Literature Review

This section reviews certain previous research on water leakage detection conducted in recent years to show the latest trends. Asada et al. [5] proposed a water leakage detection method using transient test-based techniques (TTBTs). The method uses transient phenomena in pipes depending on optimization processes and characteristics of pressure wave propagation associated with reflection to detect leakage. Furthermore, it was successfully used to detect leakage in spiral pipe, as well as network connections with different diameters (pipe reducers). Pressure gauges were installed at pipe connection points, with operated valves at the bottom to detect leakage throughout the pipe network.

Meniconi et al. [6] used the TTBT approach to detect faults on long main transmission lines, which are difficult to inspect due to limited access. The analysis results indicated that this approach can reduce the adverse effects of changes in initial conditions and flow boundaries, allowing identification of issues in the system. However, only specific areas could be thoroughly eliminated, with two leaks detected with good precision. The system’s complexity, as a result of branching off the main pipeline and interactions with pressure waves during transients, complicated the implementation of TTBTs, as additional pressure waves produced by the branching must be identified [7,8]. The development of methods focusing on transient phenomena at pipe branches aims to expand leak detection to more complex systems [9,10,11].

Duan [12] investigated the effect of pipe joints on transient frequency response using numerical simulations in the frequency and time domains and the linear transfer matrix method. The results indicated that this method is more effective for detecting pipe leaks than measuring leaks. Kim et al. [13] and Kim [14] also conducted experiments on transient-based leak detection for multi-branch pipes.

Shirahata & Numazu [15] conducted research using several ensemble learning methods, including applying CNN on infrared images of rigid PVC pipe used for sewers and then compared the results. The verified results showed that the AdaBoost method had the highest F-value (0.75) and accuracy among the ensemble learning methods. Furthermore, the artificial neural network was the second highest-performing method, with accuracy of 0.72 and F-value of 0.73.

In recent years, research on water leakage detection focusing on the application of machine learning methods and IoT technology using sensor or camera data was introduced [16,17,18,19]. Quantitative data often used included hydraulic physics variables such as water pressure and flow in pipe. This also comprised acoustic data from sensors installed on pipe. In general, the purpose of using IoT communication technology was to monitor pipe networks. The machine learning methods adopted comprised feature extraction using PCA [20,21], ensemble learning [22,23], and CNN [24,25,26]. Specific research used deep learning such as CNN, and this method needs to be considered for developing leakage detection technology.

Hu et al. [27] classified various methods for detecting and determining leakage locations in water distribution systems into two categories: model and data-based methods. They also reviewed leakage detection methods. They stated that the weakness of model-based method was due to the lack of a calibrated hydraulic model used to distinguish water leakage. The discrimination results were also greatly influenced by model and measurement errors. The weakness of the data-based method was because it required a large amount of data to identify water leakage, and the detection results were greatly influenced by data shortages, abnormal values, and noise. Therefore, the preference for model or data-based methods depends on the amount of data obtained from the actual network and difficulty in developing a hydraulic model of the network in question. Tina et al. [28] proposed a leakage detection method combining pipe flow rate measurements using sensors and an Arduino system merged with IoT technology. The sensors were placed at the start and end points of the pipe, while the flow data obtained from the two were compared to detect leakage. This research verified leakage detection using the proposed method prototype. The results showed that assuming no difference in flow between the start and end points, then no leakage was detected.

Recent research on monitoring water distribution networks and leakage detection using IoT technology and hydraulic and deep learning model are important themes. Based on the literature review, our research focuses on two attributes that have not been adequately considered. First, this is centered on leakage detection methods when acoustic data are used for analysis. In previous research, data such as changes in water pressure, pipe vibrations, or infrared images were used for analysis or training. However, research on the use of acoustic data to detect leakage is lacking. This method was considered suitable for continuous monitoring of leakage in pipelines, due to the easy acquisition of acoustic data, and the conventional process adopted by water management companies, namely the survey procedure relying solely on human hearing.

Second, there is a focus on the usefulness of deep learning method in this context. This method is associated with the black box problem – the process of information generated that cannot be clearly understood. Therefore, research may be conducted without discerning the fundamental reasons behind the ability of deep learning to detect leakage. Ito et al. [4] explored the CNN model using a RP-CNN model, and reported that at certain points, detection accuracy was below 80%. However, the causes of this low accuracy were not properly investigated.

The present research analyzes past failure cases, by using PCA to reduce the dimensions of features based on frequency components and to investigate the causes of inaccuracy. The distinctiveness of points with low accuracy was visualized through a sample plot diagram obtained from the principal component scores. This method was used to address the earlier-mentioned problems, serving as a differentiating element from previous research. Additionally, our research refers to the results of the blind test conducted by leakage investigators using human hearing, particularly when evaluating statistical analysis and model accuracy. This increased the reliability of the results, providing original values not recorded in previous research.

2. Materials and Methods

2.1 Overview of Dataset

Acoustic data collection, focusing on water leakage sound and background noise in fields located in the Kanto region, was carried out using sensors. Additionally, the water leakage sound was recorded before carrying out repairs, and recording the background noise depended on reinstalling the sensors after repairs had been completed. Regarding background noise, other conditions remained the same except for the presence or absence of water leakage. When leakage occurred, a location was selected enabling the installation of sensors, such as gate valves, water control valves, or a fire hydrant, at the shortest distance. This ensured the sound label did not only contain noise. Sensors A and B installed at two locations closest to leakage point were used to collect both sounds. The measurement distance from each sensor to the leakage point differed, and there may also have been branches or bends in pipe from the location. Despite there being only one leakage point at each location, the sounds obtained from the two sensors differed. Leakage was detected using a correlation formula to compare the sound when it occurred and the background noise after repairs had been completed. The dataset focused on ductile iron pipe, because this research aimed to conduct a basic analysis of acoustic data recorded from such a pipe network.

Water leakage sound and background noise were recorded before and after repairs at the same location. The data about the existing conditions of leakage location were the same. This resulted in a quality dataset for machine learning, perceived as a significant advantage. Furthermore, acoustic data obtained at five leakage locations using 10 observation points, were used. When leakage occurred at a particular location, the sound data were collected at two points: sensors A and B. Therefore, the acoustic data were symbolized in pairs as {1-A | 1-B},..., {5-A | 5-B}, with a sampling frequency of 10,000 Hz. Frequency components greater than 5,000 Hz (Nyquist frequency) could not be reproduced accurately due to aliasing effects, leading to exclusion from the data analysis. The information about 10 acoustic data used as the research object are shown in Table 1, including leakage volume (L/min), measurement distance (m), sensor installation location (m), causes of leakage, water pressure (MPa), pipe diameter (mm), and soil cover (mm). The results of the blind test conducted by leakage investigators using the acoustic data are shown in Table 2.

2.2. Overview of Water Leakage Determination Based on RP-CNN Model

A RP-CNN model was adopted to determine leakage, using the CNN, a type of deep learning method. This was also applied using the representation of recorded water leakage sound and background noise (RP) data for learning and assessment (Figure 1). Leakage was distinguished using the deterministic nature of the difference between the leak sound and background noise [1]. The discrimination accuracy of the RP-CNN model was previously verified [3,4], which showed it could distinguish between leakage sound and background noise with an accuracy rate of approximately 80% in some locations.

The learning and assessment datasets are crucial when using a machine learning method. However, previous research [4] focused on evaluating the generalization performance using learning data from several locations (besides assessment data) in a 9-point model (Figure 2). The present research concentrated on investigating the causes of less accurate cases (points 3-B and 4-B) and the improvement process. In order to verify the effectiveness of feature amplification and the use of band-stop filters to eliminate AC frequency interference (pre-processing), a 10-point model was adopted with the assessment data applied to the learning process. Therefore, the difficulty in determining leakage sound and background noise observed at various locations, including the possibility of assessment errors, were evaluated using a general model.

3. Results and Discussion

3.1. PCA Application Method

Principal component analysis is a statistical method that reduces multidimensional data to a low-dimensional space without losing important information. It is often used for dimension reduction and visualization in data evaluation. According to the Scikit-Learn Data Analysis Implementation Handbook (Shuwa System), the main purposes of dimensionality reduction are data compression and data visualization. This is provided by algorithms such as PCA, feature selection, and non-negative matrix factorization. Additionally, PCA has been previously used to investigate the causes of low accuracy at several locations [4,29]. The implementation of dimension reduction and visualization comprised two stages:

Stage 1 Application of Fast Fourier Transform (FFT) (Data Dimension Reduction)

Based on previous research, the effective frequency range for detecting leakage in ductile iron pipe is less than 1,500 Hz [30]. Our research focused on 153 frequency components of approximately 1,500 Hz obtained from FFT. For data grouping, the frequency components were divided into 15 categories (<100 Hz, <200 Hz, ..., <1400 Hz, <1500 Hz), with each represented by a mean value. In this process, the data were converted into 15 dimensions. The implementation of this method ensured the number of variables (n = 15) used in the PCA did not exceed the number of samples (n = 20).

Stage 2 Application of PCA (Dimension Reduction and Visualization)

Considering the 15-dimensional data (FFT), PCA was used to obtain the first (PC1) and second (PC2) principal components. We then positioned the samples according to the PC1 and PC2 scores to classify locations with (10 locations) and without leakage (10 locations). In the PCA stage, the data were also re-standardized by centralizing (making the mean value 0) and scaling (making the standard deviation 1). This analysis was performed using version 4.3.2 of Windows software.

3.2. Sample Position Results Based on PCA Scores

A box plot diagram was generated using 15-dimensional data to distinguish between water leakage sound and background noise. The median value for water leakage reached a peak within the frequency range of variables X4–X6. Meanwhile, background noise reached a peak at variable X1, gradually decreasing in the higher frequency range (Figure 3). This implied that the average characteristic pattern between the two differed.

The results of the sample position in accordance with the principal component scores are shown in Figure 4. The contributions of PC1 and PC2 were 0.696 and 0.150, respectively. The focus on leakage sound denoted by ● in Figure 4 implied that it was mostly distributed in the positive direction on PC1 (horizontal axis). However, the background noise denoted by ▲ was mostly distributed in the negative direction on PC1 (horizontal axis). Leakage sound in samples 3-B and 4-B was not located in the zone where the other points were distributed but was projected into the area occupied by the background noise. Compared to the results of the blind test (Table 2), the sound in samples 3-B and 4-B were consistently disregarded. The low accuracy in samples 3-B and 4-B in the RP-CNN model was caused by low recall and specificity outcomes for leakage sound and background noise, respectively. This suggested that there may be a problem with the acoustic data of leakage sound (recall). The acquired sound characteristics were distorted by certain influences resembling background noise. This made assessment difficult for both the RP-CNN model and the investigators. Additionally, both leakage sound and background noise from the same location were detected on the right and left sides of the PC1 axis, respectively. In sample 4-B with low accuracy, the left–right relationship on the PC1 axis was maintained, but in 3-B, leakage sound and background noise were on the left and right sides, respectively. Aside from the left–right inversion pattern, a characteristic of sample 3-B was a short distance between leakage sound and background noise compared to other samples.

3.3. Pre-Processing for Emphasizing Leakage Sound and Eliminating Background Noise

Section 3.2 showed that the possible causes of low accuracy at points 3-B and 4-B was the interference in acoustic data. Recording and transmission were carried out using acoustics by altering the frequency characteristics, while considering distortion and noise. An acoustic device (an equalizer) was used to improve sound quality and reduce certain frequencies, thereby overcoming the problem of noise, howling, or excessive echo.

Water leakage sound is similar to background noise because its unique characteristics do not fully reach the recording point (sensor). This causes the characteristics that should be present in leakage sound data to be weakened. In such cases, it becomes necessary to focus on emphasizing leakage sound characteristics. Therefore, to improve accuracy of sound detection on data with such problems, a pre-processing method that concentrates or reduces certain frequency bands was considered an effective countermeasure.

A typical example of the FFT spectrum compares leakage sound and the background noise at point 1-B (Figure 5). This showed that the sound spectrum had a peak in the relatively high-frequency region, greater than 500 Hz. For the background noise, the frequency region greater than 500 Hz showed a flat pattern without peaks. This pattern was observed at other locations, including 1-B. The box plot of data transformed into 15 dimensions showed that leakage sound had a peak of approximately 500 Hz (Figure 3).

Considering this result, further analysis assumed the frequency region that characterized leakage sound was within 500–600 Hz, leading to the application of an amplification process. However, not all frequency components within 500–600 Hz were amplified. Amplification was performed only when the absolute value of the Fourier transform result, |X(k)|, exceeded 5.5 (with k as the frequency value), to reduce the influence on background noise. Additionally, for sound to reach ideal conditions, it is important to increase and reduce necessary and unnecessary sound, respectively. A reduction can also be performed to eliminate weak noise, aside from amplifying certain frequencies. Therefore, frequencies with |X(k)| > 0.9 were also reduced in this process.

The present research also concentrated on the eradication of sounds similar to water leaks (pseudo-leak sounds), such as transformer sound, by adopting a method of eliminating 50 and 60 Hz frequency components including respective harmonics using a band-stop filter. This also aimed to reduce errors in identifying acoustic data affected by transformer sound, thereby increasing specificity.

Figure 6 shows an example of changes in RP of leakage sound at point 4-B (from 2000 to 2002) after the amplification process, where a honeycomb pattern was observed. These results implied that by amplifying the frequency components of 500–600 Hz, the typical leakage sound at point 4-B was successfully acquired.

The dimension reduction and data visualization methods with PCA used in Section 3.2 were applied to verify the effect of pre-processing. The outcome before and after the application of the process was compared (Figure 7). Adjustments are made by amplifying and reducing frequency components with |X(k)| > 5.5 and |X(k)| < 0.9, respectively. Meanwhile, this amplification and reduction improved leakage sound and background noise. Similar peak changes were obtained when assuming that the background noise within the frequency range of 500–600 Hz had the same characteristics as leakage sound. The slope pattern of the 15-dimensional data implied that these frequencies did not show changes as observed in leakage sound. Therefore, the assumption that the typical region of leakage sound was within 500–600 Hz, including the decision to target this frequency range for the amplification process, can be considered appropriate.

3.4. Water Leakage Determination Test Using RP-CNN Model

This research examined the effect of the proposed pre-processing: (1) strengthening the frequency components within 500–600 Hz only for |X(k)| > 5.5 (prominence in the characteristics of typical leakage sound); (2) removing frequency components with |X(k)| < 0.9 (weak noise reduction); and (3) using a band-stop filter to eliminate the 50 Hz frequency and respective harmonics (AC noise removal). These three pre-processing methods were verified in the present research. Specifically, two types of datasets (with and without pre-processing) were acquired and accuracy of RP-CNN model compared after usage. The testing data differed from the training data used to build the model. The setting values of RP-CNN model follow: interval 0.016 seconds, size 64, LPF 4000 Hz, and batch size 16. In terms of determining the number of epochs, the model with the highest accuracy on the testing data was adopted as the best (epoch = 10). It is generally recommended to treat validation data and training data separately. Validation data are used to adjust hyperparameters, such as batch size, when building the model.

The results after applying the RP-CNN model on the three methods are shown in Table 3. Initially, the results for without pre-processing showed that recall for 3-B and 4-B was 0%. Furthermore, for 3-B and 4-B, the RP-CNN model considered leakage sound as background noise, resulting in a recall of 0%. These results are in accordance with previous research [4] and the positioning performed using principal component scores. However, the results with pre-processing showed that recall for 3-B and 4-B increased to 64.4% and 81.4%, respectively. This suggested improvement due to application of pre-processing. Although the increase in recall accuracy was confirmed in all cases, a negative aspect (adverse impact of applying pre-processing, i.e. a decrease in specificity) was observed in five out of every 10 cases. The issue was observed in 1-B, where specificity decreased by more than 20%. Despite the improved recall for the expected results, the problem of specificity remained.

Recall for 3-B was lower than for other points and the limitations proven by the pre-processing effect. The PCA showed that 3-B tended to have unique acoustic data and a pattern differing from the other points, with the leakage sound and background noise on the left and right sides of the PC1 axis, respectively. Further data collection and accumulation are required, irrespective of whether the data are unique or there are similar cases. This research needs further in-depth observation, considering implementation in the real world.

4. Conclusions

This research investigated the causes of inaccurate cases associated with leakage determination using RP-CNN to gain insights that improved the generalization performance of the model. A basic analysis of leakage sound and background noise was conducted in water pipe, while performing visualized acoustic data using FFT and RP, as well as the evaluating the results. Additionally, a pre-processing method was proposed for acoustic data, including RP pattern changes. Accuracy of leakage detection using the RP-CNN model was studied. The results from this research follow:

• The frequency components obtained from the FFT to 1,500 Hz showed that the data obtained were converted into 15 dimensions, leading to PCA. The results from the sample classification using the PC1 and PC2 scores showed that leakage sound at points 3-B and 4-B was in an area occupied by a lot of background noise. This was consistent with the blind test conducted by leakage investigators, in which the sound at points 3-B and 4-B could not be identified.
• Based on the differences in the FFT spectrum, an amplification process was applied within the frequency range of 500–600 Hz for water leakage sound. After its application, a new honeycomb pattern was found in RP at the problematic location. This showed that the amplification process within this range effectively focused on the characteristics of leakage sound.
• To test the effectiveness of the proposed pre-processing method, two datasets were obtained (with and without pre-processing), followed by comparing accuracy of the RP-CNN model. The results without pre-processing showed recall for points 3-B and 4-B of 0%, while after pre-processing application, these increased to 64.4% and 81.4%, respectively. This implied improvement in recall with application of pre-processing.

The present research focused on the frequency components of 500–600 Hz. The amplification process confirmed a honeycomb pattern in leakage sound RP at the problematic location. However, the location of frequency components that best reflected the characteristics of leakage sound was not completely clear. Sensitivity analysis in the future should change the amplified frequency components while drawing RP and identifying those containing the most characteristics of leakage sound. This was crucial in exploring more effective ways of applying the proposed pre-processing method.