Data Prediction in Datasets of Internet of Things with Recurrent Neural Networks

1. Introduction

Internet of Things (IoT) is a growing technology development branch in the market, providing solutions in various social an economic aspect through the establishment of development standards on low-operational cost computational systems. IoT can be used in monitoring with sensor devices in different types of scenarios, such as, industry, health, environmental monitoring, smart cities, among others. Indeed, the widespread distribution of these devices implies a large volume of data being continuously created.

Commonly, sensor data are ordered by time, formatted as Time Series. According to [1], a time series can be defined as a set of observations ordered by time. These observations may demonstrate various behaviors: (i) Trend - Long-term elements related to the time series; (ii) Cycle - Relatively regular waves, around a trend line; (iii) Seasonality - Regular patterns of the time series; (iv) Randomness - Effects that were not incorporated by the time series through the three components cited above. In other words, the residue.

Time series analysis is not only related to the time series behavior identification, but also to the prediction of future values. This feature is important in some contexts, such as reservoirs level monitoring, earthquakes, air quality, among others.

Nowadays, there are different time series prediction models, from the simplest and easy to implement to the most complicated, based on mathematical formalities and statistics involving a great effort in the prediction of future values. As prediction is not a trivial task, it must be combined powerful algorithms, such as those mentioned above, with large data processing capabilities, to reach good results with the least possible response time.

This paper brings as its main contribution a Prediction Model Based in Recurrent Neural Networks (RNN) [2] for dataset in time series format. Recurrent Neural Networks differ from traditional neural networks (feedforward) because the RNN has the ability to receive signals from both the input layer and the hidden layer in the previous iteration [2]. In a way, the hidden layer simulates how memory works. Furthermore, RNN works with sequential data in both the input layer and the output layer, fitting perfectly into the context of time series.

In the literature, approaches for predicting time series based on bio-inspired Algorithms, Machine Learning, Big Data, Deep Learning, Neural Networks, and others are found, which have great data processing capacity to achieve good results with the shortest possible response time. As prediction is not a trivial task, this work, in turn, intend to purpose a model easy to implement and capable of dealing with any time-series generated by IoT sensors.

To verify the accuracy of the model, experiments were carried out with five datasets from IoT monitoring applications. It’s them: Data from air quality sensors [3]; Power consumption data [4]; Estuarine water quality monitoring data [5]; Data of Carbon Dioxide emission records [6]; Dental chairs compressors monitoring [7]. The proposed solution was compared to the models, Autoregressive Integrated Moving Average (ARIMA) [8] and Support Vector Machine (SVM) [9].

The road map of this paper is organized as follows. Section 2 presents the related work; Section 3 describes the proposed Prediction Model with Recurrent Neural Networks; Section 4 presents the results obtained and discussion, and finally, in Section 5, the conclusions and future work.

2. Current State of the Art

In this section, the background of the work is presented to give a brief context of the model that deal with the problem of time series prediction based on Neural Networks.

2.1. Time Series Prediction

Data Prediction technique aims to estimate future values based on historical data. Based on this principle, the measurements history from a particular sensor node can be used by the sink node to predict future measurements. According to [10], prediction models can be classified into, statistics-based and machine learning-based.

Regarding statistical models, the Autoregressive Integrated Moving Average (ARIMA) [11] model stands out. The following stand out among the models based on machine learning: Support Vector Machine (SVM) [12] and Recurrent Neural Networks (RNN) [13].

An autoregressive integrated moving average (ARIMA) model [11] is a statistical time series forecasting model that aims to find autocorrelations in observations of a given time series in order to make future predictions. According to [14], the ARIMA model is composed of two models: autoregressive (AR) and moving averages (MA). Autoregressive (AR) and moving average (MA) models deal only with stationary time series [15]. A time series is stationary if the series mean and the covariance among its observations do not change over time and do not follow trends. Therefore, time series with a tendency or seasonality cannot be considered stationary [16].

For the prediction of non-stationary time series to be possible, it is necessary to apply differentiation [17]. Differentiation aims to transform non-stationary time series into stationary time series by calculating the difference between consecutive values of the time series. The differentiation process aims to stabilize the time series average by removing large variations, resulting in the reduction of trend and seasonal behavior. ARIMA is a time series prediction model composed of the combination of autoregressive (AR) and moving average (MA) models, together with the differentiation process.

The Support Vector Machine (SVM) [12] is a machine learning technique widely used in different domains, such as text categorization, image analysis and Bioinformatics. SVM has a set of supervised learning methods for data analysis and pattern recognition and can be applied to both classification and regression.

SVR is a derivation of SVM, proposed by [18], considered a non-parametric technique, as it depends on the use of kernels, which are mathematical functions used to receive input data and transform it into the required form. According to [19,20], several kernels can be used in SVR for time series prediction, such as: linear kernel, radial basis function kernel (Radial Basis Function), least square kernel (Least Square), among others.

Artificial Neural Networks are computational models inspired by biological neural networks. Such models seek to learn how to perform certain tasks through the analysis of examples. Artificial Neural Networks’ main objective is to learn how to solve complex problems in a reasonable amount of time [21]. Artificial Neural Networks have been massively used in several areas, such as image classification [22], speech recognition [23], natural language processing [24] and time series prediction [25].

Artificial Neural Networks are composed of an input layer, one or more hidden layers, and an output layer. Each layer contains a certain number of neurons, with each neuron being connected to all neurons in the next layer through synapses. According to [26], the function of synapses is to assign a weight to transferred data. The neurons in the input layer are responsible for inserting data into the network, while the neurons in the hidden layers receive the weighted sums of their inputs and pass on the new data through a non-linear activation function.

There are neural networks in which connections between neurons form cycles, with the purpose of capturing temporal and contextual information in time series data. These networks are called Recurrent Neural Networks [27]. Recurrent Neural Networks receive data from the input layer and keep it to be used in the next iteration. The flow of information does not go in one direction, nor does the result depend exclusively on the current input, but also on previous inputs.

[28] explains that the hidden state of the Recurrent Neural Network receives information from the independent variables, as well as its own processing result from the previous iteration. The activity of neurons in Recurrent Neural Networks is like the behavior of a short-term memory, being very useful in the context of time series prediction, considering that the data from the input and output layers are sequential data.

Data Prediction in Internet of Things can be challenging due to the heterogeneity of the dataset and the volume of monitoring data, which can be massive. To address this challenge, researchers have proposed various data Prediction solutions.

2.2. Related Works

The work presented by [29], proposes a solution based on single-layer feedforward neural networks with an Online Sequential Learning Algorithm (OSLA). This work presents a mathematical perspective that aims to map the current and previous values through a General Non-Linear System. Comparisons with other models using several datasets are shown and the results confirm that the OSLA obtained better performance than the other models compared.

The paper [30] proposes a model for predicting multivariate time series using Hierarchical Neural Networks allied to single cycle reservoirs (SCR) and machine learning. The particle swarm optimization algorithm (PSO) is used as training set to optimize SCR parameters. Simulation results show that the proposed model surpasses the other models evaluated. Its hierarchical structure is specially designed to capture the dynamic and complex characteristics of multivariate time series.

In the work [31], the authors present a method based on Coevolutionary Recurrent Neural Networks for the prediction of chaotic time series with a focus on predictions of many steps ahead. For the training of the neural network, the techniques of Backpropagation Through Time (BPTT) and Cooperative Neuro-Evolution (CNE) were used. For the tests, synthetic and real datasets were used. The results showed that the prediction of many steps ahead is still a great challenge, since the accumulation of errors in the predictions end up with big errors in the results.

In [32], the authors proposed an adaptive gradient learning method for Recurrent Neural Networks to predict time series with anomalies. The proposed algorithm aims to incrementally learn the time series and provide robust predictions by adapting to changing patterns as well as resisting outliers. Experiments have shown that the proposed approach shows superiority over other methods evaluated.

Although the works mentioned above demonstrate good results, the implementations aim at very specific objectives and are quite complex. This work aims to propose a simple and easy implementation model for the prediction of any type of time series generated by devices from Internet of Things.

3. Time Series Prediction Model with Recurrent Neural Networks

This section describes the proposed solution for time series prediction based on RNN in datasets from IoT sensors.

As pointed out in [33], the RNN receives information from the independent variables, as well as from its own processing result from the previous iteration, as illustrated in Figure 1.

In this example, $I_t$ is the input data in the time iteration t, which is going to be processed by the neuron B. $D_t$ is the output from neuron B in the time iteration t, that can be used in the next iteration by the same neuron, as a recurrence standard behavior. It is quite similar to the behavior of a short-term memory and it is quite useful in the time series prediction context, given that the data of the input and output layers are sequential [28].

Neural Networks are commonly used for regression and classification problems. The proposed model uses Recurrent Neural Networks for a regression problem. Which means an equation to estimate the expected value of a variable y, given the values of some other variables i.

When developing the model, we used TensorFlow [35]. TensorFlow is a project from Google to large scale machine learning models. It’s a flexible dataflow-like programming model and can be applied in different platforms of hardware.

TensorFlow can even be used to express a wide variety of algorithms, including training algorithms for scalable neural networks. The main Application Programming Interface (API) is in the Python programming language and in C ++ [35]. TensorFlow supports both CPU and GPU processing to reach higher level of parallelization in executions. In this work, the Python programming language was used, together with some libraries used for data manipulation, such as Pandas [36], NumPy [37] and MatPlotLib [38].

3.1. Definition of Cost Function and Metrics

In Machine Learning, there is a cost, also called loss, which is related to the learning process. [28] states that this cost is represented by the size of the obtained error according to the parameter values that were used. Effectively, cost is the obtained deviation size to specific parameters set. In this case, it is necessary minimize the cost function so that the learning process can be improved.

The Gradient Descent method is used in optimization for the purpose of minimize the cost function using an iterative scheme. Given that the function in question is convex, the negative direction of the gradient is taken at each iteration, until the overall minimum is reached. Nevertheless, the Gradient Descent method may not be ideal when the data set is very large, because the algorithm would loop through the whole data set to compute the gradient and make just one move towards the optimization process.

Since IoT data is massive, in this work it was used the Stochastic Gradient Descent method [39]. In this method, mini batches of data are created randomly to be looped through the algorithm. Consequently, it is not necessary to loop through the entire data set to compute the gradient. It is possible to obtain an approximation just looping through some examples of the data set. However, to get closer of the global minimum, it will be necessary a great number of iterations. Either way, the time of each iteration should be remarkably short, and the learning should be faster [39]. It is common to find this method being used for training of Neural Networks and linear regression problems and this motivated the use of this method in our work.

When evaluating the accuracy of the prediction model, some metrics need to be defined. The most used metrics in machine learning are the Mean Absolute Error (MAE) and the Coefficient of Determination (${\mathrm{R}}^2$).

The MAE measures the average distance from the predictions related to the observed values, focusing the estimation on the conditional median. The coefficient of determination measures the adjustment of a generalized linear statistical model, such as linear regression, in relation to the observed values. The value of ${\mathrm{R}}^2$ vary between 0 and 1, indicating in percentage, how much the model can explain the observed values [40].

4. Performance Evaluation

In this section, we present results of experiments conducted on a computer with the following characteristics: AMD Phenom (tm) II X4 B93 Processor with 4 cores and 4Gb of RAM running Ubuntu Linux 16.04.

The results obtained with the proposed approach in Section 3 are compared with the results obtained by the Autoregressive Integrated Moving Average (ARIMA) [8] and the Support Vector Machine (SVM) [9] algorithms. All models were implemented with the Python programming language.

In experiments, five datasets from Internet of Things were used:

• Data from air quality sensors - University of California, Irvine [3].
• Power consumption data from Low Carbon London Project - UKPN [4].
• Burnett river estuarine water quality monitoring data - Queensland Government (Australia) [5].
• Globally averaged Carbon Dioxide emission records - Earth System Research Laboratory (U.S. Federal Government) [6].
• Dental chairs compressors monitoring [7]. Often, a portion of the air compressors suffer condensation, and the compressors get full of water, leading to the need to purge the water.

Table 1 presents the details of each dataset in terms of sampling, period and number of samples.

In the tests carried out, the aim is to identify whether the model can predict one step ahead according to the latest data observed in each data set.

4.1. Training and Test sets

When a RNN receives sequential data in the input layer, each data sample needs to be a sequence whose format is composed by time periods and variables. As a result, once various data samples are stacked, the mini batches will show a three-dimensional aspect: number of samples, time and variables. In this way, it will be possible to submit the data to the training and test sets. Therefore, each mini batch must contain the discrepancy of one or more rows so that the data maintains the sequential pattern.

To obtain an accurate approximation of the real scenario, it is necessary to split the last data from the time series for the test set. Thus, we defined that the first 70% of each dataset would be used as the training set and the last 30% would be used for the set test.

Since the Stochastic Gradient Descent method is used in the proposed model, it is necessary to shuffle the mini batches for the training set so that the cost function can be optimized. It is possible to choose randomly the number of iterations for the training set. It was observed that the iterations number for the mean absolute error varies according to each dataset.

The parameter related to the number of iterations for the training set was implemented in an online and deterministic way, based on [41]. The algorithm measures the mean absolute error and concludes the training set if the error doesn’t decrease after a certain number of iterations. The value of this parameter was specified as 20.

Other parameters from the RNN were adjusted: number of neurons, learning rate and mini-batches size. The adjustment was performed in the offline way, that is, before the algorithm execution. These values were chosen empirically, based on algorithm behavior. They are: number of neurons: 64; Learning Rate: 0.001 and finally, mini batches size: 64.

The ARIMA model parameters were defined based on [14]: $p,d$ and q, with p being the number of delay observations included in the model, d the degree of differentiation and q the size of the moving average window. The algorithm implemented in ARIMA follows the steps: (i) Call the $A\mathrm{R}IMA\left(\right)$ function and pass the parameters $p,d$ and q; (ii) Define the training data using the $fit\left(\right)$ function; (iii) Make predictions by calling the $fo\mathrm{R}ecast\left(\right)$ function, which performs the one-step-ahead forecast.

The SVM model [12] is made up of a set of supervised learning methods used for classification and regression. To deal with the regression problem the Support Vector Regression (SVR) [18] method was used. Therefore, it is necessary to define the SVR kernel function. In this implementation, the RBF [20] kernel was used.

4.2. Results and Analysis

In this section, we present results of experiments based on real datasets from Internet of Things monitoring applications. Specifically, only the last 10,000 detections from the air compressor monitoring dataset were used to predict future values, because it is an extensive dataset (almost four million detections). The remaining datasets were used in their entirety. In the results, the confidence interval is 95%.

The Figure 2 presents the results of predictions, Real and Predicted, obtained with the Recurrent Neural Networks (of the following datasets: SmartMeter (a), Water Quality (b), Dental Compressors (c), CO2 Emission (d) and Air Quality (e).

Based on the results, it was found that the prediction model with Recurrent Neural Networks finds it difficult to deal with values positioned at the ends of the series. Therefore, prediction in series with predominantly seasonal behavior may not achieve the best possible performance. The ARIMA model (Figure 3), in turn, demonstrates more assertiveness in predicting values positioned at the extremities, but ends up accumulating errors throughout the series, demonstrating a lower final performance than the Neural Network in this context.

However, compared with ARIMA (see Figure 3) and SVM (see Figure 4) models, the Recurrent Neural Network obtains better results. For instance, from the results with Turbidity (Burnett River) time series, it was found that, the model based on ARIMA obtained a lower value of Mean Absolute Error (MAE), but the model based on Recurrent Neural Network reached the highest level of Coefficient of Determination (${\mathrm{R}}^2$).

Regarding the execution time, results in Figure 5 show that the ARIMA model was faster than Recurrent Neural Network model. This behavior in the Recurrent Neural Network model is explained by the fact that the execution time comprises the training and test phases. Furthermore, the number of iterations in the training set depends on minimizing the Mean Absolute Error (EAM), which can lead to a very high number of iterations.

Based on the search results in Figure 5a, it appears that SVM models generally outperform ARIMA models in terms of processing time. It has been identified that the time taken by the SVM algorithm to completion is directly proportional to the size of the training set.

The box plots presented in Figure 5b and Figure 5c demonstrate the variation in observed data of Mean Absolute Error (EAM), Coefficient of Determination (${\mathrm{R}}^2$) through quartiles. The time series prediction model with Recurrent Neural Networks achieved the lowest mean of Mean Absolute Error and the highest mean of Coefficient of Determination.

In Figure 5c, in certain cases, the values obtained were negative, demonstrating the low quality of the prediction. According to [42], in cases where negative values of $R^2$ arise, the average of the actual data provides a better fit to the results than the values predicted by the prediction model.

Table 2 shows the values of Mean Absolute Error (MAE), Coefficient of Determination (${\mathrm{R}}^2$) and Execution Time (ET) obtained in the prediction from evaluated datasets. The model based on Recurrent Neural Network was executed 10 times for each dataset. The values presented in RNN column represent the average of the executions.

Statistical tests were performed to investigate the statistical difference between the results obtained from the EAM in each model presented in Table 2. The results do not follow a normal distribution, and it was identified, with a confidence interval calculated at 95%, that there is a statistical difference in the results obtained by the prediction models. The SVM model obtained the best result, followed by the Recurrent Neural Network and the ARIMA model, respectively.

In summary, the results presented regarding Prediction in the evaluated IoT datasets, EAM, $R^2$ and execution time with the models, RNN, ARIMA and SVM indicate that the Recurrent Neural Networks may have some difficulties in dealing with outliers. It is concluded that, the predictions of predominantly seasonal behavior series may not perform well with Recurrent Neural Network.

On the other hand, the model ARIMA presents better accuracy in dealing with outliers, but ends up accumulating errors throughout the time series, presenting an inferior performance regarding the EAM when compared with the RNN.

5. Conclusion

This paper presented a time series prediction model based on Recurrent Neural Networks implemented with the Stochastic Gradient Descent method to minimize cost. RNN was compared to the models, Auto-Regressive Integrated Moving Average (ARIMA) and Support Vector Machine (SVM). In the experiments, real datasets from IoT applications were used. We analyzed the prediction for many steps ahead with the optimization of the Mean Absolute Error and Coefficient of Determination.

The model based on the Recurrent Neural Network obtained better performance when compared to the SVM model, which attained higher Mean Absolute Error (MAE) and lower Coefficient of Determination (${\mathrm{R}}^2$).

As future work, it is suggested to improve the model so that it can use the multi-variety of the time series to obtain more assertive predictions in seasonal series. Also, to carry out new comparative tests with other datasets that demonstrate different behaviors (cycle, tendency, randomness).