Time Series Prediction by Learning Interval Patterns

1. Introduction

Due to the negative impact of some physical and natural phenomena in man-made systems, it has always been challenging to predict the behavior of such phenomena to deal with these impacts. In this regard, it is highly important to predict the time series pertaining to such phenomena, and many methods have been proposed by researchers so far. A common model used to predict time series is (1):

$$y_t=f\left(y_{t-1},y_{t-2},\dots ,y_{t-n},{\mathbf{u}}_t\right).$$

(1)

In this model, the signal of time series $y_t$ is a function of n previous observations, and it is also a vector of $\mathbf{u}$ external factors. Different methods have been proposed to estimate the mapping function f which can be used to predict the future time values. The difference of these methods is in determining the number and type of factors in the argument in Relation (1) and also in determining a model by which the mapping function f can be related to the factors of argument. A group of initial methods such as Auto Regression (AR), Auto Regression Moving Average (ARMA), and Auto Regressive Moving Average Exogenous Inputs (ARMAX) are based linear regression modeling. The regression vector can be obtained from some nonlinear transformations of the initial information [1,2,3]. However, the majority of such methods have not resulted in a satisfactory answer due to the nonlinear nature of model in time series. A large group of nonlinear methods are based on using neural networks to find mapping f. Many architectures have been introduced for these networks to predict time series so far. In [4], a Finite Impulse Response (FIR) network, benefiting from a feed-forward neural network for learning, was proposed for prediction. In fact, FIR linear filters were used in this method for the weights of an MLP network. Similarly, a two-layered neural network was used in [5] where the network weights were considered dynamic functions. In [6], it has been proposed to use MLP neural networks of quaternionic type because this mathematical technique would facilitate learning in cases in which network connections become excessive due to high input and output dimensions.

In [7] a Recurrent Neural Network (RNN) with missing values, was used in learning a time series based on novel deep learning model. Like [4], a set of FIR filters was used in [8].

The only difference was that these filters were used in the first layer to preprocess information before the neural network. In [9], the Radial Basis Function (RBF) neural network was used to predict time series. Neural Fuzzy neural networks named Dynamic Evolving Neural-Fuzzy Inference System (DENFIS) were used in [10] to predict time series. Moreover, it was proposed in [10] to use neural networks, fuzzy logic and fractal theory at the same time. In many of these methods, different architectures were introduced to design neural networks; however, one method was used in preprocessing information in some other common neural networks.

1.1. Related Developments in Neural Time-Series Forecasting

Since the original interval-pattern formulation, neural time-series forecasting has developed substantially. Recurrent architectures remain important because long short-term memory (LSTM) networks address long-range gradient propagation [11], while gated recurrent units provide a compact recurrent alternative [12]. Convolutional sequence models have also shown that temporal convolutional networks can provide long effective memory without recurrent connections [13]. Probabilistic deep forecasting has been advanced by DeepAR [14] and deep state-space models [15], whereas N-BEATS introduced a strong basis-expansion architecture for univariate forecasting [16]. Attention-based multi-horizon prediction was further developed through the Temporal Fusion Transformer [17].

Transformer-based forecasting subsequently produced several architectures designed for long sequences, including Informer [18], Autoformer [19], FEDformer [20], Pyraformer [21], and PatchTST [22]. TimesNet generalized temporal modelling by transforming one-dimensional series into two-dimensional representations [23]. At the same time, the strong performance of simple linear baselines such as DLinear demonstrated the importance of careful benchmarking in long-term forecasting [24]. Broader surveys and empirical studies have documented the rapid expansion of deep-learning methods for forecasting and time-series analysis [25,26]. Other influential directions include scalable additive forecasting [27], deep belief networks for nonlinear forecasting [28], sample-convolution and interaction networks [29], transformer approaches for influenza-like illness forecasting [30], and neural forecasting methods that explicitly quantify predictive uncertainty [31]. These developments provide a contemporary context for the interval-pattern approach: rather than increasing model complexity directly, the proposed method seeks to simplify the mapping learned by each neural-network layer.

More broadly, neural networks and computational-intelligence methods have been applied to nonlinear modelling, prediction, recognition, and control across diverse domains. Of particular relevance to forecasting, multilayer perceptron and fuzzy-inference models have been investigated for short-term load forecasting [32], while learning-based approaches have also been used for longitudinal and clinical prediction problems [33,34,35,36]. These applications illustrate the ability of data-driven models to represent complex nonlinear relationships.

Related machine-learning and pattern-recognition research includes face detection and recognition [37,38,39,40], palmprint and signature recognition [41,42], deformable texture matching [43], and broader developments in computer vision [44]. More recent applications include multiclass machine-learning software [45], retinal-vessel analysis [46], and transformer-based Alzheimer’s disease detection from retinal imaging [47].

Computational modelling and optimization methods have additionally been investigated in wireless computing and communication systems [48,49], model-predictive mobile-robot control [50], and scalable load balancing in distributed computing [51]. Computational approaches have also been employed in studies outside engineering, including investigations of relationships between thinking styles and self-efficacy [52]. Although these applications address different problem domains, they collectively demonstrate the broad use of computational modelling, learning, and optimization techniques for representing complex relationships in data and dynamic systems.

In this paper, it was intended to estimate a mapping for modeling with the help of MLP networks after preprocessing the information of time series and without using external factors. This preprocess includes generating interval patterns based on the input and output patterns. These patterns are defined in enough numbers between the information of input and output patterns, and learning the mapping between them turns into learning the mapping of successive interval patterns. These patterns can weaken the nonlinear and non-identity factors of the general mapping to replace it with some quasi-identity mappings. Using a number of neural networks having one hidden layer which will finally reconstruct an MLP network, these mappings were successively learned. Therefore, a simple and efficient architecture of this neural network was used for modeling. In the second part, the inference of model structure is discussed with respect to the state space, and the theorem of possible minimum dimension of time series values is presented and proven for a nonlinear mapping of relation (1). In Part 3, the way interval patterns are generated and used to design an MLP network is discussed. Finally, the capability of this method is compared with some methods in predicting time series pertaining to the number of activities of solar storms, and then the results are presented.

2. Inferring the Model for Time Series Based on the State Space

The state space is a comprehensive and appropriate tool used to investigate physical phenomena. Many of the dynamic behaviors of the system can be displayed and analyzed in the state space which is widely used in dynamic and control problems. On the other hand, a time series represents a part of the physical reality of a natural phenomenon which can be considered a result of observing a part of the main fuzzy path in a natural system in the state problem. All the quantities of the reference device do not necessarily influence a time series. However, it is certain that the information of time series is enough to observe all the quantities of state influencing it; therefore, a second device of state space can be assumed. It is actually a time series resulting from the direct observation of fuzzy path of this subsystem. The type of signal behavior in time series can be both influenced by the linearity or nonlinearity of the observed tool and the type of fuzzy path behavior according to dynamic equations. The state equations resulting from the second state device and the time series equation, assuming the invalidity of these equations to time, are indicated in discrete time in (2) and (3), respectively.

$${\mathbf{x}}_{t+1}=\mathbf{G}\left({\mathbf{x}}_t\right).$$

(2)

$$y_t=h\left({\mathbf{x}}_t\right).$$

(3)

It is assumed that the vector of mapping function G(0) is continuous and even to create a peer-to-peer mapping among all the state vectors. It is also assumed that the mapping function h(0), which is the generator of time series on the state vector, is continuous and even to create one and only one output for each state input vector. The dimension of subsystem cannot be specified by only observing the output observation signal.

Theorem: Assuming that the path phase of the system does not alternate during the observation time and does not reach a balance point, it can be stated that this path never crosses itself, and one state vector is not placed on one state path two times. Therefore, two different time signal sets like $\{y_t,y_{t-1},\dots ,y_{t-n+1}\}$will not be equal for the observed time sets of the time series which is$\{y_{t^{\prime }},y_{t^{\prime }-1},\dots ,y_{t^{\prime }-n+1}\}$. To prove this theorem, Equations (2) and (3) can be used as follows:

$$y_{t+i}=h\left({\mathbf{G}}^i\left({\mathbf{x}}_t\right)\right),i=0,1,\dots ,n-1.$$

(4)

Therefore, n successive and independent external observations result in:

$${\mathbf{Y}}_t={\left[\begin{array}{cccc}{y}_t& y_{t-1}& \dots & y_{t-n+1}\end{array}\right]}^T=\mathbf{H}\left({\mathbf{x}}_t\right).$$

(5)

On the other hand, it was assumed that${\mathbf{x}}_{t_1}\ne {\mathbf{x}}_{t_2}$; therefore, according to Equation (5) and given the peer-to-peer characteristics of G function and G, we have:

$${\mathbf{Y}}_{t_1}\ne {\mathbf{Y}}_{t_2}\mathrm{for}{\mathbf{x}}_{t_1}\ne {\mathbf{x}}_{t_2}.$$

(6)

Having only one time series or system (2), the dimension n is unknown; however, it can be stated that if Equation (6) is true for$n=n_1$, n₁ has the necessary condition to be a dimension of system. Moreover, given equations (4) and (5), it can be stated that:

$$y_t=g\left(y_{t-1},y_{t-2},\dots ,y_{t-n}\right).$$

(7)

Equation (7) indicates that there was a nonlinear and autonomous relationship for the observed output values for prediction, and the time series was predicted from the previous n values at each moment only if mapping g(0) could be predicted. There are three states in dimension selection:

3. Interval Patterns and MLP Network

Interval patterns can be defined from input and output patterns. The dimensions of input and output patterns were assumed to be equal in order to for the interval patterns to be defined simpler. Figure 1 indicates a visual sample of defining interval patterns.

According to this figure, both successive patterns became more similar after entering interval patterns. For more investigation, it is assumed that N_p is the pair input and output pattern of available ${\mathbf{X}}_p$and${\mathbf{Y}}_p$$(p=1,\dots ,N_p)$, then a simple and useful definition of interval patterns can be obtained on the patterns existing in Equation (7).

$${\mathbf{P}}_p^{\left(k\right)}={\mathbf{X}}_p+{\displaystyle \frac{k}{K+1}}\left({\mathbf{Y}}_p-{\mathbf{X}}_p\right),k=0,1,\dots ,K+1,$$

(8)

where ${\mathbf{P}}_p^{\left(0\right)}={\mathbf{X}}_p$ and ${\mathbf{P}}_p^{(K+1)}={\mathbf{Y}}_p$.

As it is obvious, K refers to the number of interval patterns between two input and output patterns. In a theorem based on definition (8), it is indicated here how it is possible to learn successive mappings quicker and more easily.

Theorem: The search space of an output pattern of ${\mathbf{Y}}_p$ to reduce the learning error through the input pattern of ${\mathbf{X}}_p$is $M_p$ which is equal to the corresponding value in the two interval and successive patterns in Definition (8) in which there are n dimensions of patterns and k interval patterns.

It is assumed that each group of input and output patterns of ${\mathbf{X}}_p$and${\mathbf{Y}}_p$ in which ${\mathbf{X}}_p,{\mathbf{Y}}_p\in {\mathbb{R}}^n$ is placed in a state space of n dimensions, and the following mapping is true for these two groups of patterns.

$${\mathbf{Y}}_p=f\left({\mathbf{X}}_p\right),p=1,\dots ,N_p.$$

(9)

On the other hand, the Euclidean distance between the two corresponding input and output patterns is defined as follows:

$$d_p={∥{\mathbf{Y}}_p-{\mathbf{X}}_p∥}_2=\sqrt{\sum _{i=1}^n{\left(y_{p,i}-x_{p,i}\right)}^2}.$$

(10)

The value of $d_p$indicates the distance between two input and output patterns in the state space, and the subspace of Mj can be defined in the form of a circular cloud as the center of input pattern with the diameter of $d_p$:

$$M_p=\left\{\mathbf{z}\in {\mathbb{R}}^n:{∥\mathbf{z}-{\mathbf{X}}_p∥}_2\le d_p\right\}.$$

(11)

The size of this subspace is the searchable space for each of the output patterns so that the learning error can be minimized. This volume is equal to:

$$V_p=C_n{d}_p^n,$$

(12)

In which $C_n$ is a constant. In a similar way, the following equation can be true for the two interval patterns of the distance between two successive patterns with respect to definition (8):

$$d_p^{\left(k\right)}={∥{\mathbf{P}}_p^{(k+1)}-{\mathbf{P}}_p^{\left(k\right)}∥}_2={\displaystyle \frac{d_p}{K+1}}.$$

(13)

The maximum searchable space for the interval pattern ${\mathbf{P}}_p^{\left(k\right)}$ is also as follows:

$$V_p^{\left(k\right)}=C_n{\left({\displaystyle \frac{d_p}{K+1}}\right)}^n={\displaystyle \frac{V_p}{{(K+1)}^n}}.$$

(14)

Therefore, relations (12) and (14) result in:

$${\displaystyle \frac{(K+1)V_p^{\left(k\right)}}{V_p}}={\displaystyle \frac{1}{{(K+1)}^{n-1}}}.$$

(15)

The theorem has now been proven. Just as a secondary result, it can be stated that:

• The total search space used for each output patterns to reduce the estimation error and optimize learning parameters with the help of interval patterns method, benefiting from K interval patterns, is $1/{(K+1)}^{n-1}$ times its corresponding value without using interval patterns.

This theorem obviously indicates that the search domain will be smaller even in the presence of interval patterns, and learning is quicker. Therefore, if the right number of input patterns is defined, a neural network having one hidden layer and a small number of neurons can be used to learn each of successive mappings.

Moreover, in the Taylor expansion pertaining to the mapping between two input and output patterns, we have:

$$f(\mathbf{X}+\Delta \mathbf{X})=f\left(\mathbf{X}\right)+{\mathbf{J}}_f\left(\mathbf{X}\right)\Delta \mathbf{X}+\mathcal{O}\left({\parallel \Delta \mathbf{X}\parallel }^2\right),$$

(16)

Considering (8) and (16), it is clear that the summation of nonlinear expressions and also the subtraction of linear expression from identity expression in the mapping relation of both two successive patterns were decreased K+1 times its corresponding value in the total mapping. On the other hand, neural networks having a smaller number of neurons with simpler structures can be found to operate as an almost identity operator. Therefore, with the similar number of sample neural networks for both successive patterns, a mapping can be found between the set of input and output patterns. The diagrams indicated in Figure 2 and Figure 3 show the architecture used in the MLP network.

After obtaining the coefficient matrices for each training neural network between two successive patterns, the coefficient matrix of output layer in each neural network is merged into the coefficient matrix of the input layer of the next neural network to create an MLP network.

However, it is difficult to use interval patterns according to definition (8) regarding time series because an output signal like $y_t$ is preferred to be modeled and finally predicted for the previous multi-signal set like ${[y_{t-1},y_{t-2},\dots ,y_{t-n}]}^T$ (relation (1)). Nevertheless, some techniques can be used to equalize the dimension of the output pattern with the dimension of the input pattern. It should only be considered that signal $y_t$ is to be obtained from the output pattern at each time in this dimension increase, and the patterns should be defined in a way that the mapping between two patterns will not be more complicated than the mapping presented in (1). The method which was taken into account here was very simple. It was assumed that the output pattern was defined like ${\mathbf{Y}}_t={[y_t,y_{t-1},\dots ,y_{t-n+1}]}^T$ for each input pattern like ${\mathbf{X}}_t={[y_{t-1},y_{t-2},\dots ,y_{t-n}]}^T$. The only difference is that the first member of the output pattern was signal $y_t$ which replaced y_t-1 in the input pattern. Using the input and output patterns, the interval patterns can be defined according to (8).

4. Simulation

One of the applications of predicting time series is to predict solar storms. This phenomenon has a tangible impact on the weather. On the one hand, the radiation of cosmic ray resulting from solar storms influences the operations of satellites and telecommunications. For instance, the prediction of time series pertaining to this phenomenon was stimulated using the proposed method. The results were compared with some known methods. One of the common criteria used to determine the quality of predicting solar storms was to predict it in the method of a few steps forward between 1871 and 1881. In this study, the information pertaining to the annual solar storms between 1770 and 1870 was used to form patterns and train the network. The prediction results can be seen in Table 1 [53].

This table clearly indicates that prediction done in the interval patterns method resulted in fewer errors. Figure 3 indicates the results of prediction. In this method, 4 interval patterns (IP(4)) were used, and each neural network had 4 neurons in the hidden layer. IP(4) was finally used in a 5-layer MLP network. Using back propagation method, each layer was learnt. Symmetric sigmoid functions were used as the activating functions in each hidden layer. In the second layer, the linear activating functions were used. All the specifications of the above network were obtained by trial and error for the optimized state. The appropriate dimension was also detected for the regression vector in trial and error while considering $n=2$. However, using result (7) in determining the minimum dimension also resulted in $n=2$.

In the second part, simulation was done for the prediction of solar storms for 2 years forward between 1970 and 1889, in which several known methods of prediction were used. Table 2 indicates the results [54].

Table 1. Comparing the Prediction Results of Some Known Methods in 11 Steps forward with the Interval Patterns Method.

Maximum Absolute Error	Root Mean Square Error	Prediction Horizon	Explain	Predictor
81	32.6	11	Autoregressive	AR[1]
81	32.6	11	Bilinear Predictor	BL[55]
76	35	11	-	SETAR[56]
47	21.4	11	Optimal Error Predictor	OEP[57,58]
81	29.4	11	-	GMDH[58,59]
45	19.9	11	Nonlinear Set Membership	NSM11[53]
63	23.4	11	Neural Network	NN11[53]
31	12.7	11	Interval Patterns	I P(4)

Table 1. Comparing the Prediction Results of Some Known Methods in 11 Steps forward with the Interval Patterns Method.

Maximum Absolute Error	Root Mean Square Error	Prediction Horizon	Explain	Predictor
81	32.6	11	Autoregressive	AR[1]
81	32.6	11	Bilinear Predictor	BL[55]
76	35	11	-	SETAR[56]
47	21.4	11	Optimal Error Predictor	OEP[57,58]
81	29.4	11	-	GMDH[58,59]
45	19.9	11	Nonlinear Set Membership	NSM11[53]
63	23.4	11	Neural Network	NN11[53]
31	12.7	11	Interval Patterns	I P(4)

As observed, the proposed method indicated better results. The criterion normalized mean square error, used as the comparison criteria in Table 2, is defined as follows:

$$E_{\mathrm{NRMSE}}=\sqrt{{\displaystyle \frac{{\sum }_{t=1}^N{({\widehat{y}}_t-y_t)}^2}{{\sum }_{t=1}^N{(y_t-\overline{y})}^2}}}.$$

(17)

Table 2. Comparing the Results of Predicting Some Known Methods in 20 Steps forward in the Method of Interval Patterns.

Normalized Mean Square Error	Prediction Horizon	Explain	Predictor
0.252	20	Autoregressive Moving Average	ARMA[1]
0.348	20	Specified Network Recurrent	Elman[54]
0.115	20	Finite Impulse Response	FIR[54]
0.162	20	Elman & FIR	Extended Elman[54]
0.0957	20	Interval Patterns	IP(4)

Table 2. Comparing the Results of Predicting Some Known Methods in 20 Steps forward in the Method of Interval Patterns.

Normalized Mean Square Error	Prediction Horizon	Explain	Predictor
0.252	20	Autoregressive Moving Average	ARMA[1]
0.348	20	Specified Network Recurrent	Elman[54]
0.115	20	Finite Impulse Response	FIR[54]
0.162	20	Elman & FIR	Extended Elman[54]
0.0957	20	Interval Patterns	IP(4)

In this criterion,$y_t$,${\widehat{y}}_t$ and $\overline{y}$ refer to the observed information, their mean, and information resulting from prediction, respectively. The MLP network used to predict 20 steps forward is the same as the specifications of neural network used to predict 11 steps forward. The only difference was that the information pertaining solar storms between 1700 and 1869 was used here to create patterns and train the network. The result of this prediction can be seen in Figure 4.

Figure 4. Prediction of 11 steps forward using the interval-pattern method.

Figure 4. Prediction of 11 steps forward using the interval-pattern method.

Figure 5. Prediction of 20 steps forward using the interval-pattern method.

Figure 5. Prediction of 20 steps forward using the interval-pattern method.

5. Conclusion

Pointing out the fact that many methods have been proposed to predict time series so far, a new architecture of the MPL neural network was introduced in this paper. In this architecture, layers were trained independently based on interval patterns which were defined with respect to the input and output patterns resulting from the information of time series. Therefore, a complicated and volatile mapping between two inputs and output patterns would turn into simple and identity mappings between successive patterns of the interval patterns, and the neural network would be trained simply and more efficiently. The minimum dimension which would be possible for a time series was also discussed. The prediction of some steps forward of solar storms was considered and compared with other known methods. The results clearly indicated the superiority of the proposed method.