Exploring Entropy-Based Classification of Time Series Using Visibility Graphs from Chaotic Maps

1. Introduction

Classification of time series based on entropy analysis and machine learning (ML) is a trending task in studying nonlinear signals. For example, EEG classification in diagnosing Alzheimer's disease [1,2] and Parkinson's disease [3,4,5,6]. There are many types of entropies, which in turn have several customizable parameters, for example, sample entropy (SampEn) [7], cosine similarity entropy (CoSiEn) [8], singular value decomposition entropy (SVDEn) [9], fuzzy entropy (FuzzyEn) [10], permutation entropy (PermEn) [11], etc. In this context, it is a problem of paramount importance to assess the effectiveness of the different entropies when used as features in ML classification. Recently, Velichko et al. proposed the use of a LogNNet neural network [12,13] for neural network entropy (NNetEn) calculation [1]. LogNNet neural network is a feedforward neural network that uses filters based on the logistic function and a reservoir inspired by recurrent neural networks, thus enabling the transformation of a signal into a high-dimensional space. Its efficiency was validated on the MNIST-10 data set [14]. They showed that the classification performance is proportional to the entropy of the time series and has a stronger correlation than the Lyapunov exponent of the time series used to feed the reservoir. The effectiveness of using NNetEn was shown on EEG signals in diagnosing Alzheimer's disease [1]. The effectiveness of fuzzy entropy was also demonstrated in the paper of Belyaev et al. [3] in diagnosing Parkinson's disease.

Natural visibility graph (NVG) graphs were introduced in [15] as a simple and computationally efficient method to represent a time series as a graph. Visibility graphs preserve the periodic and chaotic properties of the discrete map [15] , see also [16,17,18]. For example, periodic series result in regular graphs, random series in random graphs, and fractal series in scale-free graphs. Horizontal Visibility Graphs (HVG) were introduced in [19] to simplify the previously described NVG. Visibility graphs (VG) enable to reduce the complexity of calculations which depend on time series, while preserving the accuracy on the results, see for instance [20,21,22]

In [23], the authors describe the advantages of using the amplitude difference distribution instead of the degree distribution to collect information from the network formed by the horizontal visibility graph. Li and Shang introduce a combination of the amplitude difference distribution with discrete generalized past entropy to present a new method called Discrete Generalized Past Entropy based on the Amplitude Difference Distribution of the Horizontal Visibility Graph (AHVG-DGPE). The authors note its efficiency in systems evaluation and higher accuracy and sensitivity rate compared to the traditional method in characterizing dynamic systems, see also [24,25,26]

In this paper, we propose a method for assessing the effectiveness of entropies using chaotic mappings: We use it for analyzing the FuzzyEn and NNetEn entropies on four discrete mappings is given: the logistic map, the sine map, the Planck map, and the two-memristor based map. We use the corresponding HVG degrees representation of these time series, which implies that the resulting time series does not consist of real numbers but only of integer numbers. The results of using horizontal visibility graphs (HVG) to classify time series are also shown:

The major contributions of the paper are:

• A concept for comparing the efficiency of classifying chaotic time series using entropy features is presented.
• A new characteristic for assessing the global efficiency of entropy (GEFMCC) is presented.
• A comparison of the effectiveness of FuzzyEn and NNetEn was investigated. FuzzyEn is shown to have improved GEFMCC in the classification task compared to NNetEn. At the same time, there are local areas of the time series dynamics in which the classification efficiency NNetEn is higher than FuzzyEn. Matthews correlation coefficient was used to evaluate binary classification.
• The results of using HVG are shown. GEFMCC decreases after HVG time series transformation, but there are local areas of the time series dynamics in which the classification efficiency increases after HVG.

This paper is organized as follows: In Section 2, we introduce the methods we have used, while in Section 3, we explain the results we obtained, and finally, in Section 4, we discuss the results and state the conclusions, and we outline some ideas for future works.

2. Materials and Methods

2.1. Generation of synthetic time series

To generate synthetic time series, we used several types of the discrete chaotic map. The control parameter r_j (j = 1...Nr) varied discretely with step dr.

1. Logistic map [27,28]:

(1)

2. Sine map [29]:

(2)

3. Planck map [30]:

(3)

4. Two-memristor based map (TMBM) [31]:

(4)

The first 1,000 elements (x_-999…x₀) are ignored due to the transient period. If n > 0, then the time series is calculated for x_n. To generate a class corresponding to one value of r_j, 100-time series were generated with a length of N = 300 elements. Elements in each series were calculated sequentially, (x₁, …, x₃₀₀), (x₃₀₁, ..., x₆₀₀), etc. A set of 100 time series was generated at a given r_j. The value r_j ran through the entire range with a certain step dr, see equations (1-4).

2.2. Natural and Horizontal Visibility Graphs

Given a time series {(n,x_n)}_n∈ℕ indexed on the set of natural numbers ℕ, such that at time n the time series takes the value x_n, an association between each node and each pair (n, x_n) in order to obtain the graph associated to the time series. A natural visibility graph (NVG) is constructed as follows: given two nodes (n,x_n) and (m, x_m), these two nodes have visibility and thus they are connected in the graph by an edge if any other pair (c, x_c) with n < c < m satisfies

(5)

Horizontal Visibility Graphs (HVG) were introduced in [19,32] to simplify the requirements described for NVG.

When computing the HVG, each time series value is related a node in the resulting graph, as in the case of NVG. Two nodes (n, x_n) and (m, x_m) in this graph are connected if a horizontal line can be drawn connecting their corresponding visibility index without intersecting any intermediate value that is, if x_n, x_m > x_c for all n < c < m, see the examples in Figure 1.

Python library ts2vg was used to calculate HVG (‘time series to visibility graphs’) [33], which implements algorithms for plotting graphs based on time series data. The package utilizes a highly effective C backend for its operations (using Cython) and seamlessly integrates with the Python environment. As a result, ts2vg can effortlessly process input data from various sources using established Python tools. Additionally, it enables the examination and interpretation of the generated visibility graphs using a wide range of graph analysis, data science, visualization packages and tools compatible with Python. The HorizontalVG method was used to construct the HVG.

2.3. Time series classification metrics

We describe the method for calculating the classification metric for the time series of a discrete map for neighboring sets corresponding to two neighboring partitions by r.

Figure 2a shows a section of the buffering diagram of the logistic mapping with two adjacent sets of series corresponding to r_j-1 = 3.634 and r_j = 3.636; the distance between them corresponds to dr. Each set contains 100 time series. Examples of the first time series (x₁, …, x₃₀₀) for each set are shown in Figure 2b,c. FuzzyEn values for 100 time series in each set are shown in Figure 2d. We denote the average entropy value in each set as Entropy_AV (FuzzyEn_AV or NNetEn_AV).

As a result, we compiled a database with two classes. Class 1 contains 100 entropy values of time series generated at r_j = 3.636, and Class 2 contains 100 entropy values of time series generated at r_j-1 = 3.634. To classify the two classes, we will use the threshold model.

The single feature threshold approach involves a simple ML model with a single threshold Vth separating the two classes. A formula can represent the separation algorithm.

$$\mathrm{if} \mathrm{Entropy} \mathrm{value} \ge Vth \mathrm{then} (\mathrm{Class} 1) \mathrm{else} (\mathrm{Class} 2)$$

(6)

The search for Vth was carried out by sequential search within the limits of changes in the entropy feature, with the determination of the maximum MCC (Matthews correlation coefficient [34]) value. We calculated MCC for the entire database without dividing it into test and training data, which is equivalent to calculating MCC on training data.

MCC is the correlation coefficient between observed and predicted classifications; it returns a value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 is a random prediction, and -1 indicates the opposite, inverted prediction. The higher the MCC module is, the more accurate the prediction is, too. A negative MCC value means that the classes must be swapped. The MCC is calculated using the values of the confusion matrix as [34].

$$\mathrm{M}СС=\frac{(\mathrm{TP}\cdot \mathrm{TN}-\mathrm{FP}\cdot \mathrm{FN})}{\sqrt{(\mathrm{TP}+\mathrm{FP})\cdot (\mathrm{TP}+\mathrm{FN})\cdot (\mathrm{TN}+\mathrm{FP})\cdot (\mathrm{TN}+\mathrm{FN})}}$$

(7)

where TP stands for (True Positive), TN for (True Negative), FP for (False Positive), and FN for (False Negative). The MCC metric is a popular metric in machine learning, including binary classification.

Figure 2d shows an example where the classes are well separable and MCC = 1. Figure 3 shows an example of entropy distribution for classes with r_j-1 = 3.688 and r_j = 3.69. It can be seen that the classes are poorly separable and MCC ~ 0.45.

The MCC(r_j) dependence was calculated for all neighboring r_j-1 and r_j within the range of changes in r of each mapping j = 2…N_r. Let us introduce the concept of global efficiency (GEFMCC), which is calculated within the entire mapping under study using the formula

$$GEFMCC=\frac{{\displaystyle \sum _{j=2}^{Nr}\left|MCC(r_j)\right|}}{Nr-1}$$

(8)

where j = 2…Nr is the index of partition by r, N_r is the maximum number of partitions, see equations (1-4). The GEFMCC characteristic is equivalent to the average value of the dependence modulus MCC(r_j), and estimates the degree of entropy efficiency over the entire variety of time series of the chaotic mapping.

2.4. FuzzyEn calculation

FuzzyEn measures entropy based on fuzzily defined exponential functions for comparison of vectors similarity. It differs from approximate entropy and sample entropy, which use Heaviside function to calculate the irregularities in a time series data [35]. Fuzzy entropy can be calculated as follows. For a given time series $X=\left[x_1,x_2,\dots ,x_N\right]$ with given embedding dimension $\left(m\right)$, an $m-vectors$ as a form as:

$$X_m\left(i\right)=\left[x_i,x_{i+1},\dots ,x_{i+m-1}\right]-{x0}_i$$

(9)

These vectors represent $m$ consecutive $x$ values, starting with $ith$ point, with the baseline ${x0}_i=\frac{1}{m}{\sum }_{j=0}^{m-1}{x}_{i+j}$ removed. Then, the distance between vectors $X_m\left(i\right)$ and $X_m\left(j\right)$, $d_{ij,m}$ can be defined as the maximum absolute difference between their scalar components. Given $n$ and $r$, the degree of similarity $D_{ij,m}$ of the vectors $X_m\left(i\right)$ and $X_m\left(j\right)$ is calculated using fuzzy function.

$$D_{ij,m}=\mu \left(d_{ij,m}r\right)=exp\left(\frac{-{\left(d_{ij,m}\right)}^n}{r}\right)$$

(10)

The function ${\varphi }_m$ is defined as:

$${\varphi }_m\left(n,r\right)=\frac{1}{N-m}{\sum }_{i=1}^{N-m}\left(\frac{1}{N-m-1}{\sum }_{j=1,j\ne i}^{N-m}{D}_{ij,m}\right)$$

(11)

Repeating the same procedure from equation (9-10) for the dimension to $m+1$, vectors $X_{m+1}\left(i\right)$ are formed and the function ${\varphi }_{m+1}$ is obtained. Therefore, FuzzyEn can be estimated as:

$$FuzzyEn\left(m,n,r,N\right)=ln{\varphi }_m\left(n,r\right)-ln{\varphi }_{m+1}\left(n,r\right)$$

(11)

In the computation of fuzzy entropy, the embedding dimension $m=1$ and tolerance r = 0.2 x std were used in the analysis, where std is a standard deviation of $x_n$, argument exponent (pre-division) r₂ = 3 and time delay $\tau =1$.

2.5. NNetEn сalculation

Figure 4 shows the process for calculating NNetEn [1], it involves several key steps, which are detailed below:

Step 1: The initial step encompasses inputting the time series X = (x₁…x_N) of length N, into the reservoir.

Six main methods for filling the reservoir are researched in detail, as outlined in [36]. The methods M1 to M6 involve various techniques for filling the reservoir. M1—Row-wise filling with duplication; M2—Row-wise filling with an additional zero element; M3—Row-wise filling with time series stretching; M4—Column-wise filling with duplication; M5—Column-wise filling with an additional zero element; M6—Column-wise filling with time series stretching.

Step 2: Selection of embedded Datasets 1 and 2, upon which of the classification metric will be computed.

Step 3: Formation of the Y vector from the dataset, with the inclusion of a zero offset Y[0] = 1.

Step 4: Normalization of the Y vector.

Step 5: Multiplication of the Y vector with the reservoir matrix and the input vector Sh = W × Y to convert it into Sh vector.

Step 6: Feeding vector Sh into the input layer of the classifier, with a dimension of P_max = 25.

Step 7: Normalization of the vector Sh.

Step 8: Utilization of a single-layer output classifier.

Step 9 to 10: The neural network is trained according to backpropagation method with a variable number of epochs (Ep) and then tested. The parameter of the entropy function is referred as Ep.

Step 11: Transformation of the classification metric into the NNetEn entropy.

The parameters used in this work to calculate the entropy of NNetEn are: MNIST database data set (database = 'D1' and mu = 1), method for forming a reservoir from the M3 time series (method = 3), number of neural network training epochs (epoch) = 5 and the accuracy metric ('Acc').

3. Results

We present the results of calculating the dependencies between the FuzzyEn_AV(r) and NNetEn_AV(r) for various discrete mappings before and after the HVG transformation of the time series. In this way, we observe if visibility graphs retain enough information from the time series in order to calculate the entropies. The results of calculating the MCC(r) dependencies from which the characteristic of the global efficiency of GEFMCC from (8) is calculated are presented.

3.1 Results for logistic, sin and Planck maps

Figure 5a shows an example of a bifurcation diagram for a logistic map in the range of the control parameter 3.4 ≤ r ≤ 4, with a sampling step dr = 0.002. Figure 5b shows the FuzzyEn_AV(r) dependences before and after applying the HVG transformation. We can see that the HVG transformation leads to a significant increase in the entropy value, while some areas change their relative position. In regions A and B, we have reduced the entropies after having computed the HVGs, which is natural since they consist of ordered time series. The relative position of area C remains unchanged. The application of the HVG transformation has virtually no effect on the shape of the NNetEn_AV(r) graph, causing only a slight upward shift of entropies. The increase in FuzzyEn and NNetE values after HVG is due to the fact, in our opinion, that HVG has filtering properties and reduces the constant components of time series. However, FuzzyEn and NNetEn are sensitive to the constant component of the time series, which can be seen, for example, from the entropy values for r < 3.45.

Figure 6a shows the MCC(r) dependences for FuzzyEn before and after HVG transformation, as well as their difference ΔMCC, which is computed as follows:

$$\Delta \mathrm{MCC}=\left|\mathrm{MCC} \mathrm{after} \mathrm{HVG}\right|-\left|\mathrm{MCC}\right|$$

(14)

Positive values of ΔMCC > 0 indicate that, for a given value of r, the degree of classification of time series for r_j-1 and r_j increases due to the HVG transformation. Conversely, negative ΔMCC values indicate a decrease in classification efficiency after HVG transformation. According to the lower figure, we see that the HVG transformation can lead to both an increase and a decrease in classification efficiency for different r_j. We provide detailed calculations of the GEFMCC values in Table 1.

Figure 6b shows the MCC(r) dependences for NeNetEn before and after HVG transformation, and their difference ΔMCC. It can be seen that the amplitude of MCC for FuzzyEn is more significant than for NNetEn, which also affects the GEFMCC value in Table 1.

It is convenient to compare the local values of MCC(r) for FuzzyEn and NeNetEn using their difference ΔMCC (Figure 7).

$$\Delta \mathrm{MCC}=\left|\mathrm{MCC} \mathrm{for} \mathrm{NNetEn}\right|-\left|\mathrm{MCC} \mathrm{for} \mathrm{FuzzyEn}\right|$$

(14)

Figure 7 shows local areas of the dynamics of time series in which the classification efficiency NNetEn is higher than FuzzyEn (ΔMCC > 0), but most of the graph has ΔMCC < 0.

Similar results were obtained for the sine and Planck maps. Figure A1a (Appendix A) shows an example of a bifurcation diagram for a sin map in the control parameter range 0.7 ≤ r ≤ 2, with a sampling step dr = 0.005. Figure A1b,c shows the FuzzyEn_AV(r) and NNetEn_AV(r) dependences before and after applying the HVG transformation. Figure A2 shows the MCC(r) dependences for FuzzyEn and NNetEn before and after HVG transformation, and their difference ΔMCC.

Figure A3a (Appendix A) shows an example of a bifurcation diagram for a Planck map in the control parameter range 3 ≤ r ≤ 7, with a sampling step dr = 0.01. Figure A3b,c shows the FuzzyEn_AV(r) and NNetEn_AV(r) dependences before and after applying the HVG transformation. Figure 4 shows the MCC(r) dependences for FuzzyEn and NNetEn before and after HVG transformation, and their difference ΔMCC.

3.2. Results for TMBM map

The TMBM map is a multi-parametric and more complex than the mappings from section 3.1. Figure 8a shows an example of a bifurcation diagram for a TMBM map in the control parameter range -1.7 ≤ r ≤ -1.5, with a sampling step dr = 0.0005.

After applying of the HVG transformation, there is a notable increase in the entropy values of FuzzyEn_AV, as depicted in Figure 8b. Additionally, Figure 8c illustrates a consistent decrease in NNetEn_AV across a wide range of r following the utilization of HVF. Figure 9 shows the dependencies of MCC(r) and the discernible differences, denoted as ΔMCC, before and after the HVG transformation for both FuzzyEn (refer to Figure 9a) and NNetEn (refer to Figure 9b).

Figure 10 shows that there are local areas of the time series dynamics in which the classification efficiency NNetEn is higher than FuzzyEn (ΔMCC > 0), but most of the graph has ΔMCC < 0.

4. Discussion and Conclusions

In this work, we have proposed a method for assessing the effectiveness of entropy features using chaotic mappings, that enables to explore the efficiency of entropy-based classification of time series.

From Table 1, we have seen that the FuzzyEn has a better GEFMCC performance without the use of HVG transformation. At the same time, there are local areas of the time series dynamics in which the classification efficiency of NNetEn is higher than using FuzzyEn (Figure 7 and Figure 10). Nevertheless, despite of reducing the amount of signal information after HVG transformations, we see that there are local areas of the time series dynamics in which the classification efficiency increases when an HVG transformation is applied to the time series (Figure 6, Figure 9, Figure A2 and Figure A4). As it has been already seen in other contexts, HVGT transformations preserve structural properties of the time series [15,18]

All chaotic mappings analyzed in this work, present a similar GEFMCC trend when applying HVGs, and when comparing FuzzyEn and NNetEn, which indicates the universality of this characteristic. It is necessary to consider the fact that the results are given for specific entropy settings; for other parameters, the results may differ radically, since the effectiveness of entropies very much depends on the entropy calculation parameters. For future research, differing types of entropies can be compared under different parameters on one of the maps, for example, on the sine map. In addition, it is interesting to identify the dependence of GEFMCC on the length of the time series. Also, further research should be conducted in exploring other dynamical systems, as it is the case of fractional dynamical systems based on the logistic and sine maps [37,38,39], where visibility graphs have been already considered [40].

The fact that FuzzyEn turned out to be more effective in classifying short (N = 300) time series than NNetEn confirmed the results of our work on the classification of EEG signals [1]. However, individual pairs of time series can be better classified by NnetEn; this was also confirmed in the EEG experiment, where one channel performed better when using NNetEn as a feature. In the same work on EEG signals, the idea was put forward that classification based on several features may be better and the use of FuzzyEn and NNetEn may lead to a synergistic effect.