Entropy of Difference

1. Introduction

Permutation entropy (PE), introduced by Bandt and Pompe [1], as well as its modified version [2], are both efficient tools to measure the complexity of chaotic time series. Both methods propose to analyze time series: $X=(x_1,x_2,\cdots x_k\cdots )$ by first choosing an embedding dimension m to split the original data in a subset of m-tuples: $((x_1,x_2\cdots x_m),(x_2,x_3,\cdots x_{1+m}),\cdots )$, then to substitute to the m-tuples values by the rank of the values, resulting in a new symbolic representation of the time series. For example, consider the time series $X=(0.2,0.1,0.6,0.4,0.1,0.2,0.4,0.8,0.5,1.,0.3,0.1,\cdots )$. Choosing, for example, an embedding dimension $m=4$, will split the data in a set of 4-tuples: $X_4=((0.2,0.1,0.6,0.4),(0.1,0.6,0.4,0.1),(0.6,0.4,0.1,0.2),\cdots )$. The Bandt-Pompe method will associate the rank of the value with each 4-tuples. Thus, in $(0.2,0.1,0.6,0.4)$ the lowest element $0.1$ is in position 2, the second element $0.2$ is in position 1, $0.4$ is in position 4 and finally $0.6$ is in position 3. Thus the 4-tuple $(0.2,0.1,0.6,0.4)$ is rewritten as $(2,1,4,3)$. This procedure thus results in each $X_4$ to be rewritten as a symbolic list:$\left(\right(2,1,4,3),(1,4,3,2),(3,4,2,1)\cdots )$. Each element is then a permutation $\pi$ of the set $(1,2,3,4)$. Next, the probability of each permutation $\pi$ in $X_m$ is then computed: $p_m\left(\pi \right)$, and finally the PE for the embedding dimension m, is defined as ${\mathrm{PE}}_m\left(X\right)=-{\sum }_{\pi }{p}_m\left(\pi \right)log\left(p_m\left(\pi \right)\right)$. The modified permutation entropy (mPE) just deals with those cases in which equal quantities may appear in the m-tuples. For example for the m-tuple $(0.1,0.6,0.4,0.1)$, computing PE will produce $(1,4,3,2)$ while computing mPE will associate $(1,1,3,2)$. Both methods are widely used due to their conceptual and computational simplicity [3,4,5,6,7,8]. For random signals, PE leads to a constant probability $q_m\left(\pi \right)=1/m!$ (for white Gaussian noise), which does not make it possible to evaluate the “distance" between the probability found in the signal: $p_m\left(\pi \right)$ and the probability produced by a random signal: $q_m$, with the Kullback-Leibler (KL) divergence [9,10]: ${\mathtt{KL}}_m(p\parallel q)={\sum }_{\pi }{p}_m\left(\pi \right){log}_2(p_m\left(\pi \right)/q_m\left(\pi \right))$. Furthermore, the number ${\mathtt{K}}_m$ of m-tuples are $m!$ for PE and even greater for mPE [2], thus requiring then a large data sample to perform significant statistical estimation of $p_m$.

2. Entropy of difference-method

The entropy of difference (ED) method proposes to substitute to the m-tuples with strings s containing the sign (“+" or “-"), representing of the difference between subsequent elements in the m-tuples. For the same $X_4$: $\left(\right(0.2,0.1,0.6,0.4),(0.1,0.6,0.4,0.1),(0.6,0.4,0.1,0.2),\cdots )$ this leads to the representation: $(``-+-",``+--",``--+",\cdots )$. For an m value, we have $2^{m-1}$ strings s from $``+++\cdots +"$ to $``---\cdots -"$. Again we compute, in the time series, the probability distribution $q_m\left(s\right)$ of these strings s and define the entropy of difference of order m as: ${\mathrm{ED}}_m=-{\sum }_s{q}_m\left(s\right){log}_2{q}_m\left(s\right)$. The number of elements: ${\mathtt{K}}_m$ to be treated, for an embedding m, are smaller for ED compared with the number of permutations $\pi$ in PE or to the elements in mPE (see Table 1).

Furthermore the probability distribution for a string s, in a random signal: $q_m\left(s\right)$ is not constant and could be computed through the recursive equation. Indeed let’s $P(X_t=x)=p\left(x\right)$ be the probability density for the signal variable $X_t$ at time t, and let’s $F\left(x\right)$ the corresponding cumulative distribution function ($F\left(x\right)={\int }^{x}p\left(x\right)dx$). Let’s make the hypothesis that the signal is not correlated in time, which means that the join probability is just the product of probability: $P(X_{t_1}=x_1,X_{t_2}=x_2)=P(X=x_1)P(X=x_2)$. Under these conditions, we can easily evaluate the $q_m\left(s\right)$. For example for $m=3$, we have 4 probabilities: $q_3(+,+),q_3(+,-),q_3(-,+)$ and $q_3(-,-)$. These give respectively:

$$\begin{array}{ccc}\hfill q_3(+,+)& =& \int \int \int d{x}_{1}d{x}_{2}d{x}_{3}p\left(x_1\right)p\left(x_2\right)p\left(x_3\right)\theta (x_3-x_2)\theta (x_2-x_1)\hfill \\ & =& \int d{x}_{3}p\left(x_3\right){\int }^{x_3}d{x}_{2}p\left(x_2\right){\int }^{x_2}d{x}_{1}p\left(x_1\right)\hfill \\ & =& \int d{x}_{3}p\left(x_3\right){\int }^{x_3}d{x}_{2}p\left(x_2\right)F\left(x_2\right)\hfill \\ & =& \int d{x}_{3}p\left(x_3\right)\frac{1}{2}F{\left(x_3\right)}^2\hfill \\ & =& \frac{1}{6}\hfill \end{array}$$

(1)

and

$$\begin{array}{ccc}\hfill q_3(+,-)& =& \int \int \int d{x}_{1}d{x}_{2}d{x}_{3}p\left(x_1\right)p\left(x_2\right)p\left(x_3\right)\theta (x_3-x_2)\theta (x_1-x_2)\hfill \\ & =& \int \int d{x}_{2}d{x}_{3}p\left(x_2\right)p\left(x_3\right)\theta (x_3-x_2)-q_3(+,+)\hfill \\ & =& \int d{x}_{3}p\left(x_3\right)F\left(x_3\right)-\frac{1}{6}=\int d{x}_3{F}^{\prime }\left(x_3\right)F\left(x_3\right)-\frac{1}{6}\hfill \\ & =& \frac{1}{2}-\frac{1}{6}=\frac{2}{6}\hfill \end{array}$$

(2)

This result is totally independent of the probability density $p\left(x\right)$ provided that the signal is not correlated in time. We can proceed in the same way for any $q_m\left(s\right)$ and thus obtain a recurrence on the $q_m\left(s\right)$ 1 (in the following equations x and y are strings made of “+" and “-"):

$$\begin{array}{c}\hfill q_2(+)=q_2(-)=\frac{1}{2}\\ \hfill q_{m+1}\left(\underset{m}{\underbrace{+,+,+,\cdots ,+}}\right)=\frac{1}{(m+1)!}\\ \hfill q_{m+1}(-,\mathrm{x})=q_m\left(\mathrm{x}\right)-q_{m+1}(+,\mathrm{x})\\ \hfill q_{m+1}(\mathrm{x},-)=q_m\left(\mathrm{x}\right)-q_{m+1}(\mathrm{x},+)\\ \hfill q_{m+1}(\mathrm{x},-,\mathrm{y})=q_{a+1}\left(\mathrm{x}\right)q_{b+1}\left(\mathrm{y}\right)-q_{m+1}(\mathrm{x},+,\mathrm{y})\mathrm{with}a+b+1=m\end{array}$$

(3)

leading to a complex probability distribution for the $q_m\left(s\right)$. For example for $m=9$ we have $2^8=256$ strings with the highest probability for the $``+-+-+-+-"$ string (and its symmetric $``-+-+-+-+"$): $q_9\left(\max \right)=\frac{62}{2835}\approx 0.02187$ (see Figure 1). These probabilities $q_m\left(s\right)$ could then be used to determine the KL-divergence between the time series probability $p_m\left(s\right)$ and the random uncorrelated signal.

To each string s we can associate an integer number, it’s binary representation, through the substitutions $-\to 0$, $+\to 1$. So, for $m=4$ we have $``---"=0$, $``--+"=1$, $"-+-"=2$, $``-++"=3$ and so on up to $``+++"=7$.

Table 3. ${\mathrm{ED}}_m$ values, for different m-embedding.

${\mathrm{ED}}_2=$	1
${\mathrm{ED}}_3=$	$\frac{1}{3}+{log}_2\left(3\right)=1.9183$
${\mathrm{ED}}_4=$	$3+\frac{1}{2}{log}_2\left(3\right)-\frac{5}{12}{log}_2\left(5\right)=2.82501$
${\mathrm{ED}}_5=$	$\frac{47}{30}+\frac{3}{10}{log}_2\left(3\right)+{log}_2\left(5\right)-\frac{11}{60}{log}_2\left(11\right)=3.72985$

Table 3. ${\mathrm{ED}}_m$ values, for different m-embedding.

${\mathrm{ED}}_2=$	1
${\mathrm{ED}}_3=$	$\frac{1}{3}+{log}_2\left(3\right)=1.9183$
${\mathrm{ED}}_4=$	$3+\frac{1}{2}{log}_2\left(3\right)-\frac{5}{12}{log}_2\left(5\right)=2.82501$
${\mathrm{ED}}_5=$	$\frac{47}{30}+\frac{3}{10}{log}_2\left(3\right)+{log}_2\left(5\right)-\frac{11}{60}{log}_2\left(11\right)=3.72985$

The recurrence gives some specific $q_m$. To simplify the notations, let’s write $a_{+}$ a set of a successive “+". For example the second and third rules gives

$$\begin{array}{ccc}& q_{m+1}(a_{+},-)& =q_m\left(a_{+}\right)-q_{m+1}(a_{+},+)=\frac{1}{m!}-\frac{1}{(m+1)!}\hfill \\ & q_{m+1}(a_{+},-)& =q_{m+1}(-,a_{+})=\frac{m}{(m+1)!}\hfill \end{array}$$

(4)

then

$$\begin{array}{ccc}& q_{m+1}(a_{+},-,b_{+})& =q_{a+1}\left(a_{+}\right)q_{b+1}\left(b_{+}\right)-q_{m+1}(a_{+},+,b_{+})=\hfill \\ & q_{m+1}(a_{+},-,b_{+})& =\frac{1}{(m+1)!}\left(C_{a+1}^{m+1}-1\right)\mathrm{with}:b+a+1=m\hfill \end{array}$$

(5)

We can also write

$$\begin{array}{c}\hfill q_{m+1}(a_{+},-,b_{+},-,c_{+})=q_{a+1}\left(a_{+}\right)q_{b+c+2}(b_{+},-,c_{+})-q_{m+1}(a_{+},+,b_{+},-,c_{+})=\\ \hfill a+b+c+2=m\\ \hfill q_{m+1}(a_{+},-,b_{+},-,c_{+})=\frac{1}{(a+1)!}\frac{1}{(m-a)!}\left(C_{b+1}^{m-a}-1\right)-\frac{1}{(m+1)!}\left(C_{m-c}^{m+1}-1\right)=\\ \hfill =\frac{1}{(a+1)!(b+1)!(c+1)!}-\frac{1}{(c+1)!(a+b+2)!}-\frac{1}{(a+1)!(b+c+2)!}+\frac{1}{(a+b+c+3)!}\end{array}$$

(6)

This equation is also valid when $b=0$ so for $q_{m+1}(a_{+},-,-,c_{+})$ (with $m=a+c+2$) or for $c=0$. We can continue in this way and determine the general values of $q_{m+1}(a_{+},-,b_{+},-,c_{+},-,d_{+})$ and so on.

In the case where the data are integers, we can avoid the situation where two successive data are equal ($x_i=x_{i+1}$) by adding a small amount of random noise. For example, we take the first ${10}^4$ decimal of $\pi$ (and we add a small noise $ϵ\in [-0.01,0.01]$) and we have:

Table 4. $q_m$ values for $\pi$, for different m-embedding.

$6q_3=$	0.982	2.01	2.01	0.991
$24q_4=$	0.924	3.00	5.05	3.00	3.00	5.05	3.00	0.960
$120q_5=$	0.756	3.86	9.10	5.92	9.23	16.0	11.0	4.03	3.86	11.1	16.2	9.10
	5.78	9.22	4.03	0.768

Table 4. $q_m$ values for $\pi$, for different m-embedding.

$6q_3=$	0.982	2.01	2.01	0.991
$24q_4=$	0.924	3.00	5.05	3.00	3.00	5.05	3.00	0.960
$120q_5=$	0.756	3.86	9.10	5.92	9.23	16.0	11.0	4.03	3.86	11.1	16.2	9.10
	5.78	9.22	4.03	0.768

Table 5. ${\mathrm{ED}}_m$ values for $\pi$, for different m-embedding.

${\mathrm{ED}}_2=$	$0.999998$
${\mathrm{ED}}_3=$	$1.91361$
${\mathrm{ED}}_4=$	$2.81364$
${\mathrm{ED}}_5=$	$3.71059$

Table 5. ${\mathrm{ED}}_m$ values for $\pi$, for different m-embedding.

${\mathrm{ED}}_2=$	$0.999998$
${\mathrm{ED}}_3=$	$1.91361$
${\mathrm{ED}}_4=$	$2.81364$
${\mathrm{ED}}_5=$	$3.71059$

Despite the complexity of $q_m\left(s\right)$, the Shannon entropy for a random signal: ${\mathrm{ED}}_m=-{\sum }_s{q}_m\left(s\right){log}_2{q}_m\left(s\right)$ increases linearly with m (see Figure 2): ${\mathrm{ED}}_m=-0.799574+0.905206m$ . If the m-tuples where equiprobable it will leads to $-{log}_2\left(2\right)+m{log}_2\left(2\right)=m-1$.

3. Periodic signal

Let’s see what happens with a period 3 data $X=(x_1,x_2,x_3,x_1,x_2,x_3,\cdots )$. To evaluate the $q_m$ we only have 3 types of 2-tuples. For example for $q_2$ we have $((x_1,x_2)$, $(x_2,x_3)$, $(x_3,x_1))$. We have only two possible string “+" or “-", so the probabilities mustt be $q_2(+)=2/3,q_2(-)=1/3$ or $q_2(+)=1/3,q_2(-)=2/3$. For $q_3$ again we have only 3 types of 3-tuples: $((x_1,x_2,x_3)$, $(x_2,x_3,x_1)$, $(x_3,x_1,x_2))$. We have $2^2$ possible string $(+,+)$, $(+,-)$, $(-,+)$ and $(-,-)$. The consistency of the inequalities between $x_1$, $x_2$ and $x_3$ reduces the number of possible strings to 3. For example, if $(x_1,x_2,x_3)$ gives $(+,+)$, then $(x_2,x_3,x_1)$ must be $(+,-)$ and $(x_3,x_1,x_2)$ must be $(-,+)$. Due to period 3 these $\left(\mathtt{x},\mathtt{y}\right)$ will appear $1/3$ times. To evaluate $q_4$ we have again only 3 types of 4-tuples: $((x_1,x_2,x_3,x_1)$, $(x_2,x_3,x_1,x_2)$, $(x_3,x_1,x_2,x_3))$ and again these will appear $1/3$ times in the data. This reasoning can be generalised to a signal of period p: $q_p=1/p$, consequently ${\mathrm{ED}}_p={log}_2\left(p\right)$ and remains constant for $m\ge p$. Obviously, since we are only using the differences between the $x_i$’s, the periodicity in terms of signs $x_{i+1}-x_i$, may be smaller than the periodicity p of the data, so ${\mathrm{ED}}_p\le {log}_2\left(p\right)$.

4. Chaotic logistic map example

Let us illustrate the use of ED on the well know logistic map [13] $\mathtt{Lo}(x,\lambda )$ driven by the parameter $\lambda$.

$$x_{n+1}=\mathtt{Lo}(x_n,\lambda )=\lambda x_n(1-x_n)$$

(7)

It is obvious that for a range of values of $\lambda$ where the time series reaches a periodic behavior (any cyclic oscillation between n different values), the ED will remain constant. The evaluation of the ED could thus be used as a new complexity parameter to determine the behavior of the time series (see Figure 3).

For $\lambda =4$ we know that the data are randomly distributed with a probability density given by [14]

$$p_{\mathtt{Lo}}\left(x\right)=\frac{1}{\pi \sqrt{(1-x)x}}$$

(8)

But the logistic map produce correlations in the data, so we expect a deviation from the uncorrelated random $q_m$.

We can then compute exactly the ED for an m-embedding, and the KL-divergence from a random signal. For example, for $m=2$, we can determine the $q_2^{\mathtt{Lo}}(+)$ and $q_2^{\mathtt{Lo}}(-)$ by solving the inequality $x<\mathtt{Lo}\left(x\right)$ and $x>\mathtt{Lo}\left(x\right)$ respectively which implies that $0<x<3/4$ and $3/4<x<1$, and then

$$q_2^{\mathtt{Lo}}(+)={\int }_0^{3/4}dx{p}_{\mathtt{Lo}}\left(x\right)=\frac{2}{3}{q}_2^{\mathtt{Lo}}(-)={\int }_{3/4}^{1}dx{p}_{\mathtt{Lo}}\left(x\right)=\frac{1}{3}$$

(9)

In this case the logistic map produces a signal that contains twice as many increasing pairs $``+"$ than decreasing pairs $``-"$. So:

$${\mathrm{ED}}_2=-(\frac{2}{3}{log}_2\frac{2}{3}+\frac{1}{3}{log}_2\frac{1}{3})={log}_2\frac{3}{2^{2/3}}\approx 0.918{\mathrm{KL}}_2=\frac{1}{3}{log}_2\frac{32}{27}\approx 0.082$$

(10)

For $m=3$ we can perform the same calculation, we have respectively:

$$\begin{array}{ccc}\hfill x_1<x_2<x_3& \to & (+,+):0<x<\frac{1}{4}\hfill \\ \hfill x_1<x_3<x_2& \to & (+,-):\frac{1}{4}<x<\frac{1}{8}(5-\sqrt{5})\hfill \\ \hfill x_3<x_1<x_2& \to & (+,-):\frac{1}{8}(5-\sqrt{5})<x<\frac{3}{4}\hfill \\ \hfill x_2<x_1<x_3& \to & (-,+):\frac{3}{4}<x<\frac{1}{8}(5+\sqrt{5})\hfill \\ \hfill x_2<x_3<x_1& \to & (-,+):\frac{1}{8}(5+\sqrt{5})<x<1\hfill \end{array}$$

Graphically we have:

$$\begin{array}{c}\hfill q_3^{\mathtt{Lo}}(++)=\frac{1}{3}{q}_3^{\mathtt{Lo}}(+-)=\frac{1}{3}{q}_3^{\mathtt{Lo}}(-+)=\frac{1}{3}{q}_3^{\mathtt{Lo}}(--)=0\\ \hfill \to {\mathrm{ED}}_3={log}_{2}3\approx 1.58{\mathrm{KL}}_3=\frac{1}{3}\approx 0.33\end{array}$$

(11)

Effectively the logistic map with $\lambda =4$ forbids the string “- -" where $x_1>x_2>x_3$. For strings of length 3 we have:

$$\begin{array}{c}\hfill q_4^{\mathtt{Lo}}(+++)=q_4^{\mathtt{Lo}}(++-)=q_4^{\mathtt{Lo}}(-++)=q_4^{\mathtt{Lo}}(-+-)=\frac{1}{6}{q}_4^{\mathtt{Lo}}(+-+)=\frac{2}{6}\\ \hfill \to {\mathrm{ED}}_4={log}_2{108}^{\frac{1}{3}}\approx 2.25{\mathrm{KL}}_4={log}_2{\left(\frac{16384}{1125}\right)}^{1/6}\approx 0.64\end{array}$$

(12)

The probability of difference $q_m\left(s\right)$ for some string length m versus s the string binary value, where “+"$\to 1$ and “-"$\to 0$, give us the “spectrum of difference" for the distribution q (see Figure 4).

5. ${\mathrm{KL}}_m\left(p\right|q)$ divergences versus m on real data and on maps

The manner in which the ${\mathrm{KL}}_m\left(p\right|q)$ evolves with m is another parameter of the complexity measure. ${\mathrm{KL}}_m\left(p\right|q)$ measures the loss of informations when the random distribution $q_m$ is used to predict the distribution $p_m$. Increasing m introduces more bits information in the signal and the behavior versus m shows how the data diverges from a random distribution.

The graphics (see Figure 6) shows the behavior of ${\mathrm{KL}}_m$ versus m for two different chaotic maps and for real financial data [15]: the opening value of the nasdaq100, bel20 everyday from 2000 to 2013. For maps, the logarithmic map $x_{n+1}=ln\left(a\right|x_n\left|\right)$ and logistic map are shown (see Figure 6 for the logarithmic map).

For maps the simulation starts with a random number between 0 and 1, then first iterate 500 times to avoid transients. Starting with that seeds, 720 iterates where kept on which the ${\mathrm{KL}}_m$ where computed. It can be seen that the Kullback-Leibler divergence from the logistic map at $\lambda =4$ to the random signal is fitted by a quadratic function of m: ${\mathrm{KL}}_m=-0.4260+0.2326m+0.0095m^2$ (p-value $\approx 2{10}^{-7}$ for all the parameter), while the logarithmic map behavior is linear in the range $a\in [0.4,2.2]$. Financial data are also quadratic ${\mathrm{KL}}_m\left(\mathrm{nasdaq}\right)=0.1824-0.0973m+0.0178m^2$, ${\mathrm{KL}}_m\left(\mathrm{bel}20\right)=0.1587-0.0886m+0.0182m^2$ with a higher curvature than the logistic map due to the fact that the spectrum of the probability $p_m$ is compatible with a constant distribution (see Figure 6) rendering the prediction of increase or decrease signal completely random, which is not the case in any true random signal.

Figure 8. The KL-divergence for the data.

Figure 8. The KL-divergence for the data.

Figure 9. The spectrum of $q_8$ versus the string binary value (from 0 to $2^7-1$) for the bel20 financial data.

Figure 9. The spectrum of $q_8$ versus the string binary value (from 0 to $2^7-1$) for the bel20 financial data.

6. Conclusions

The simple property of increases or decreases in a signal makes it possible to introduce the entropy of difference ${\mathrm{ED}}_m$ as a new efficient complexity measure for chaotic time series. The probability distribution of string $q_m$ for random signal is used to evaluate the Kullback-Leibler divergence versus the number of data m used to build the difference string. This ${\mathrm{KL}}_m$ shows different behavior for different types of signal and can also be used also to characterize the complexity of a time series.