Collision Entropy Estimation in a One-Line Formula

1. Introduction

The information theory indices belonging to the parametric family of Rényi entropies are able to express, each with a different weight, the information content of a discrete probability distribution ($DPD$) [1]. Typical members of this family are, for example, Shannon entropy, collision entropy and min-entropy. These indices can also be used to classify the output of experimental processes studied in any branch of the applied sciences, provided their reduction to pseudostationary discrete-state processes and then in the form of $DPD$s. Since usually, during the experiments, only brief realizations can be obtained from the process under investigation, and since the realizations give rise to relative frequency distributions ($RFD$s) and not to $DPD$s, then these indices, being based on probabilities, have to be estimated through the elaboration of the few available data. In this regard, the methods for the estimation of Rényi entropies are of two kinds: 1) those that first aim to estimate the probability distribution from the relative frequencies and then plug the estimated probabilities into the formulas of the entropies and 2) those that circumvent the still-open problem of the estimation of the probabilities and aim to estimate the entropy indices through the application of other elaborations to the data. Despite the numerous studies carried out in the last decades (e.g., [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34]), definitive and universally accepted results for these issues have not been found yet. Moreover, this persistent lack of satisfactory solutions for the estimation of the indices belonging to the Rényi family (and for the estimation of their more rapidly converging derived quantities called Rényi entropy rates) has prompted, as a side effect, an anomalous proliferation of other similar indices conceived in many different ways (e.g. [35,36,37]), but all having the same purpose of classifying data with a nonparametric approach. An overview of this peculiar situation, which, by the way, Shannon in 1956 [38] recommended to avoid, can be found in [39], where Ribeiro et al. collected and described a "galaxy" of at least thirty indices somehow functionally equivalent to those of the family initially proposed by Rényi (and to their rates). Returning to the original question, Skorski ([40,41]) rightly pointed out that the estimation of those integer-order Rényi entropies that have a parameter value greater than one reduces to the estimation of the moments of a $DPD$. Our work just starts from this latter consideration and limits its investigation only to the case of the estimation of the second raw moment, which, in turn, allows the collision entropy to be estimated.

2. Theoretical Methods

2.1. Transforming a Discrete-State Stochastic Process into a DPD

Consider a discrete-state stochastic process ($DS{P}_q$) whose infinite values

$\cdots ,x_{i-1},$$x_i,x_{i+1},\cdots$ belong to an alphabet A containing q ordered symbols. Let $\mathsf{\Omega }(q,d)$ be a d-dimensional discrete sample space resulting from the Cartesian product d times of A

$$\mathsf{\Omega }(q,d)=\underset{dtimes}{\underbrace{A\times A\times ...\times A}},$$

(1)

and let $n=q^d$ be the cardinality of the sample space $\mathsf{\Omega }(q,d)$. Each elementary event ${\mathit{e}}_k$, with $k\in \{1,2,...,n\}$, is uniquely identified by a vector with d coordinates ($x{1}_k,x{2}_k,...,x{d}_k$), with $x{1}_k,x{2}_k,...,x{d}_k\in A$. Following the procedure indicated by Shannon in [42] at pages 5 and 6, the infinite sequence of samples constituting the $DS{P}_q$ can be transformed into occurrences $\#\left({\mathit{e}}_k\right)$ of the elementary events of $\mathsf{\Omega }(q,d)$ by progressively considering all the d-grams taken from the samples as if they were the coordinates of the events and counting the number of times that each coordinate appears in the sequence. Then, according to the frequentist definition of probability, the final resulting $DPD$ is expressible in set theory notation as

$$\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}=\left\{p{\left({\mathit{e}}_k\right)}_{DS{P}_q}=\frac{\#{\left({\mathit{e}}_k\right)}_{DS{P}_q}}{{\sum }_{k=1}^n\#{\left({\mathit{e}}_k\right)}_{DS{P}_q}}|{\mathit{e}}_k\in \mathsf{\Omega }(q,d)\right\}.$$

(2)

In the following, in the absence of ambiguity, $\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}$—that is, a $DPD$ obtained by elaborating the data of a $DS{P}_q$— will be indicated with the bold symbol $\mathit{p}$ and one of its elements with $p_k$.

2.2. Integer-Order Rényi $\alpha$-Entropies as Synthetic Indices for the Characterization of $DPD$s

In general, a $DPD$ can be characterized by some indices, each of which can quantify the presence rate of a particular feature in the distribution. The parametric family of integer-order Rényi α-entropies is composed of synthetic indices suitable for the characterization of $DPD$s from the point of view of their informative content. According to [1] and imposing $\alpha \in {\mathbb{N}}^{+}$, they are defined as

$$\begin{array}{c}\hfill \alpha =1H_1\left(\mathbf{p}\right)\triangleq -\sum _{k=1}^n{p}_k\log p_k\\ \hfill \alpha \ne 1H_{\alpha }\left(\mathbf{p}\right)\triangleq \frac{1}{1-\alpha }\log \left(\sum _{k=1}^n{p}_k^{\alpha }\right)0\le H_{\alpha }\left(\mathbf{p}\right)\le \log n\\ \hfill \alpha \to \infty H_{\infty }\left(\mathbf{p}\right)\triangleq -\log max\left\{\mathbf{p}\right\}.\end{array}$$

(3)

The corresponding specific integer-order Rényi α-entropy of the $DPD$ $\mathit{p}$ is then defined as

$$\begin{array}{c}\hfill \alpha =1{\eta }_1\left(\mathit{p}\right)\triangleq \frac{H_1\left(\mathit{p}\right)}{\log n}=-\sum _{k=1}^n{p}_k{\log }_n{p}_k\\ \hfill \alpha \ne 1{\eta }_{\alpha }\left(\mathit{p}\right)\triangleq \frac{H_{\alpha }\left(\mathit{p}\right)}{\log n}=\frac{1}{1-\alpha }{\log }_n\left(\sum _{k=1}^n{p}_k^{\alpha }\right)0\le {\eta }_{\alpha }\left(\mathit{p}\right)\le 1\\ \hfill \alpha \to \infty {\eta }_{\infty }\left(\mathit{p}\right)\triangleq \frac{H_{\infty }\left(\mathit{p}\right)}{\log n}=-{\log }_{n}max\left\{\mathit{p}\right\}.\end{array}$$

(4)

Once the value of a specific entropy is known, it is always possible to retrieve the value of the corresponding plain entropy, expressed in a particular base b and for a particular cardinality n, using the following conversion formula:

$$H_{\alpha }(\mathit{p},b,n)\triangleq {\eta }_{\alpha }\left(\mathit{p}\right){log}_{b}n.$$

(5)

Specific entropies are preferable to plain entropies because:

1

they are the result of a min-max normalization, that is obtained using the minimum and the maximum possible values of plain entropies (respectively 0 and $\log n$);

2

they are formally independent from the number of ordered symbols q chosen for the quantization of the range of the output values of the process and independent from the cardinality of the sample space n; for this reason, they allow comparable values to be obtained, even for different distributions in different sample spaces;

3

they allow the doubt on the choice of the base for the logarithm present in the formula of entropies ($2$ or $e$ or $10$) to be removed, thanks to the use of a variable base, depending on the cardinality of the considered sample space ($n$);

2.3. Rényi Entropy Rates

Unlike Rényi entropies, whose utility is mainly related to the classification of $DPD$s, Rényi entropy rates are important theoretical quantities useful for the characterization of $DS{P}_q$s [43,44]; they are defined as

$$H_{\alpha }^{\prime }\left(DS{P}_q\right)\triangleq \underset{d\to \infty }{lim}\frac{1}{d}{H}_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)0\le H_{\alpha }^{\prime }\left(DS{P}_q\right)\le \log q.$$

(6)

Moreover, it is known that, for strongly stationary $DS{P}_q$, any Rényi entropy rate converges to the same limit of a sequence of Cesaro means of conditional entropies:

$$H_{\alpha }^{\prime }\left(DS{P}_q\right)=\underset{d\to \infty }{lim}{H}_{\alpha }\left(\mathit{p}\left(A_d\right)\right|\mathit{p}(A_1\times A_2\times \cdots \times A_{d-1})).$$

(7)

and, as conditional Rényi entropies preserve the chain rule [45,46,47], they can also be calculated as

$$H_{\alpha }^{\prime }\left(DS{P}_q\right)=\underset{d\to \infty }{lim}\left[H_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)-H_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d-1)\right)}_{DS{P}_q}\right)\right].$$

(8)

2.4. Specific Rényi Entropy Rate

Similarly to Formula (4), specific Rényi entropy rate is defined by the following min-max normalization:

$$\begin{array}{c}\hfill {\eta }_{\alpha }^{\prime }\left(DS{P}_q\right)=\frac{H_{\alpha }^{\prime }\left(DS{P}_q\right)}{\log q}=\\ \hfill =\underset{d\to \infty }{lim}\frac{\left[H_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)-H_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d-1)\right)}_{DS{P}_q}\right)\right]}{\log q}=\\ \hfill =\underset{d\to \infty }{lim}\left[d{\eta }_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)-(d-1){\eta }_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d-1)\right)}_{DS{P}_q}\right)\right],\end{array}$$

(9)

with $0\le {\eta }_{\alpha }^{\prime }\left(DS{P}_q\right)\le 1$.

2.5. Relationship between Specific Rényi Entropy Rate and Specific Rényi Entropy

In summary, the following relationship subsists:

$${\eta }_{\alpha }^{\prime }\left(DS{P}_q\right)=\underset{d\to \infty }{lim}{\eta }_{\alpha }\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)$$

(10)

This means that, varying d, the specific Rényi entropy tends to the same value of the specific Rényi entropy rate, with the important difference being that the rate of convergence of the specific Rényi entropy rate is much faster than the rate of convergence of the specific Rényi entropy. For this reason, when possible, using the specific Rényi entropy rate is preferable to using the specific Rényi entropy.

3. Empirical Methods

3.1. Transforming a Realization into a Distribution of Relative Frequencies

For the practical cases, the theoretical procedure described in § Section 2.1 can be adapted according to the following procedure already presented with greater generality in [48] and in [49]: consider the N samples $x_1,x_2,\cdots ,x_d,x_{d+1},\cdots ,x_N$ of a realization $r_q$ extracted from a $DS{P}_q$. Each d-gram composed of d adjacent samples of $r_q$ is interpreted as the occurrence of the elementary event of a d-dimensional sample space $\mathsf{\Omega }(q,d)$ having just those values as vector components. For example, the first two d-grams taken from $r_q$, $(x_1,x_2,...,x_d)$ and $(x_2,x_3,...,x_{d+1})$ identify the first occurrences of two elementary events. The count of the occurrences of the events is performed for all the d-grams progressively identified in the sequence of the samples of $r_q$. Finally, the absolute frequency of every elementary event $\#\left({\mathit{e}}_k\right)$ is divided by the total number of occurrences ($L={\sum }_{k=1}^n\#{\left(e_k\right)}_{r_q}=N-d+1$), yielding its relative frequency $f{\left({\mathit{e}}_k\right)}_{r_q}$. The final resulting $RFD$ is expressible in set theory notation as

$$\mathit{f}{\left(\mathsf{\Omega }(q,d)\right)}_{r_q}=\left\{f{\left({\mathit{e}}_k\right)}_{r_q}=\frac{\#{\left({\mathit{e}}_k\right)}_{r_q}}{{\sum }_{k=1}^n\#{\left({\mathit{e}}_k\right)}_{r_q}}|{\mathit{e}}_k\in \mathsf{\Omega }(q,d)\right\}$$

(11)

In the following, in the absence of ambiguity, an RFD $\mathit{f}{\left(\mathsf{\Omega }(q,d)\right)}_{r_q}$ resulting from the insertion of the data of a realization in a sample space will be simply indicated with the bold symbol $\mathit{f}$ and $f_k$ indicates one of its elements.

3.2. Estimating the Second Raw Moment of a $DPD$

Preliminarily the ${\alpha }^{th}$-raw moment of a $DPD$ $\mathit{p}$ and the ${\alpha }^{th}$-raw moment of a $RFD$$\mathit{f}$ are defined as

$$M_{\alpha }\left(\mathit{p}\right)\triangleq \sum _{k=1}^n{p}_k^{\alpha },M_{\alpha }\left(\mathit{f}\right)\triangleq \sum _{k=1}^n{f}_k^{\alpha }\frac{1}{n^{\alpha -1}}\le M_{\alpha }(·)\le 1$$

(12)

Limited to the raw moments of Poissonian distributions, Grassberger in [2], Formula (8), and subsequently Schürmann after sixteen years in [12], Formula (6), reported the theoretically demonstrable, unique unbiased estimator, repeated in Formula (13):

$$\begin{array}{c}\hfill {\widehat{M_{\alpha }\left(\mathit{p}\right)}}_{Poisson}={〈\sum _{k=1}^n\widehat{p_k^{\alpha }}〉}_{r_q}=\sum _{k=1}^n{〈\frac{1}{L^{\alpha }}\frac{\#{\left({\mathit{e}}_k\right)}_{r_q}!}{(\#{\left({\mathit{e}}_k\right)}_{r_q}-\alpha )!}〉}_{r_q}\\ \hfill \widehat{p_k^{\alpha }}:=0for\#{\left({\mathit{e}}_k\right)}_{r_q}<\alpha ,\end{array}$$

(13)

where $<·{>}_{r_q}$ is the mean over the infinite number of realizations that can be taken from the underlying process. For the specific case of the estimation of the second raw moment, Formula (13) becomes:

$$\widehat{M_2(\mathit{p}}{)}_{Poisson}=\sum _{k=1}^n{〈\frac{[\#{\left({\mathit{e}}_k\right)}_{r_q}-1]\#{\left({\mathit{e}}_k\right)}_{r_q}}{L^2}〉}_{r_q}={〈\sum _{k=1}^n{f}_k^2-\frac{1}{L}〉}_{r_q}={〈M_2\left(\mathit{f}\right)-\frac{1}{L}〉}_{r_q}.$$

(14)

As far as we know, the scientific literature does not indicate whether Formula (14) can also be valid for distributions different from Poissonians. So, from now on we proceed assuming provisionally that this hypothesis is true, and we leave the decision concerning its acceptance or rejection to the phase of the interpretation of the results of the Monte Carlo experiments described in one of the following sections. The hypothesis can be resumed as:

$$\forall DS{P}_q{\widehat{M_2\left(\mathit{p}\right)}}_{DS{P}_q}={〈max\left\{M_2\left(\mathit{f}\right)-\frac{1}{L},\frac{1}{n}\right\}〉}_{r_q},$$

(15)

where the lower limit $\frac{1}{n}$ is necessary because, when the cardinality of the sample space becomes high and the data density becomes too rarefied, the only possible estimate of the probability distribution results in the uniform distribution.

3.3. Estimating the Specific Collision Entropy of a $DS{P}_q$

Collision entropy is the particularization of Formula (3) for $\alpha =2$, and it is defined as

$$H_2\left(\mathit{p}\right)\triangleq -\log \left(\sum _{k=1}^n{p}_k^2\right)=-\log M_2\left(\mathit{p}\right)0\le H_2\left(\mathit{p}\right)\le \log n$$

(16)

Inserting Formula (16) into Formula (4), the specific collision entropy is defined as

$${\eta }_2\left(\mathit{p}\right)\triangleq -\frac{H_2\left(\mathit{p}\right)}{\log n}=-{\log }_n{M}_2\left(\mathit{p}\right)0\le {\eta }_2\left(\mathit{p}\right)\le 1.$$

(17)

In the steps of Formulas (13) and (14), the displacements of the symbol that indicates the average over different realizations ${〈·〉}_{r_q}$ from the outside to the inside of the symbol of summation ∑ and vice versa are mathematically indisputable. But the application of the logarithm to the second raw moment for arriving at the estimation of the collision entropy does not allow these shifts anymore. In fact, although the two possible expressions for the evaluation of the mean over the realizations give similar results in the presence of $RFD$s (i.e $-{〈{\log }_n{M}_2\left(\mathit{f}\right)〉}_{r_q}\simeq -{\log }_n{〈M_2\left(\mathit{f}\right)〉}_{r_q}$, in general they differ remarkably when the logarithm is applied to the estimate of the second raw moment:

$$\underset{\mathrm{Mean} \mathrm{of} \mathrm{Logs} \mathrm{of} \mathrm{Moment} \left(\mathrm{MLM}\right)}{\underbrace{-{〈{\log }_{n}max\left\{M_2\left(\mathit{f}\right)-\frac{1}{L},\frac{1}{n}\right\}〉}_{r_q}}}\ne \underset{\mathrm{Log} \mathrm{of} \mathrm{Mean} \mathrm{of} \mathrm{Moments} \left(\mathrm{LMM}\right)}{\underbrace{-{\log }_n{〈max\left\{M_2\left(\mathit{f}\right)-\frac{1}{L},\frac{1}{n}\right\}〉}_{r_q}}}.$$

(18)

Consequently, the estimation of the specific collision entropy is performed averaging the previous two possible expressions:

$$\widehat{{\eta }_2}{\left(\mathit{p}\right)}_{DS{P}_q}=\widehat{-{\log }_n{M}_2\left(\mathit{p}\right)}=\frac{MLM+LMM}{2}.$$

(19)

This is also the main result of this paper. The estimation of plain collision entropy can be obtained by inserting Formula (19) into Formula (5).

3.4. Estimating the Specific Collision Entropy Rate of a DSPq

Moreover, from Formula (9) and Formula (19), it can be inferred that

$$\widehat{{\eta }_2^{\prime }}\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)=\left[d\widehat{{\eta }_2}\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)-(d-1)\widehat{{\eta }_2}\left(\mathit{p}{\left(\mathsf{\Omega }(q,d-1)\right)}_{DS{P}_q}\right)\right]$$

(20)

and

$$\widehat{{\eta }_2^{\prime }}\left(DS{P}_q\right)=min\left\{\widehat{{\eta }_2^{\prime }}\left(\mathit{p}{\left(\mathsf{\Omega }(q,d)\right)}_{DS{P}_q}\right)|1\le d<\infty \right\}.$$

(21)

3.5. Method of Validation of Entropy Estimators

Monte Carlo simulations are the most correct experiments for observing the average effect of the application of an entropy estimator to every realization extracted from a process under examination. The protocol for the validation of the estimators of entropy and entropy rate consists of the following steps:

1.

choice of a convenient $DS{P}_q$,

2.

choice of the number of realizations R,

3.

choice of the length N of each realization,

4.

transformation of the samples of any realization in a $RFD$ according to § 3.1,

5.

extraction of the estimated indices according to Formulas (19) and (20),

6.

production of the diagrams,

7.

and evaluation of the performances of the estimator.

4. Materials: Choice of Convenient $DS{P}_q$s Suitable for the Experiments

For the validation of the previous estimation formulas three completely different types of processes were used: two types, located at the opposite extreme borders of the entropy scale, are regular processes and independent, identically distributed (IID) processes exhibiting maximum entropy; the third type, located in between, is composed of simple processes with minimal memory, such as stationary, irreducible, and aperiodic Markov processes. All these types of processes have the fundamental characteristic of having known theoretical values of entropy; in this way the empirical values obtained by elaborating the realizations can be compared with precise reference values.

1.

Regular Processes. The first important sanity check for entropy estimators involves the use of a completely regular process, that consists of an infinitely repeating brief symbolic sequence. Once the initial sequence is known, no additional information is brought by the following samples, and the evolution of the process becomes completely determined. So, for these processes we have

$$\forall d\ge 2{\eta }_2^{\prime }\left(Regular\right)=0.$$

(22)

Then, even for short realizations of this kind of processes, any good estimator of the specific Rényi entropy rate has to rapidly fall to zero during the progressive increment of the dimension of the sample space.

2.

Markov Processes. When the $DS{P}_q$ is a stationary, irreducible, and aperiodic Markov process, it is possible to calculate the theoretical value of its specific Rényi entropy rate. In fact, given the transition matrix ${\mathit{p}}_{qq}$ and the unique stationary distribution ${\mu }_q^{*}$ obtained as the scaled (with rule $\sum {\mu }_i^{*}=1$) right eigenvector associated to eigenvalue $\lambda =1$ of the equation

$${\left|\begin{array}{cccc}{p}_{11}& p_{12}& ·& p_{1q}\\ p_{21}& p_{22}& ·& p_{2q}\\ ·& ·& ·& ·\\ p_{q1}& p_{q2}& ·& p_{qq}\end{array}\right|}^T\left|\begin{array}{c}{\mu }_1\\ ·\\ ·\\ {\mu }_q\end{array}\right|=\left|\begin{array}{c}{\mu }_1\\ ·\\ ·\\ {\mu }_q\end{array}\right|$$

then

$${\eta }_2^{\prime }\left(Markov\right)\triangleq \underset{d\to \infty }{lim}\frac{1}{d}\frac{H_2\left(\mathit{p}\left(\mathsf{\Omega }(q,d)\right)\right)}{\log q}=-\sum _{i=1}^q{\mu }_i^{*}{\log }_q\left(\sum _{j=1}^q{\mathit{p}}_{ij}^2\right).$$

(23)

3.

Maximum Entropy IID Processes. A third sanity check for entropy estimators involves the use of memoryless IID processes with maximum entropy, because:

• with these processes, the distance between the entropy of the relative frequencies and the actual theoretical entropy of the process is the maximum possible (i.e., using these processes, the estimator is tested in the most severe conditions, obliging it to generate the greatest possible correction);
• the theoretical value for the specific entropy of the processes generated is a priori known and results in being constant, regardless of the choice of the dimension of the considered sample space because the outcome of each throw is independent from the past history.
• having an L-shaped one-dimensional distribution, with one probability bigger than the others, which remain equiprobable, the calculation of their theoretical entropy is trivial;
• they are easily reproducible by, for example, simulating the rolls of a loaded die on which a particular preeminence of the occurrence of a side is initially imposed; the general formula is:

  $${\eta }_2^{\prime }\left(MaxEnt\right)\triangleq {\eta }_2{\left(\mathit{p}(q,d)\right)}_{MaxEnt}{|}_{\forall d}=-{\log }_q\left(p_{main}^2+\frac{{(1-p_{main})}^2}{q-1}\right){|}_{d=1}.$$

  (24)

5. Results and Discussion

As part of this research, countless Monte Carlo experiments were conducted to validate the novel specific collision entropy estimator $\widehat{{\eta }_2}\left(\mathit{p}\right)$ obtained in Formula (19) and, consequently, to verify the plausibility of the hypothesis proposed for the estimation of the second raw moment of any $DS{P}_q$ described by Formula (15). Here, only some of the most significant results are reported. Each figure presented in this section contains two diagrams that show, for an established number of realizations and for an established length of each realization, the trend of the estimated specific collision entropy and the trend of the estimated specific collision entropy rate, calculated as the dimension of the sample space varies.

5.1. Experiments with Realizations Coming from Completely Regular Processes

For the experiment whose results are reported in Figure 1 the input parameters are:

• $DS{P}_q$ = Regular process obtained repeating the ordered numerical sequence of the values associated with the six faces of a die ($q=6$).
• $N=250$ and $R=1$, because every realization is identical.

The upper diagram of Figure 1 shows that, in general, the theoretical specific collision entropy ${\eta }_2\left(\mathit{p}\right)$ decreases only asymptotically to zero and does not reach a minimum value in the dimensional range $1\le d\le 20$. For this reason, this quantity is not indicated for the procedure of process classification. Instead, the lower diagram shows that the specific collision entropy rate ${\eta }_2^{\prime }\left(\mathit{p}\right)$ rapidly decreases to the minimum value of zero, overlapping the theoretical trend for $d>2$. This example shows that, as a first necessary prerequisite, any entropy rate estimator has to exhibit this behavior when dealing with regular processes to be able to be considered suitable for the classification of processes.

5.2. Experiments with Realizations Coming from Processes Presenting Some Sort of Regularity

Consider a Markov process with six possible states (alphabet $A=\{1,2,3,4,5,6\}$ and $q=6$); let the associated transition matrix ${\mathit{p}}_{66}$ and stationary distribution ${\mu }_6^{*}$ be

$${\mathit{p}}_{66}=\left|\begin{array}{cccccc}0.04& 0.80& 0.04& 0.04& 0.04& 0.04\\ 0.04& 0.04& 0.80& 0.04& 0.04& 0.04\\ 0.04& 0.04& 0.04& 0.80& 0.04& 0.04\\ 0.04& 0.04& 0.04& 0.04& 0.80& 0.04\\ 0.04& 0.04& 0.04& 0.04& 0.04& 0.80\\ 0.80& 0.04& 0.04& 0.04& 0.04& 0.04\end{array}\right|{\mu }_6^{*}=\left|\begin{array}{c}\frac{1}{6}\\ \frac{1}{6}\\ \frac{1}{6}\\ \frac{1}{6}\\ \frac{1}{6}\\ \frac{1}{6}\end{array}\right|.$$

For this process, the theoretical value of specific collision entropy rate ${\eta }_2^{\prime }\left(\mathit{p}\right)$ results:

$$\forall d\ge 2{\eta }_2^{\prime }\left(\mathit{p}\right)=\frac{H_2^{\prime }\left(\mathit{p}\right)}{\log q}=-\frac{1}{6}6\frac{\log (0.8^2+5·0.{04}^2)}{\log 6}\simeq 0.242.$$

The upper diagram of Figure 2 shows that, in general, for processes whose samples have a dependence from the past, the trend of the estimated specific collision entropy, calculated using Formula (19), presents, at the beginning, a decrease, which depends on the progressive reduction of the topological ambiguity encountered during the detection of recurrences hidden in the data when the dimension of the sample space is increased. The curve subsequently rises due to the reduction of the density of the occurrences in the sample space. This corresponds to a reduction in the reliability of the information supplied by the relative frequencies; as a consequence, the uncertainty contained in the probability estimates grows, and the entropy increases accordingly. This ability to ramp up the curve when the estimate is no longer reliable is the second necessary prerequisite for an estimator. The observation of the diagrams of Figure 2 allow also to infer that $RFD$s cannot be used in place of $DPD$s because they intrinsically lack this capability. In fact, the use of the $RFD$s in the estimator gives poor results because their mean specific collision entropy seamlessly decreases even when the density of the data is actually no longer sufficient for producing any kind of estimation. In the middle of the curve, the minimum value of the specific collision entropy represents the best possible compromise between the request to observe in ever greater detail the regularities contained in the data and the limitations imposed by the shortness of the data. From Figure 2 it is also possible to establish a third necessary prerequisite that an entropy estimator must fulfill: in fact its output has always to be greater or equal than the corresponding theoretical value, because otherwise the estimator would erroneously signal the presence of an excessive amount of regularities in the process, thus violating the fundamental precaution principle required by all those situations in which statistical fluctuations are present. In a sentence: an estimator that expresses values of entropy higher than the correct theoretical ones is preferable to an estimator that expresses lower values.

Moreover, when the trend of the estimated specific collision entropy is compared with the trend of the estimated specific collision entropy rate, it becomes clear once again that this second index produces an impressively more rapid convergence towards the theoretical value (blue line) than the first one.

In the lower diagram of Figure 2 it is possible to see that the adherence of $\widehat{{\eta }_2^{\prime }}\left(\mathit{p}\right)$ to ${\eta }_2^{\prime }\left(\mathit{p}\right)$ persists up to dimension 11; for this dimension the data density in the sample space results:

$${\delta }_{min}(Markov,R=300,N=500)=\frac{L}{n}=\frac{N-d+1}{q^d}=\frac{490}{6^{11}}=1.35·{10}^{-6}$$

5.3. Experiments with Realizations Coming from Maximum Entropy Memoryless IID Processes

For the experiment whose results are reported in Figure 3, the input parameters are:

• $DS{P}_q$ = process generated by tossing a loaded die with 50% of the outcomes equal to “1” ($q=6$);
• Upper diagram: $R=2000$ and $N=250$;
• Lower diagram: $R=500$ and $N=1000$.

From Formula (24) it results that

$${\eta }_2^{\prime }(MaxEnt50\%)={\eta }_2(MaxEnt50\%)=-{\log }_{6}0.3\simeq 0.672.$$

Both diagrams of Figure 3 show that:

• the proposed estimator satisfies the aforementioned third prerequisite of never falling below the theoretical line, even in the heaviest test conditions, represented by the elaboration of data coming from a maximum entropy IID process;
• when using $RFD$s to estimate specific collision entropy, there is only a slight difference between the two possible ways of averaging the logarithm of the second raw moment (dotted and dashed lines in orange);
• on the contrary, there is a remarkable difference between the two possible ways of averaging the estimates of the logarithm of the second raw moment (dotted and dashed lines in grey) as indicated in Formula (18);
• when the data density in the sample space becomes insufficient for a reliable estimate of the entropy, its value rises toward the value corresponding to the uniform distribution.

In the upper diagram of Figure 3 it is possible to see that considering 250 samples per realization the adherence of $\widehat{{\eta }_2}\left(\mathit{p}\right)$ to ${\eta }_2\left(\mathit{p}\right)$ persists up to dimension 6; for this dimension the data density in the sample space results:

$${\delta }_{min}(MaxEnt50\%,R=2000,N=250)=\frac{L}{n}=\frac{N-d+1}{q^d}=\frac{245}{6^6}=5.25·{10}^{-3}$$

and the statistical fluctuations are considerable because of the shortness of the realizations. In the lower diagram of Figure 3 it is possible to see that considering 1000 samples per realization the adherence of $\widehat{{\eta }_2}\left(\mathit{p}\right)$ to ${\eta }_2\left(\mathit{p}\right)$ persists up to dimension 9 (three dimensions more than the other situation); for this dimension the data density in the sample space results:

$${\delta }_{min}(MaxEnt50\%,R=500,N=1000)=\frac{L}{n}=\frac{N-d+1}{q^d}=\frac{992}{6^9}=9.84·{10}^{-5}$$

and the statistical fluctuations are reduced because of the greater number of samples of each realization. From the comparison of the two diagrams, it can be seen that the increment in the availability of the data improves all the performance indicators of the estimator, and this fact proves its consistency even in the most severe test conditions. In general, to obtain an adequate horizontal trend of $\widehat{{\eta }_2^{\prime }}$ for at least two consecutive dimensions, it is necessary to rely on a sufficiently large number of samples per realization N or, alternatively, on a sufficiently high number of realizations R. The total number of aggregated samples (i.e., R x N) necessary for a good result of the estimation depends on the effective degree of irregularity of the signal. In fact, for completely regular processes with an alphabet composed of few symbols, even only $5q$ samples are sufficient for a correct estimate. In contrast, for almost random processes, at least $1,000,000$ aggregated samples seem to be necessary.

Finally, concerning the hypothesis made at the beginning about the possibility of estimating the second raw moment of the $DPD$s coming from any kind of $DS{P}_q$ using Formula (15), the evidences that emerged from the results of the experiments made for the validation of the estimator have not provided any counterexample that may exclude its validity. For this reason, the following statistics postulate is proposed:

Postulate  

Given a sample space $\mathsf{\Omega }(q,d)$ with cardinality $n=q^d$, and given a set of relative frequency distributions ${\left\{\mathit{f}{\left(\mathsf{\Omega }(q,d)\right)}_{r_q}\right\}}_{DS{P}_q}$, each composed of L occurrences, resulting from the transformation of R short realizations $r_q$ taken from the underlying discrete stochastic process $DS{P}_q$, to which an unknown discrete probability distribution $\mathit{p}\left(\mathsf{\Omega }\right(q,d\left)\right)$ is associated, then the unbiased and consistent estimator of the second raw moment of $\mathit{p}\left(\mathsf{\Omega }\right(q,d\left)\right)$ is inferred as

$$\forall DS{P}_q{\widehat{M_2\left(\mathit{p}\right)}}_{DS{P}_q}={〈max\left\{M_2\left(\mathit{f}\right)-\frac{1}{L},\frac{1}{n}\right\}〉}_{r_q}.$$

6. Conclusions

Figure 2 and Figure 3 show that the proposed specific collision entropy rate estimator $\widehat{{\eta }_2^{\prime }}$ allows a very prolonged and consistent stay of its output, exactly at the values expected by the theory. This highly desirable and very uncommon feature, the simplicity of its formula and its complete usability with any discrete stationary process make this estimator a valid tool, suitable for measuring the degree of irregularity in experimental data from the perspective given by the collision entropy. Possible future research directions include:

• the evaluation of the admissibility of this estimator by comparing it to other similar estimators and by using the same kind of processes for the tests;
• the characterization of the variability of the values returned by the estimator $\widehat{{\eta }_2^{\prime }}$ as the number of aggregated samples and the irregularity of the processes vary;
• further studies on the methods of estimation in presence of the logarithm operator.